From kde-core-devel Mon Sep 17 18:33:02 2001 From: Lubos Lunak Date: Mon, 17 Sep 2001 18:33:02 +0000 To: kde-core-devel Subject: Reducing the number of conflicts in libraries X-MARC-Message: https://marc.info/?l=kde-core-devel&m=100075444227846 Hello, until there's a gcc/binutils/whatever based solution to reduce the bloat caused by conflicts, we could probably do some changes in KDE to reduce the number of conflicts manually (especially if some of the things probably might be needed anyway, and also, who knows when a gcc version doing something with it will be available). In the following text, there are described some ways how that could be done. Some of the changes described there are fairly simple, while other (templates) may be a bit complicated (I'm not sure how much actually). So, the two important questions are : - Are some of the things below worth the trouble, and if yes, which ones? - The second question is just like the first one, only aimed at folks from TT. The following text is also available at http://dforce.sh.cvut.cz/~seli/en/linking2/whatnow.txt , the list of things causing conflicts ( c++filt-ed, sorted, uniq-ed ) is at http://dforce.sh.cvut.cz/~seli/en/linking2/conflicts.txt . The list of conflicts may be also obtained by running an executable linked to KDE libraries with LD_TRACE_PRELINKING set (and glibc patched for prelinking) and grepping the output for 'conflict', or by running nm -D | sed -n 's/^[^ ]* W \(.*\)$/\1/p' and checking which lines appear in more than one library (any idea how to do this with some script or something?). Questions, opinions, flames ? ( volunteers ? :) ) Lubos Lunak llunak@suse.cz ; l.lunak@kde.org http://dforce.sh.cvut.cz/~seli ============ In case you haven't read http://dforce.sh.cvut.cz/~seli/en/linking2 , do so now. In the following text, I'll try to describe how to reduce the number of conflicts in KDE libraries and applications until a gcc/binutils based solution is available (actually, I it might be that some of the problems won't be solvable in gcc/binutils, so some of the things described below will be probably needed anyway). Conflicts in KDE libraries are caused by these things: - virtual tables, and all other virtual class related things like thunks, out-of-line copies of inline virtual methods, typeinfo data - inline functions - templates (I everywhere explicitly say if something is a template or belongs to a class. In other words, 'inline functions' above means global non-template functions that aren't members of any class. Also, 'virtual class' means a non-template class that has at least one virtual method.) Virtual classes : =================== Conflicts here are caused by multiple copies of vtables and other virtual class related things. As described in the linking2 document, gcc in most cases generates only one copy, however there are cases where gcc doesn't try hard enough, and there are also cases where gcc simply can't generate only one copy. As explained in gcc info documentation, vtable is generated only in the module where the first non-inline virtual method of the class is implemented. This is often not enough. One example : class KStaticDeleterBase { public: virtual void destructObject() = 0; }; In the example, there is one virtual method, so vtable is needed, however, because the method is pure virtual, it's not implemented anywhere. So gcc has no other choice but to create the vtable whenever it's needed. Here, the problem can be solved by changing KStaticDeleterBase to this : class KStaticDeleterBase { public: virtual void destructObject() = 0; private: virtual void put_vtable_only_here(); }; and in kstaticdeleter.cpp : void KStaticDeleterBase::put_vtable_only_here() { } Now this makes things much easier for gcc - it can create the vtable only in kstaticdeleter.o . Note that this hack breaks binary compatibility, a solution that wouldn't have this disadvantage is to implement empty KStaticDeleterBase::destructObject() in the source instead of adding a new virtual method. Second example : class Q_EXPORT QGArray { // unimportant virtual ~QGArray(); // unimportant }; template class Q_EXPORT QArray : public QGArray { // only non-virtual methods here ~QArray() {} // only non-virtual methods here }; typedef QArray QByteArray; class Q_EXPORT QCString : public QByteArray { // no virtual method here, not even ~QCString() // only non-virtual methods }; QCString has a virtual method, its destructor (people, why are you so lazy to repeat the 'virtual' keyword in derived classes? Writing the QArray destructor as 'virtual ~QArray(){}' makes things easier to read, and it's only few more keypresses). But ~QCString() is not implemented anywhere, it will be generated automatically by the compiler. The situation is similar with QArray, where the destructor is implemented, but it's inline (methods implemented in the class body are treated as inline), so there's again no .cpp file implementing it. Here gcc could be smarted and realize it can generate the vtable in the module implementing the first non-virtual method of the class, but it doesn't do so (feel free to send mails to your favourite g++ developer asking when it will do so). Here again the hack with private put_vtable_only_here() can help, the other choice is to really implement the destructor as non-inline. Out-of-line copies of inline virtual methods and typeinfo data are generated together with the vtable, so this approach helps here too. But this doesn't seem to apply to thunks these days. Also, note that when the first virtual method in a class is not explicitly marked as inline, but is later implemented as inline, this may lead to conflicts. But this doesn't seem to be a common technique in KDE. Inline functions : ==================== Inline functions are usually compiled directly into the code that calls them, so they actually don't really exist in the code like normal functions, unless this is needed for some reason (a so-called out-of-line copy of inline function). And out-of-line copy is needed e.g. when compiling without optimizations (i.e. without -O), or when a pointer to the function is needed. Again, the compiler here must create a copy whenever an out-of-line copy of an inline function is used. The only solution is not to create inline functions to which pointers are taken. One example of such function is endl(kdbgstream &) . A pointer to this function is taken when calling kdbgstream& kdbgstream::operator<<(KDBGFUNC f). Inline non-template global functions are only rarely causing conflicts. Templates : ============= Templates are causing a high number of conflicts in KDE. Example : a.cpp : void foo1() { // ... QValueList< QString > some_list; // ... } b.cpp : void foo2() { // ... QValueList< QString > some_other_list; // ... } Here both in a.cpp and in b.cpp an instance of template QValueList< T > needs to exist for T = QString. By default such instance is generated automatically by compiler (implicit instantiation) whevener it's needed. So in the example both in a.o and in b.o QValueList< QString > is created. But it already exists in libqt, and therefore it's causing conflicts. The solution of course is to create every instance of every template only once, so if QValueList< QString > exists already in libqt, there shouldn't be another one created elsewhere. This leads to three problems : - where and how to create an instance of a template - how to prevent other places from creating another (same) instance - instances of templates may not disappear from libraries, otherwise binary compatibility would be broken Let's try to do something with QValueList< QString > in the example. Since it already exists in libqt, there's no need to create it again elsewhere. One way how to prevent creation of template instances is to use 'extern template QValueList< QString >;' at the beginning of both a.cpp and b.cpp . This will prevent instantiation of QValueList< QString > in a.o and b.o, and code that uses it will only reference it (i.e. it's similar to 'extern void foo();' or 'extern int bar;' ). But declaring templates as extern is a GCC extension (it gives a warning with -pedantic), so it might be better to split template definitions into two parts : Declarations and inlines in one part (i.e. things that don't create any real code) and the rest in the other part. If you look in qvaluelist.h, you'll notice that all template classes in this header file have their methods implemented in the class body, which automatically makes them inline, even if they aren't explicitly marked so. Some of the methods are quite large and this causes larger executables. Moreover gcc won't inline some of them (compile with -Winline to see), because they are simply too large or for other reasons. Instead inlining them, an out-of-line copy for them will be created, and such multiple copies will cause conflicts. All large methods in templates should be moved out of the class body, so they won't be treated as inline. Also, they should be either enclosed by an #ifdef block, or they should be in a separate source file. Since the definitions of non-inline template methods won't be available, they won't be instantiated anywhere. That should answer how to prevent other places from creating duplicate copies. The question where and how the one single instance of template should be created still remains. An instance can be created by putting 'template QValueList< QString >;' in one source file, after #including a file with complete definition of QValueList< T >, including non-inline methods. With older C++ compilers that can't handle this construct, one uses 'typedef QValueList< QString > QValueListQString;' instead. The place where one specific template instance should be created is of course the library where it's first used, in this QValueList< QString > case, it's libqt. There are still two problems remaining. One of them is the fact that QValueList< QString > may one day disappear from a newer Qt version (e.g. because QMap< QString > is used instead it). This would break binary compatibility for KDE libraries relying on QValueList< QString > being present in libqt - after upgrading to this newer Qt version, KDE libraries would cease to work. A solution would be to run a script on every library version that would list all such symbols and make sure that every new version contains all the symbols from the older version. ( nm -D | sed -n 's/^[^ ]* W \(.*\)$/\1/p' | c++filt | grep '<' | LC_ALL=C sort should print such list. I'm not actually sure what exactly it does, but it seems to work). All missing symbols could be then explicitly instantiated using the template keyword, e.g. 'template QString& QValueList< QString >::operator[](size_type);'. Just in case you ask what the second problem is, the second problem would be maintaining it. One would have to check all those (probably many) symbols with every new library version. When simply using 'template QValueList< QString >;', this would lead to creating many probably unused instances of template methods, which would lead to increasing libraries sizes, on the other hand, if not all methods would be instantiated, they might be needed in other libraries. Also, for every instance of QValueList< T >, there must exists also one instance of QValueListPrivate< T >, QValueListNode< T > and usually also QValueListIterator< T > and QValueListConstIterator< T >. There may be also another ways of handling templates that I'm not aware of, e.g. using the 'export' keyword (which is not implemented in gcc yet). I'm not exactly sure how it is supposed to work, and I had only a C++ draft dated 2 December 1996 available, so there may be some differences to the final ISO C++ document. ---------------------------------------------------------------------- Lubos Lunak