C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  STL and link speeds          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message Michael Marcin Guest Posted: Thu May 17, 2007 8:18 am    Post subject: STL and link speeds Hello, I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. To do this they remove all STL from the codebase which brought the link times down from several minutes to ~30 seconds. The claim is that the linker was brought to its knees by having to process all the object code that was duplicated in each object file. This seems wrong to me. There should be a way to get good link times while maintaining the clarity, correctness, and elegance that is STL. If their interpretation of the cause is correct then the only solution I can think of is to replace the STL you are using with forward declaration headers and then include the true implementations in one translation unit with explicit instatiations for all types. This is a relatively major undertaking and non-standard as you are putting things declarations into namespace std, but if it's required to keep using the STL I would be willing to bear it. Does anyone have experience with problems such as this in the past? Does anyone have less drastic measures to address the issue of long link times. It isn't currently an issue for our company but I would like to be prepared in case it becomes one later. Thanks, Michael Marcin -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Zeljko Vrba Guest Posted: Thu May 17, 2007 1:01 pm    Post subject: Re: STL and link speeds On 2007-05-17, Michael Marcin <Mike.Marcin (AT) gmail (DOT) com> wrote: Quote: Does anyone have experience with problems such as this in the past? Does anyone have less drastic measures to address the issue of long link times. Yes. I worked on a large project that didn't even use STL (it began when compiler support was poor), so it had a bunch of custom-made code emulating STL. Link time was around 40 minutes on 600 MHz machine with 1G of RAM. This made incremental development very painful, so I split a single monolithic executable into multiple shared libraries and rebuilt only the library which I was currently working on. Linking of a single .so was under a minute; that was acceptable. Ok, now for the matter: linkers have problems with extremely long symbol names which result from C++ name mangling. Yes, to avoid code duplication, the compiler (at least GNU), emits the code for every template function into a separate section which name corresponds (roughly) to the mangled name of the function. Linker then collects all identically-named sections into ONE section and, after performing relocations, dumps the resulting code into the .text section. You can use eg. elfdump or readelf to see which sections are present in every object file and how their names look like. Many of these issues (also ld performance) are discussed in Ulrich Drepper's paper "How to write shared libraries": http://people.redhat.com/drepper/dsohowto.pdf -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Thu May 17, 2007 1:11 pm    Post subject: Re: STL and link speeds Michael Marcin wrote: Quote: Hello, I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. To do this they remove all STL from the codebase which brought the link times down from several minutes to ~30 seconds. The claim is that the linker was brought to its knees by having to process all the object code that was duplicated in each object file. This seems wrong to me. There should be a way to get good link times while maintaining the clarity, correctness, and elegance that is STL. so in order to save a few minutes of link time (hardly a major issue, a dew hours might be, a few days would be)they are willing to spend money handcrafting code that can, and is, generated efficiently by using the STL. The mind boggles. They need to look at the whole process from design to execution -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Hyman Rosen Guest Posted: Thu May 17, 2007 10:15 pm    Post subject: Re: STL and link speeds Francis Glassborow wrote: Quote: so in order to save a few minutes of link time If they do the typical "change,compile,link,test" cycle, spending 30 minutes on a link is not something that can be ignored. But instead of changing their code, they need to find a better implementation. -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Mathias Gaunard Guest Posted: Thu May 17, 2007 10:40 pm    Post subject: Re: STL and link speeds On May 17, 10:18 am, Michael Marcin <Mike.Mar...@gmail.com> wrote: Quote: Hello, I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. To do this they remove all STL from the codebase which brought the link times down from several minutes to ~30 seconds. The claim is that the linker was brought to its knees by having to process all the object code that was duplicated in each object file. This seems wrong to me. There should be a way to get good link times while maintaining the clarity, correctness, and elegance that is STL. What were the compiler, linker and implementation of the standard library which were used? And which options were used? -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top morningcoder Guest Posted: Thu May 17, 2007 10:40 pm    Post subject: Re: STL and link speeds Quote: Try putting all the #include lines for the STL headers you will be using into one single header file and #include that one in all of your .CPP and other header files at the very top...and turn on "use precompiled headers" I'm not sure that would help. The STL headers contain definition of templates not the specialized class you might be using. So including the template definitions in precompiled headers may not improve compilation speed since there are nothing to be compiled yet. Maybe if you also instantiate all the STL templates you're using in the PCH would help, but there may be other technical challenges in doing so. MorningCoder -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Michael Marcin Guest Posted: Fri May 18, 2007 3:49 am    Post subject: Re: STL and link speeds Bob Hairgrove wrote: Quote: On Thu, 17 May 2007 02:18:29 CST, Michael Marcin Mike.Marcin (AT) gmail (DOT) com> wrote: I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. To do this they remove all STL from the codebase which brought the link times down from several minutes to ~30 seconds. The claim is that the linker was brought to its knees by having to process all the object code that was duplicated in each object file. This seems wrong to me. There should be a way to get good link times while maintaining the clarity, correctness, and elegance that is STL. This is probably more of a compiler than a linker issue. The overall build time, which includes both compilation and linkage, is being perceived by your colleague as "link time". In that case, precompiled headers should help. Try putting all the #include lines for the STL headers you will be using into one single header file and #include that one in all of your .CPP and other header files at the very top. Put a "#pragma hdrstop" line after the line including that single header and turn on "use precompiled headers" in your compiler options. The first time it will still take awhile to build, but subsequent builds should go much faster. Of course, any other headers which are included globally should also go into that single header. I was assured this was not a question of compile speeds... it was incremental builds generally changing something in one file and compile + link + test. -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Michael Marcin Guest Posted: Fri May 18, 2007 3:49 am    Post subject: Re: STL and link speeds Zeljko Vrba wrote: Quote: On 2007-05-17, Michael Marcin <Mike.Marcin (AT) gmail (DOT) com> wrote: Does anyone have experience with problems such as this in the past? Does anyone have less drastic measures to address the issue of long link times. Yes. I worked on a large project that didn't even use STL (it began when compiler support was poor), so it had a bunch of custom-made code emulating STL. Link time was around 40 minutes on 600 MHz machine with 1G of RAM. This made incremental development very painful, so I split a single monolithic executable into multiple shared libraries and rebuilt only the library which I was currently working on. Linking of a single .so was under a minute; that was acceptable. That certainly makes sense... and I consider library centric design a bet practice anyways. Quote: Ok, now for the matter: linkers have problems with extremely long symbol names which result from C++ name mangling. Yes, to avoid code duplication, the compiler (at least GNU), emits the code for every template function into a separate section which name corresponds (roughly) to the mangled name of the function. Linker then collects all identically-named sections into ONE section and, after performing relocations, dumps the resulting code into the .text section. You can use eg. elfdump or readelf to see which sections are present in every object file and how their names look like. Many of these issues (also ld performance) are discussed in Ulrich Drepper's paper "How to write shared libraries": http://people.redhat.com/drepper/dsohowto.pdf Thanks for the link I'm reading through it now. -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Guest Posted: Fri May 18, 2007 8:11 am    Post subject: Re: STL and link speeds Michael Marcin <Mike.Marcin (AT) gmail (DOT) com> wrote: Quote: Hello, I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. To do this they remove all STL from the codebase which brought the link times down from several minutes to ~30 seconds. The claim is that the linker was brought to its knees by having to process all the object code that was duplicated in each object file. This seems wrong to me. There should be a way to get good link times while maintaining the clarity, correctness, and elegance that is STL. So I am assuming the 1000ish translation units are built together in one large source tree. I would suggest using a packaging system internally at the company so that you can build, test, and release components of the system seperately. No reasonable package would have 1000s of translation units. I hope my point is clear, -- Ivan Novick http://www.0x4849.net [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Guest Posted: Tue May 22, 2007 12:12 am    Post subject: Re: STL and link speeds On May 17, 10:18 am, Michael Marcin <Mike.Mar...@gmail.com> wrote: Quote: Hello, I was told by a colleague earlier that they were given a task at their previous employer (a large known company) to improve link times on a large (1000ish TU) project. One easy fix: add the linker intermediate files to the virus scanner exception list. This can be a huge boost, as virus scanners are truly stupid. Sure, you're taking quite a few bits of executable code and putting them together, but at least cache the results. Compilation may also benefit by adding .cpp and .h files to the exception lists. Again, I'm unaware of any virus scanner that's actually able to find a virus in source, even there was one. HTH, Michiel. -- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Page 1 of 1   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.