C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Re: include tree analysis          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message Andreas Scherer Guest Posted: Thu Sep 11, 2003 10:34 am    Post subject: Re: include tree analysis Steve Toledo-Brown <StephendotToledo-Brown (AT) uk (DOT) ibm.com> wrote Quote: Are there any alternative tools which would help to root out unnecessary includes? I found the Doxygen utility (http://www.doxygen.org) quite useful for this purpose. In combination with GraphViz (http://www.graphviz.org) it generates (layered) inclusion graphs where redundant includes are easily spotted. Quote: I found an old (2000) post by "Bram Stolk" on this newsgroup referring to "idep", [...] That reminds me of the "[acl]dep" utilities mentioned in John Lakos' "Large-Scale C++ Software Design." The source code for these tools is available http://www.awprofessional.com/isapi/product_id~{9B32D0D8-7E11-429B-9172-9E9229E6D5BE}/selectDescTypeId~{A80972E0-1077-4518-954C- 44E43E341DF7}/st~{75A0BC87-B9B4-435A-86E0-36F0AC8E0923}/session_id~{EDF5BCDF-D0C1-4450-BAE5-1CC02D4F003F}/catalog/product.asp Quote: I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. Neither of the tools mentioned above is capable of doing this and I am not aware of anything in this direction. Take care, Andreas [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Gareth Stockwell Guest Posted: Thu Sep 11, 2003 9:24 pm    Post subject: Re: include tree analysis Hi Steve, I use a tool called Doxygen to achieve something similar to what you want. It's not an inclusion analysis tool per se - actually it's a documentation generator - but is capable of producing inclusion graphs for C,C++ and Java source code. Doxygen is available from http://www.stack.nl/~dimitri/doxygen/ Production of the graphs depends on you having the 'dot' tool installed http://www.research.att.com/sw/tools/graphviz/ What it will not do is notify you of unnecessary inclusions - if anyone knows of a tool which can do so, I'd be interested to hear of it. Gareth Steve Toledo-Brown <StephendotToledo-Brown (AT) uk (DOT) ibm.com> wrote Quote: Does the incl tool (from AT&T Bell Labs) work for C++ as well as C these days? The documents I foiund on the web suggested that incl depends on the cia (C Information Abstractor) tool, but a distinct cia++ tool was later produced. If so, where do I get it from? http://www.faqs.org/faqs/C++-faq/libraries/part6/ suggests it's now owned by SCO (as part of USL), which might make obtaining it politically dificult. Are there any alternative tools which would help to root out unnecessary includes? I found an old (2000) post by "Bram Stolk" on this newsgroup referring to "idep", but the web link is dead and Google didn't find a recent link. I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Ira Baxter Guest Posted: Thu Sep 11, 2003 11:45 pm    Post subject: Re: include tree analysis "Andreas Scherer" <andreas.scherer (AT) pobox (DOT) com> wrote Quote: Steve Toledo-Brown <StephendotToledo-Brown (AT) uk (DOT) ibm.com> wrote Are there any alternative tools which would help to root out unnecessary includes? I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. Neither of the tools mentioned above is capable of doing this and I am not aware of anything in this direction. To do this, you need a tool that can parse the C++ source code and its various includes, and can construct a name-dependency graph based on actual names and types. Having once done this, you can then determine which names depend on which include files, and therefore which ones to retain, with or without the forward class declarations. Ideally, you'd like the tool to then modify your sources appropriately. Our DMS Software Reengineering toolkit won't do this out of box. However, it does have a full C++ parser with name and type resolution, and it is designed explicitly with the intent of being customized in all kinds of ways, including constructing specialized analyses and carrying source transformations. It *could* be customized to do what the OP wants, as well as making the desired source changes. It is a bit of effort to set up such customizations. For a small set of software sources, it might be easier however to use ad hoc approaches as outlined in other responses. See http://www.semanticdesigns.com/Products/DMS/DMSToolkit.html. -- Ira D. Baxter, Ph.D., CTO 512-250-1018 Semantic Designs, Inc. www.semdesigns.com [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Roland Pibinger Guest Posted: Sat Sep 13, 2003 8:12 am    Post subject: Re: include tree analysis On 10 Sep 2003 19:31:29 -0400, Steve Toledo-Brown <StephendotToledo-Brown (AT) uk (DOT) ibm.com> wrote: Quote: Are there any alternative tools which would help to root out unnecessary includes? I found an old (2000) post by "Bram Stolk" on this newsgroup referring to "idep", but the web link is dead and Google didn't find a recent link. I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. This has to be done for all include directives and all files in your application. Of course, this could last for days or even weeks on large projects. Hence the tool must be automatic! But maybe you'll find optimizations to shorten the time needed. Best wishes, Roland Pibinger [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dave Harris Guest Posted: Sat Sep 13, 2003 7:58 pm    Post subject: Re: include tree analysis [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote (abridged): Quote: You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. It's possible that the file will still compile but have different semantics or performance. For example, a header may just contain a specialisation of swap; if it is left out, a more generic version may be used which is less efficient. If when a #include is removed, the file that contained it compiles but the application doesn't, then the #include was in the wrong place. I'd be tempted to just compile the one file in step 2, and if any other files break fix them by hand. -- Dave Harris, Nottingham, UK [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Michael Tiomkin Guest Posted: Sun Sep 14, 2003 1:42 am    Post subject: Re: include tree analysis [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote in message news:<3f620591.622303 (AT) news (DOT) utanet.at>... Quote: On 10 Sep 2003 19:31:29 -0400, Steve Toledo-Brown [email]StephendotToledo-Brown (AT) uk (DOT) ibm.com[/email]> wrote: Are there any alternative tools which would help to root out unnecessary includes? I found an old (2000) post by "Bram Stolk" on this newsgroup referring to "idep", but the web link is dead and Google didn't find a recent link. I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. This has to be done for all include directives and all files in your application. Unfortunately, the last item is not true. As with any other software change, the fact that it compiles doesn't mean that it works right. That include file could contain #define statements that change names used in your application. For a trivial example, you could have #ifdef WIN32 #include ..... // Windows specific #endif #ifdef hpux ....... // HP/UX specific #endif A better approach could be documenting your source files, with the dependences etc.-) BTW, you can try to find where are the declarations of the classes/functions that your code uses in the debug info of the compiled object files. Use "man nm" to see what can be done. "nm -a -l file.o" will most probably give you the names declared when compiling "file.cpp". The include files that do not appear in the objects' filenames are the first candidates to be eliminated. A simple one-liner using nm, sort and grep/awk can give you a list of the .h files that you wouldn't want to remove. For the other files, never remove the files without seeing their source! Michael Quote: Of course, this could last for days or even weeks on large projects. Hence the tool must be automatic! But maybe you'll find optimizations to shorten the time needed. Best wishes, Roland Pibinger [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Roland Pibinger Guest Posted: Sun Sep 14, 2003 11:29 am    Post subject: Re: include tree analysis On 13 Sep 2003 15:58:52 -0400, [email]brangdon (AT) cix (DOT) co.uk[/email] (Dave Harris) wrote: Quote: rpbg123 (AT) yahoo (DOT) com (Roland Pibinger) wrote (abridged): You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. It's possible that the file will still compile but have different semantics or performance. For example, a header may just contain a specialisation of swap; if it is left out, a more generic version may be used which is less efficient. This is possible in principle but also very unlikely so I would neglect it. Quote: If when a #include is removed, the file that contained it compiles but the application doesn't, then the #include was in the wrong place. I'd be tempted to just compile the one file in step 2, and if any other files break fix them by hand. Yes, we must distinguish between header an source files here. An #include in a source file cannot be misplaced, only unnecessary. On the other hand there are several possibilities to misplace an #include in a header file, eg. it is included in the base class file allthough only one derived class needs it or it's not necessary to #include it in the header at all (an #include in the source file would suffice). Only a very sophisticatd tool could detect that. BTW, some people never include anything in a header file. They demand all necessary header files be #included before their header in the source file. The underlying problem is that dependencies are often ignored in large C++ projects until it is too late to refactor the application. It is essential to design layers and packages and define the allowed dependencies between them at the outset of the project (see Lakos' book). Best regards, Roland Pibinger [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Roland Pibinger Guest Posted: Sun Sep 14, 2003 11:30 am    Post subject: Re: include tree analysis On 13 Sep 2003 21:42:15 -0400, [email]tmk (AT) netvision (DOT) net.il[/email] (Michael Tiomkin) wrote: Quote: rpbg123 (AT) yahoo (DOT) com (Roland Pibinger) wrote in message news:<3f620591.622303 (AT) news (DOT) utanet.at>... On 10 Sep 2003 19:31:29 -0400, Steve Toledo-Brown [email]StephendotToledo-Brown (AT) uk (DOT) ibm.com[/email]> wrote: Are there any alternative tools which would help to root out unnecessary includes? I found an old (2000) post by "Bram Stolk" on this newsgroup referring to "idep", but the web link is dead and Google didn't find a recent link. I don't really want tools which merely say what inclusion graphs exist. I would also like to know which included files are unnecessary, first given the existing code and secondly given obvious possible changes such as forward class declarations. You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. This has to be done for all include directives and all files in your application. Unfortunately, the last item is not true. As with any other software change, the fact that it compiles doesn't mean that it works right. That include file could contain #define statements that change names used in your application. For a trivial example, you could have #ifdef WIN32 #include ..... // Windows specific #endif #ifdef hpux ....... // HP/UX specific #endif It is possible but very unlikely that an application compiles and links after such a file has been left out. And, of course, you must test the application after these changes (see also below). Quote: A better approach could be documenting your source files, with the dependences etc.-) The OP asked for a tool! Quote: BTW, you can try to find where are the declarations of the classes/functions that your code uses in the debug info of the compiled object files. Use "man nm" to see what can be done. "nm -a -l file.o" will most probably give you the names declared when compiling "file.cpp". The include files that do not appear in the objects' filenames are the first candidates to be eliminated. A simple one-liner using nm, sort and grep/awk can give you a list of the .h files that you wouldn't want to remove. This sounds interesting allthough I don't know exactly what you mean. nm reports names (defind and undefind) but not (AFAIK) include files (I only used nm to find dependencies between *.so). And it also seems to involve manual treatment which makes it expensive for the projects that need dependency analysis most, ie. large projects (> 1000 files). Quote: For the other files, never remove the files without seeing their source! The brute force tool could only report candidates for removal. Best wishes, Roland Pibinger [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Sun Sep 14, 2003 1:31 pm    Post subject: Re: include tree analysis In article <3f64265b.95560 (AT) news (DOT) utanet.at>, Roland Pibinger <rpbg123 (AT) yahoo (DOT) com> writes Quote: Yes, we must distinguish between header an source files here. But surely header files are source files? I think the terms 'header' and 'implementation' are arguably better. -- Francis Glassborow ACCU 64 Southfield Rd Oxford OX4 1PA +44(0)1865 246490 All opinions are mine and do not represent those of any organisation [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Aaron Bentley Guest Posted: Sun Sep 14, 2003 10:01 pm    Post subject: Re: include tree analysis Francis Glassborow wrote: Quote: But surely header files are source files? I think the terms 'header' and 'implementation' are arguably better. But if you're contrasting 'implementation' with something, wouldn't they be 'declaration' files instead of 'header files'? Except of course, that headers often contain inline functions. Which makes them a form of implementation file... -- Aaron Bentley www.aaronbentley.com [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Michael Tiomkin Guest Posted: Mon Sep 15, 2003 9:45 am    Post subject: Re: include tree analysis [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote in message news:<3f644183.7048562 (AT) news (DOT) utanet.at>... ..... Quote: BTW, you can try to find where are the declarations of the classes/functions that your code uses in the debug info of the compiled object files. Use "man nm" to see what can be done. "nm -a -l file.o" will most probably give you the names declared when compiling "file.cpp". The include files that do not appear in the objects' filenames are the first candidates to be eliminated. A simple one-liner using nm, sort and grep/awk can give you a list of the .h files that you wouldn't want to remove. This sounds interesting allthough I don't know exactly what you mean. nm reports names (defined and undefined) but not (AFAIK) include files (I only used nm to find dependencies between *.so). And it also seems to involve manual treatment which makes it expensive for the projects that need dependency analysis most, ie. large projects (> 1000 files). The "-l" option means showing the line info of a name. When the object is compiled with debug and line info (e.g. "cc -g"), GNU "nm -l" would display the line and file name. When a name is undefined or weakly defined, its definition is probably in another .cpp file, but its declaration might be in a .h file. A template will probably be defined in a .h file. I don't have a Unix/Linux system at my hand right now. If you have, compile a source file where some complicated functions/methods are declared but not defined, e.g. printf or <<, and nm its object file with different options to see the info. You will see that most probably, a printf or << symbol or a function they call if they are inlined will be found in the stdio.h or iostream. These functions are probably in the shared library, and the only debug info in your objects regarding them is from the include files. Now, let's see what can be done automatically to find the candidates to remove. If nm prints the file names where your declarations appear, you can easily extract those file names for the symbol types you need, sort them to remove identical names, merge the sets of file names from different object files (using "sort"), and then find the potential include files that don't contribute to your project - using ls or dependencies files created by gcc and awk or diff. I think you can do this with 10-15 additional lines in your make file, including comments and empty lines. If your project uses 50 directories with 50 similar makefiles, you can include in every makefile a small "makefile.incl.computing" that computes the lists of useful include files. Then add 5 lines to the main makefile to compute the candidates for removal. This can help you make the change in the project. Good luck! Michael [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dave Harris Guest Posted: Mon Sep 15, 2003 3:38 pm    Post subject: Re: include tree analysis [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote (abridged): Quote: If when a #include is removed, the file that contained it compiles but the application doesn't, then the #include was in the wrong place. I'd be tempted to just compile the one file in step 2, and if any other files break fix them by hand. Yes, we must distinguish between header an source files here. An #include in a source file cannot be misplaced, only unnecessary. On the other hand there are several possibilities to misplace an #include in a header file, eg. it is included in the base class file allthough only one derived class needs it or it's not necessary to #include it in the header at all (an #include in the source file would suffice). Only a very sophisticatd tool could detect that. I take the view that headers should be self-contained. Either they compile directly (most compilers can be made to ignore file extension), or a .cpp file like: #include "project_wide.h" #include "header.h" will compile. If necessary such a .cpp file can be generated mechanically by the tool. So then it becomes easy enough for the tool to detect #includes in the wrong place. Hard to figure out the right place, of course. The advantage is that each tests only needs to compile one file. It speeds up the job by a factor of M, where M is the number of source files in the project. PS I've occasionally thought about such a tool for years, but I've never done anything about it. I sometimes simulate it by hand - in other words, I manually strip out headers and recompile to see if they were needed. -- Dave Harris, Nottingham, UK [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Mon Sep 15, 2003 10:05 pm    Post subject: Re: include tree analysis [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote in message news:<3f64265b.95560 (AT) news (DOT) utanet.at>... Quote: On 13 Sep 2003 15:58:52 -0400, [email]brangdon (AT) cix (DOT) co.uk[/email] (Dave Harris) wrote: [email]rpbg123 (AT) yahoo (DOT) com[/email] (Roland Pibinger) wrote (abridged): You could try a brute force approach: Write a little tool that automatically 1. takes a file and removes one include directive from the file 2. compiles the whole project/application 3. If the application still compiles without error the include file is unnecessary. Except that this wouldn't tell you where you could replace the include by a simple forward declaration. Quote: It's possible that the file will still compile but have different semantics or performance. For example, a header may just contain a specialisation of swap; if it is left out, a more generic version may be used which is less efficient. This is possible in principle but also very unlikely so I would neglect it. I'm not sure that it is at all unlikely. Not in C++. The overload set of some functions, such as operator>>, can be quite large, and a certain number of the parameters may have automatic conversion to other types. But IMHO the problem is worse. Suppose your file uses two classes, A and B, defined in two different headers A.hh and B.hh. Correctly, you need to include the two headers. But suppose that class A contains an instance of class B; in this case, A.hh will probably (doubtlessly?) include B.hh, and the brute force testing will say that B.hh is not necessary. Which is true IF A's contract somehow guarantees the A.hh will include B.hh. More typically, however, the authors of A reserve the freedom e.g. to use the compilation firewall pattern sometime in the future, and to remove the include of B.hh. Which breaks your code. Or more correctly, exposes an error that is otherwise hidden. Note that such a case regularly concerns the standard headers -- <string> might include <ostream> in compiler A's implementation, for example, so you would systematically suppress all #include <ostream> from files which also had an #include <string>. Only to have compilatation fail with another compiler (or a newer version of the same compiler). [...] Quote: BTW, some people never include anything in a header file. They demand all necessary header files be #included before their header in the source file. Robert Pike recommended this for C. (I don't know if he still does.) I've never heard it recommended for C++. -- James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr Conseils en informatique orientée objet/ http://www.gabi-soft.fr Beratung in objektorientierter Datenverarbeitung 11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Brian Johnson Guest Posted: Thu Oct 16, 2003 2:11 pm    Post subject: Re: include tree analysis I am coming in very late on this thread and it appears that I am missing many of the original messages, so I am not sure if I am bringing up something that was already mentioned and shot down. I am sure some of this will be a repeat, but hopefully not all. Like the OP, I am looking at cleaning up the includes/dependencies in a large project. I currently have a perl module that will generate an OO Tree of parents and children as well as identify attributes (will also identify scoping and other areas for include use). I was then planning to take this information to first determine if we need a forward declaration or include and then determine the includes in the cpp. (using parent child tree to determine if we want an include in .h instead of forward reference) (assuming class names match file names and also doing a search in each header for namespaces-for scoping, and other potential cases). Once I have determined what I obviously need, I then planned on using the perl script to remove each include and compile and check the output of the compiler. So part of my question is - is this sound?, but I guess first is there a tool that would do part/all of what I want to do (not sure if mentioned in previous messages)? [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Page 1 of 1   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.