C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  TR1; memory and build-times Goto page 1, 2  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message Jaap Suter Guest Posted: Tue Aug 23, 2005 11:45 pm    Post subject: TR1; memory and build-times Hi, Judging from the TR1 proposal, it doesn't seem like function and shared_ptr will allow customization of memory allocations. Both components require dynamically allocating internal structures and it seems the user has no control over where this memory comes from. Is this true, or am I overlooking something? If it is true, is there a rationale for this? It strongly limits their use on embedded systems with zero tolerance for fragmentation. They also seem to be the first standardized -as far as the TR1 goes- components in C++ that allocate memory without giving the user control over it. My second question is related to build-times. It seems that shared_ptr is added into <memory> and function into <functional>. Why are these components not given their own header files? Whenever I'm using shared_ptr, I'm automatically pulling in auto_ptr. And judging from the boost::shared_ptr implementation, I might also be pulling in <algorithm> just because an implementation detail needs std::swap. Any comments on why the TR1 seems to stay away from adding new headers? Thanks, Jaap Suter [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Pete Becker Guest Posted: Wed Aug 24, 2005 8:00 am    Post subject: Re: TR1; memory and build-times Jaap Suter wrote: Quote: Is this true, or am I overlooking something? If it is true, is there a rationale for this? It strongly limits their use on embedded systems with zero tolerance for fragmentation. They also seem to be the first standardized -as far as the TR1 goes- components in C++ that allocate memory without giving the user control over it. You left out regular expressions. There are several places in the standard library where memory is allocated without adding the complexity of supporting custom allocation. Locale facets, some algorithms, and valarray come to mind. Users can control allocation of individual objects (as is done by shared_ptr and the function template) by overloading operator new and operator delete. Quote: My second question is related to build-times. It seems that shared_ptr is added into <memory> and function into <functional>. Why are these components not given their own header files? Because they're like existing things that already have headers. Quote: Whenever I'm using shared_ptr, I'm automatically pulling in auto_ptr. Yup. And when you're using adjacent_find you're automatically pulling in binary_search, and when you're using fopen you're automatically pulling in fclose. Quote: And judging from the boost::shared_ptr implementation, I might also be pulling in algorithm> just because an implementation detail needs std::swap. You might, but that's not required. Quote: Any comments on why the TR1 seems to stay away from adding new headers? It adds <array>, <random>, <regex>, <tuple>, <type_traits>, <unordered_map>, and <unordered_set>. The headers that have things added are <math.h> and <cmath>, <functional>, <memory>, and <utility>. You seem to be assuming that splitting things into more headers will shorten compile times. That's not necessarily true: smaller headers probably compile faster, but in real applications you need more of them. In C the time needed to open a header file overwhelms the time needed to parse it, so splitting things into more headers makes compiles slower. In C++ implementations today that's not as clearly true. But if parsing headers with lots of templates is too slow with your compiler, complain to the compiler vendor -- they probably haven't implemented export. <g> And, of course, having more headers means more header names to learn, and more chance for errors. -- Pete Becker Dinkumware, Ltd. (http://www.dinkumware.com) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Carl Barron Guest Posted: Wed Aug 24, 2005 8:11 am    Post subject: Re: TR1; memory and build-times In article <1124832646.722357.216040 (AT) g44g2000cwa (DOT) googlegroups.com>, Jaap Suter <google (AT) jaapsuter (DOT) com> wrote: Quote: Hi, Judging from the TR1 proposal, it doesn't seem like function and shared_ptr will allow customization of memory allocations. Both components require dynamically allocating internal structures and it seems the user has no control over where this memory comes from. TR1::function is a template with an optional allocator argument. template<typename Function, typename Allocator = std::allocator class function; Boost::shared_ptr uses its own 'undocumented' small object allocator. so TR1::shared_ptr could do so. If this internal allocation is a bottleneck [profile to be sure] and you can modify the class to hold a counter or multiply inherit from the given class, and a simple struct containing the counter, and don't need weak_ptrs then boost::intrusive_ptr does not do any INTERNAL allocations. Only allocation needed is in the constructors first argument, [it has an optional bool argument as well] How you allocate it must match how you deallocate the object via the required free function overloads to manage the counter, in intrusive_ptr<T> but it looks fairly easy. Further boost::intrusive_ptr looks like it can be implimented in one file with no dependencies on boost if desired, especially if you don't need compiler workarounds, for 'older compilers'... [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Maxim Yegorushkin Guest Posted: Wed Aug 24, 2005 9:29 am    Post subject: Re: TR1; memory and build-times Carl Barron wrote: [] Quote: Boost::shared_ptr uses its own 'undocumented' small object allocator. so TR1::shared_ptr could do so. If this internal allocation is a bottleneck [profile to be sure] and you can modify the class to hold a counter or multiply inherit from the given class, and a simple struct containing the counter, and don't need weak_ptrs then boost::intrusive_ptr does not do any INTERNAL allocations. I just skimmed through boost/detail/quick_allocator.hpp and noticed that quick_allocator causes false sharing on SMP. It happens when several counters are allocated within the same cache line and those counters are used by different processors, thus thrashing processors' cache lines when the counter is written, even when the counter is used by a single processor only. More information about allocators and false sharing can be found in papers included with Hoard allocator sources. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Pete Becker Guest Posted: Wed Aug 24, 2005 2:51 pm    Post subject: Re: TR1; memory and build-times Carl Barron wrote: Quote: TR1::function is a template with an optional allocator argument. template class function; std::tr1::function takes exactly one argument. Look it up. -- Pete Becker Dinkumware, Ltd. (http://www.dinkumware.com) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Jaap Suter Guest Posted: Wed Aug 24, 2005 10:19 pm    Post subject: Re: TR1; memory and build-times Quote: Users can control allocation of individual objects (as is done by shared_ptr and the function template) by overloading operator new and operator delete. But they have no control over the shared_count object that shared_ptr allocates internally, correct? Quote: And when you're using adjacent_find you're automatically pulling in binary_search, I've never been a big fan of this either; <algorithm> is huge. Quote: and when you're using fopen you're automatically pulling in fclose. And this one seems okay with me, so I recognize it's not a black and white issue. Nonetheless, I would have preferred to see a more granular <algorithm>, <memory> and <functional>. Quote: And judging from the boost::shared_ptr implementation, I might also be pulling in algorithm> just because an implementation detail needs std::swap. You might, but that's not required. In fact, from my quick browse through the code, std::swap is only used to swap two raw pointers. All of <algorithm> just for that? I would agree it's not really required. Quote: Any comments on why the TR1 seems to stay away from adding new headers? It adds <array>, <random>, <regex>, <tuple>, <type_traits>, unordered_map>, and <unordered_set>. Oops, my bad. Apologies for jumping to conclusions too soon. Quote: But if parsing headers with lots of templates is too slow with your compiler, complain to the compiler vendor -- they probably haven't implemented export. <g , unfortunately I'm in a business where we can't wait for or afford a compiler upgrade. Quote: And, of course, having more headers means more header names to learn, and more chance for errors. If the header-name is the same as the component therein, it's trivial to find. If anything, I'm having a harder time finding my way around standard C++ headers (is it <functional> or <algorithm>) than boost headers (shared_ptr is in shared_ptr.hpp, easy). And what kind of errors are you referring to? Thanks, Jaap [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Jaap Suter Guest Posted: Wed Aug 24, 2005 10:21 pm    Post subject: Re: TR1; memory and build-times Quote: TR1::function is a template with an optional allocator argument. template<typename Function, typename Allocator = std::allocator class function; Correct, that is what n1402 describes here: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2002/n1402.html But perusing n1745.pdf which, afaik, is the official TR1 proposal, the function type template parameter list does not have an allocator argument. Am I overlooking something? Quote: Boost::shared_ptr uses its own 'undocumented' small object allocator. so TR1::shared_ptr could do so. If this internal allocation is a bottleneck [profile to be sure] It is not a performance bottleneck I care about, it's the potential for memory fragmentation. I might scatter my heap with small sizeof(shared_count) byte holes. The only option I see is to write my global allocator as follows: void* operator new (std::size_t size) { if (size == sizeof(shared_count)) return allocate_from_small_object_pool(); else return allocate_from_main_heap(); } Even on PC I'd like to see more people care about this, but on an embedded platform with 24 MB of non-virtual memory, it's a much bigger problem. I see the difficulty with giving shared_ptr an extra template parameter. It would make shared_ptr than shared_ptr<T, AllocatorB>. Could this not be solvable in practice with proper conversions between the two? Quote: ... and don't need weak_ptrs then boost::intrusive_ptr does not do any INTERNAL allocations. I realize that, and encourage people to use intrusive_ptr's where sufficient, but there are many designs in which weak_ptr's make sense. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top arketype@myrealbox.com Guest Posted: Wed Aug 24, 2005 10:23 pm    Post subject: Re: TR1; memory and build-times I looked it up in the proposal (is the proposal the same as the accepted code?) and, thank god, function DOES take an optional allocator argument. Many people (not saying you) don't seem to understand that custom allocators often have nothing to do with performance, I sometimes use custom allocators that routinely perform worse than operator new. They have to do with deterministically bounded times for time constrained applications (AKA realtime). If tr1 function does indeed drop support for allocators, I think that the C++ community is locking out one of it's loyal and growing programmer segments, obviously a big mistake. - JJJ [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Pete Becker Guest Posted: Thu Aug 25, 2005 12:07 pm    Post subject: Re: TR1; memory and build-times Jaap Suter wrote: Quote: Users can control allocation of individual objects (as is done by shared_ptr and the function template) by overloading operator new and operator delete. But they have no control over the shared_count object that shared_ptr allocates internally, correct? Users can control allocation of individual objects (as is done by shared_ptr and the function template) by overloading operator new and operator delete. Quote: And when you're using adjacent_find you're automatically pulling in binary_search, I've never been a big fan of this either; <algorithm> is huge. Seems to me it's about the right size. It holds all the stuff it needs to. What is your criterion for "huge"? Quote: and when you're using fopen you're automatically pulling in fclose. And this one seems okay with me, so I recognize it's not a black and white issue. Nonetheless, I would have preferred to see a more granular algorithm>, <memory> and <functional>. Three headers is still three headers. What is it that you're looking for? Quote: And judging from the boost::shared_ptr implementation, I might also be pulling in algorithm> just because an implementation detail needs std::swap. You might, but that's not required. In fact, from my quick browse through the code, std::swap is only used to swap two raw pointers. All of <algorithm> just for that? I would agree it's not really required. It's not required, so your complaint is about some implementation, not about TR1 itself. Quote: Any comments on why the TR1 seems to stay away from adding new headers? It adds <array>, <random>, <regex>, <tuple>, <type_traits>, unordered_map>, and <unordered_set>. Oops, my bad. Apologies for jumping to conclusions too soon. But if parsing headers with lots of templates is too slow with your compiler, complain to the compiler vendor -- they probably haven't implemented export. :), unfortunately I'm in a business where we can't wait for or afford a compiler upgrade. shrug. Quote: And, of course, having more headers means more header names to learn, and more chance for errors. If the header-name is the same as the component therein, it's trivial to find. So you want one header for each component? Big and slow. Not a good idea. -- Pete Becker Dinkumware, Ltd. (http://www.dinkumware.com) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Pete Becker Guest Posted: Thu Aug 25, 2005 12:08 pm    Post subject: Re: TR1; memory and build-times [email]arketype (AT) myrealbox (DOT) com[/email] wrote: Quote: I looked it up in the proposal (is the proposal the same as the accepted code?) and, thank god, function DOES take an optional allocator argument. It does not take an optional allocator argument. Look it up in the right place. That's the TR1 document, not the proposal that was written nearly three years ago. -- Pete Becker Dinkumware, Ltd. (http://www.dinkumware.com) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Carl Barron Guest Posted: Thu Aug 25, 2005 12:11 pm    Post subject: Re: TR1; memory and build-times In article <1124900119.134951.178320 (AT) g14g2000cwa (DOT) googlegroups.com>, <arketype (AT) myrealbox (DOT) com> wrote: Quote: I looked it up in the proposal (is the proposal the same as the accepted code?) and, thank god, function DOES take an optional allocator argument. Many people (not saying you) don't seem to understand that custom allocators often have nothing to do with performance, I sometimes use custom allocators that routinely perform worse than operator new. They have to do with deterministically bounded times for time constrained applications (AKA realtime). If tr1 function does indeed drop support for allocators, I think that the C++ community is locking out one of it's loyal and growing programmer segments, obviously a big mistake. - JJJ Latest I have seem dated in Jan 2005. function does not have an optional allocator argument in that document, that is template <class Sig> class function; I made the mistake of reading the wrong copy of tr1 [deleted it after I was corrected] Function does NOT take an allocator argument. Confusion abounds even with me, but I don't ussually use custom allocators for anything:) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Jaap Suter Guest Posted: Thu Aug 25, 2005 12:12 pm    Post subject: Re: TR1; memory and build-times Just to reply to myself some more, I actually finished the exercise into something that works. It's not a complete shared_ptr, but it demonstrates one possible way to implement a shared_ptr that allows assignment from shared_ptr<T, AllocatorA> to shared_ptr<T, AllocatorB>. You can see find it on http://www.jaapsuter.com/src/, or http://www3.telus.net/j_suter/src/shared_ptr_t_alloc.cpp as a direct link. I didn't implement operator -> and *, it probably contains some bugs, it doesn't come with tests, and it's probably not exception safe. It's just to demonstrate one possible strategy. It relies on monostate allocators, as well as a virtual function call when the reference count drops to zero. Other than that, I don't see any overhead or problems. But as usual in the most complicated language in the world , I'm probably overlooking some obvious reason why the standardization committee decided to forego on custom allocators. I'm curious to hear why. :), thanks! Jaap Suter - http://www.jaapsuter.com [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Carl Barron Guest Posted: Thu Aug 25, 2005 3:13 pm    Post subject: Re: TR1; memory and build-times In article <SuydnRjV-M435ZHeRVn-jg (AT) rcn (DOT) net>, Pete Becker <petebecker (AT) acm (DOT) org> wrote: Quote: Carl Barron wrote: TR1::function is a template with an optional allocator argument. template class function; std::tr1::function takes exactly one argument. Look it up. Yep you are right, I did look it up but in an older version [n1596] since I have [n1795] time to delete n1596, oh well. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Jaap Suter Guest Posted: Thu Aug 25, 2005 3:14 pm    Post subject: Re: TR1; memory and build-times As an exercise I just tried writing a shared_ptr<T, AllocA> that is assignable to a shared_ptr<T, AllocB>. I could not pull it of without virtual functions or some other extra indirection. To me this explains the standard-commmittee's decision to avoid the extra parameter. It's the only ay to allow libraries from different developers that use shared_ptrs in their interface to work together. I'm not sure what the best solution here is. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Peter Dimov Guest Posted: Thu Aug 25, 2005 11:09 pm    Post subject: Re: TR1; memory and build-times Jaap Suter wrote: Quote: It is not a performance bottleneck I care about, it's the potential for memory fragmentation. I might scatter my heap with small sizeof(shared_count) byte holes. The only option I see is to write my global allocator as follows: void* operator new (std::size_t size) { if (size == sizeof(shared_count)) return allocate_from_small_object_pool(); else return allocate_from_main_heap(); } All reasonable allocators already do something similar, with various degrees of success, and there's always the dlmalloc option when the built-in malloc is broken. http://gee.cs.oswego.edu/dl/html/malloc.html [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.