C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Exceeding memory while using STL containers Goto page 1, 2  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message blwy10 Guest Posted: Tue May 30, 2006 2:40 pm    Post subject: Exceeding memory while using STL containers What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. Thanks all. Benjamin Lau [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top shailesh Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers For memory management STL uses allocators. One can redefine the allocators if required. Now the default allocator provided with STL uses ::operator new for memory allocation. It may throw appropriate exceptions. i.e. it will throw std::bad_alloc when it cannot allocate memory. Here is the definition of _Allocate function inside allocator<T> class with VC distribution: template<class _Ty> inline _Ty _FARQ *_Allocate(_SIZT _Count, _Ty _FARQ *) { // allocate storage for _Count elements of type _Ty return ((_Ty _FARQ *)operator new(_Count * sizeof (_Ty))); } Given this, the next question is, what happens with the STL container in the presence of this std::bad_alloc exception. All standard containers implement basic exception safety guarantee. In general insert() operations have strong exception guarantee. But multi-element inserts are never strongly exception safe. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Alan Johnson Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. A correct implementation will throw a std::bad_alloc exception. -- Alan Johnson [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Shimon Shvartsbroit Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. Most of chances you'll get std::bad_alloc exception thrown. Since STL is exception safe library, in such case AFAIK it follows the "strong" exception gurantee. Which means that the container will hold all succesfully inserted elements untill it ran out of memory. For instance, if there were five items in your set container, and by adding the sixth item the computer ran out of memory, the container will still hold those five elements as if the sixth element had never been added. In addition to that the "insert" method will throw std::bad_alloc. http://www.gotw.ca/gotw/059.htm , look for "Strong Guarantee" Regards, Shimon [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers Hellol Benjamin, The STL containers were built so they would be able to handle their own memory. When you insert an element into a container and there is not enough memory, the container will throw a std::bad_alloc exception that you should catch and exit the program. There is not much you can do if you suspect that the size of the set is going to take up all your RAM. If the set is really that huge, you can write the set in a binary file and use your hard drive's memory instead, but this requires knowledge of C++ file I/O. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Ulrich Eckhardt Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? This is signalled via exceptions, std::bad_alloc I'd say. Uli [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top ThosRTanner Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? It throws an exception (std::bad_alloc I think) I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. What is implementation defined is at what point your system will throw bad_alloc, which is dependant not just on what STL you are using and what OS you are running but also how much memory the OS permits the program to have, how much swap space is in use, .... Note: I've come across one implementation of C++ where, given certain memory constraints, when the computer runs out of memory, your heap overwrites your stack. This is from people who should have known better. But they won't change it because apparently speed is more important than correct and predictable functioning of a program. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Mehturt@gmail.com Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. I guess it depends on the allocator used, but on my platform operators new and delete are used in the default allocator so I think when running out of memory std::bad_alloc is thrown. I don't know if this is standard though. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top n2xssvv g02gfr12930 Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. I'd suggest that the STL function that failed would throw one of the defined STL errors which your code could catch. Obviously if an error occurs that the STL library cannot possibly hope to be aware of, and therefore check for, you'll get a standard error. JB JB [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Daniel Aarno Guest Posted: Wed May 31, 2006 4:20 pm    Post subject: Re: Exceeding memory while using STL containers This would depend on the allocator you are using. You can supply your own allocator object to each of the containers in STL that takes care of allocating and releasing memory. If you use the default I guess new and delete are being used. Contrary to what many people believe new nowadays throws an exception when memory allocation fails (unless explicitly told not to). new does NOT return 0, NULL or anything else by default. Actually you can install a new_handler that is called when memory allocation fails the first time. If you new_handler is able to find memory for new (by releasing a pool for example) the operation can resume otherwise the exception is thrown. So given that you don't catch bad_alloc exceptions (or a base thereof) or have a new_handler that takes care of the problem terminate is invoked. Terminate will call you uncaught_handler if you have one set or default to calling abort which aborts the process without any cleanup (such as flushing buffers etc). /Daniel Aarno blwy10 skrev: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top dhruv Guest Posted: Wed May 31, 2006 8:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. The container's insert() function will throw an std::bad_alloc() exception, and I don't think there's much that can be done about it either. The result is NOT implementation dependant, and is defined by the C++ standard. This is of course assuming that you are using std::allocator() or some replacement that adheres to the standard. Regards, -Dhruv. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Vaclav Haisman Guest Posted: Wed May 31, 2006 8:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. Thanks all. Benjamin Lau std::bad_alloc is thrown? -- VH [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Joshua Lehrer Guest Posted: Wed May 31, 2006 9:20 pm    Post subject: Re: Exceeding memory while using STL containers ThosRTanner wrote: Quote: Note: I've come across one implementation of C++ where, given certain memory constraints, when the computer runs out of memory, your heap overwrites your stack. This is from people who should have known better. But they won't change it because apparently speed is more important than correct and predictable functioning of a program. This is the way that, in college, I was taught that computers are supposed to 'work'. The stack grows up, the heap grows down, and when they meet, you are in trouble. I currently work on a platform where the stack grows down, but is limited to 1gig, and the heap grows up, and is also limited to 1 gig. This is problematic because we use very little stack space but lots of heap, and yet we are still limited to 1 and 1. With a floating separator between stack and heap, we could get 1.8 and 0.2, which is more like what we actually need. I guess another solution would have the heap and stack growing toward eachother. If the heap ever needed to grow and it detected it was too close to the stack, it would throw bad_alloc. However, what do you do when the stack approaches the heap? Is it allowed to throw bad_alloc at the point when it detects this? If not, what do you do? If so, isn't this a lot of overhead to calculate each time you need to grow the stack? Growing the stack is supposed to be fast and cheap (usually just an add). joshua lehrer http://www.lehrerfamily.com/ [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Guest Posted: Wed May 31, 2006 9:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. What happens then? I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. To be more precise, I am using Windows XP SP 2 and VC Toolkit 2003, including its STL implementation. The container specifically is std::set. Thanks all. Benjamin Lau STL containers are "Indifferent" to memory allocation issues since one of their template parameters is the allocation scheme to be used. So rather then asking what the container will do, you have to ask yourself what the allocator will do. First you have to read about the allocator class. In general though it has 4 major functions: allocate - which do row memory allocation constact - which "move" an object to the allocated memory allocated by allocate function (this is really simplified explanation) destroy - destroy the object but not free the memory deallocate that do the opposite of allocate Now it's all deepened on what allocator you are using with your containers. There is a default allocator that STL implements and this is really OS and compiler vendor specific. Since you can see the source code of the SRL allocator and you know whether you are using it I bet you can figure the rest yourself good luck [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Guest Posted: Wed May 31, 2006 9:20 pm    Post subject: Re: Exceeding memory while using STL containers blwy10 wrote: Quote: What happens when in the process of using STL containers, we insert so many elements that our computer runs out of memory to store them. The C++ standard mandates that the default `new' should throw the `bad_alloc' exception when it runs out of memory. Quote: I assume that this behaviour is implementation-defined and dependent on what STL am I using and what OS I am running but I'm just curious to know what would/should typically happen. Benjamin Lau This behavior is not implementation-defined; it is clearly specified in the standard. And by definition, STL should behave as per the standard, irrespective of the OS on which it is implemented. If your program does not alter the standard behavior of `new' by calling set_new_handler(), it should throw the `bad_alloc' exception when memory is exhausted. If your program is not prepared to catch that exception, it will mostly be terminated. -amitav [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.