C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  resizing memory Goto page 1, 2  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message John Dill Guest Posted: Wed Feb 18, 2004 12:22 pm    Post subject: resizing memory I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? Thanks, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Thomas Mang Guest Posted: Thu Feb 19, 2004 10:31 am    Post subject: Re: resizing memory John Dill schrieb: Quote: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? Use the sequence container class-templates provided by the Standard Library to get what you need: class foo{}; std::vector<foo> FooVec; // if you have an idea how many elements the container will contain (or the maximum number it may contain), call the reserve member funtion: FooVec.reserve(1000); and then populate the Vector by calling its push_back / insert - member function. If, after you are done with populating the vector, the difference between Vec.size() and Vec.capacity() (which is the surplus of unneeded, but allocated memory) is significant and you want to get rid of it, use this trick: std::vector<foo>(FooVec).swap(FooVec); Please note that the Standard does, AFAI can tell, NOT guarantee that the temporary vector will use only as much memory as needed to hold all elements. In practice, however, I think the trick will do what it is supposed to do. Anybody knows an implementation where it does not hold? regards, Thomas [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Thu Feb 19, 2004 1:30 pm    Post subject: Re: resizing memory In message <302c79f4.0402171810.7b52a9a8 (AT) posting (DOT) google.com>, John Dill <john-dill (AT) uiowa (DOT) edu> writes Quote: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? The fundamental problem is that most OSs do not provide a mechanism for resizing dynamic memory in situ. Of course you could write your own memory management from scratch (get a block of memory from the OS which you manage yourself) but the C++ Standard does not provide that as part of its facilities. There are other issues that programmers often assume they know the answers to. For example, what happens to memory I free? Does it actually get released for other processes to use? -- Francis Glassborow ACCU Author of 'You Can Do It!' see http://www.spellen.org/youcandoit For project ideas and contributions: http://www.spellen.org/youcandoit/projects [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top bert hubert Guest Posted: Thu Feb 19, 2004 2:32 pm    Post subject: Re: resizing memory On 2004-02-18, John Dill <john-dill (AT) uiowa (DOT) edu> wrote: Quote: Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? As far as I know you'll have to resort to using malloc/realloc/free. On many platforms, realloc actually uses an efficient systemcall to remap memory. -- http://www.PowerDNS.com/ Open Source Database Driven Nameserver http://lartc.org Linux Advanced Routing & Traffic Control HOWTO [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Jack Klein Guest Posted: Fri Feb 20, 2004 12:52 am    Post subject: Re: resizing memory On 18 Feb 2004 07:22:25 -0500, [email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote in comp.lang.c++.moderated: Quote: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? Thanks, John Your data buffer sounds like it holds simple built-in types, such as unsigned char. In that case, one possibility is to use malloc() or perhaps calloc() the allocate it in the first place, and realloc() to change its size. These functions are, of course, fully ISO Standard C++. They are generally not good to use for user-defined types like classes, but sometimes are the simplest method to use for primitive types. -- Jack Klein Home: http://JK-Technology.Com FAQs for comp.lang.c http://www.eskimo.com/~scs/C-faq/top.html comp.lang.c++ http://www.parashift.com/c++-faq-lite/ alt.comp.lang.learn.c-c++ http://www.contrib.andrew.cmu.edu/~ajo/docs/FAQ-acllc.html [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Ivan Vecerina Guest Posted: Fri Feb 20, 2004 1:01 am    Post subject: Re: resizing memory "John Dill" <john-dill (AT) uiowa (DOT) edu> wrote Quote: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? It is possible for compression algorithms to perform "in-place" compression or decompression of a data block. IIRC, the LZO compression library might be a good fit for this on the decompression side at least: it is both very fast and supports in-place decompression. On the compression side, things could be more complicated for any algorithm: no compression algorithm can guarantee that data size is reduced, so a failure mode should be supported). After in-place compression (or before decompression), the equivalent of a 'realloc' function can be used to move the compressed (or to be decompressed) memory block to a new location. A NG about compression algorithms would be more appropriate to get help in picking the right library. Regards, Ivan -- http://ivan.vecerina.com/contact/?subject=NG_POST <- e-mail contact form [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Fri Feb 20, 2004 11:21 am    Post subject: Re: resizing memory [email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote in message news:<302c79f4.0402171810.7b52a9a8 (AT) posting (DOT) google.com>... Quote: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? If it is raw memory (bytes, char's, etc. -- anything with only POD types in it, in fact), allocate with malloc, and use realloc for resizing. -- James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr Conseils en informatique orientée objet/ http://www.gabi-soft.fr Beratung in objektorientierter Datenverarbeitung 11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dhruv Matani Guest Posted: Fri Feb 20, 2004 11:34 am    Post subject: Re: resizing memory On Thu, 19 Feb 2004 05:31:57 -0500, Thomas Mang wrote: Quote: John Dill schrieb: I am wondering what is the standard method to resize memory for reducing the amount of memory used. I am working with in-memory compression and want to have the memory expand enough to hold the compressed data, but then trim the excess after the algorithm is over. Must I reallocate with a lower memory area and then fill that memory with the data? Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? Use the sequence container class-templates provided by the Standard Library to get what you need: The OP asked: Can I shrink the memory down without allocating a smaller space and copying the contents to it with a c++ methodology? I don't think that the standard container (vector) makes that guarantee. To the OP: Why exactly are you interested in such shrinking behaviour? There is a way, nu it's non-portable. Also, there are a certain restriction that one has to follow while using it: 1. Do not use any other Heap-management function such as malloc/calloc when using this technique. 2. You may use only ONE(1) region of memory at a time. Code: ===== void *Get_Memory (size_t sz) { return sbrk(sz); } void Expand_Memoey (size_t sz) { sbrk(sz); } void Shrink_By (size_t sz) { sbrk(-sz); } Call Get_Memory only once. When you want to expand/shrink the memory, call the appropriarte functions. Regards, -Dhruv. Proud to be a Vegetarian. http://www.vegetarianstarterkit.com/ http://www.vegkids.com/vegkids/index.html [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Andrei Alexandrescu Guest Posted: Fri Feb 20, 2004 2:51 pm    Post subject: Re: resizing memory "Francis Glassborow" <francis (AT) robinton (DOT) demon.co.uk> wrote Quote: The fundamental problem is that most OSs do not provide a mechanism for resizing dynamic memory in situ. I'd rewrite that as "The fundamental problem of C++ object allocator is that it is unable to exploit the mechanism for resizing dynamic memory in situ provided by most OSs." Andrei the merciless one [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Fri Feb 20, 2004 8:43 pm    Post subject: Re: resizing memory In message <7j9830lu8u2aeg4t2mvf7cao6p696q4jum (AT) 4ax (DOT) com>, Jack Klein <jackklein (AT) spamcop (DOT) net> writes Quote: Your data buffer sounds like it holds simple built-in types, such as unsigned char. In that case, one possibility is to use malloc() or perhaps calloc() the allocate it in the first place, and realloc() to change its size. These functions are, of course, fully ISO Standard C++. They are generally not good to use for user-defined types like classes, but sometimes are the simplest method to use for primitive types. But the specification of realloc has nothing to say on the subject of what happens to the excess memory. On many systems using it to attempt to reduce memory use does nothing other than put the extra out of reach of anything until free is called on the current pointer (returned by realloc). Try using realloc to reduce memory use and you will discover that on many systems it just returns the original pointer, on others it copies the original and returns the old memory to the programs own resource heap. -- Francis Glassborow ACCU Author of 'You Can Do It!' see http://www.spellen.org/youcandoit For project ideas and contributions: http://www.spellen.org/youcandoit/projects [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Stephen Howe Guest Posted: Sat Feb 21, 2004 4:07 am    Post subject: Re: resizing memory Quote: I'd rewrite that as "The fundamental problem of C++ object allocator is that it is unable to exploit the mechanism for resizing dynamic memory in situ provided by most OSs." Absolutely. And being told to "use a vector" is not the right answer. As far as I am concerned, until renew [] exists and the allocator interface (and the hint is not good enough as different vendors can use it differently which means no portability for the library writer) is enhanced, the last mileage of possible performance will never be squeezed out of vector. That is a weakness in C++. vector may be spending its time, needlessly invoking copy constructors and destructors when it might be avoided. Also merciless Stephen Howe [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top John Dill Guest Posted: Sat Feb 21, 2004 10:41 am    Post subject: Re: resizing memory Quote: If it is raw memory (bytes, char's, etc. -- anything with only POD types in it, in fact), allocate with malloc, and use realloc for resizing. I allocate my internal buffers using "operator new", and mixing realloc and "operator new" together is not attractive. How hard would it be to say write my own "operator renew()" as a wrapper to realloc seeing as most "operator new()" are implemented in terms of malloc. I don't see it as being too difficult, though probably non-portable in the strictest sense for those allocators that don't use strictly malloc (I can't think of any). I think that this is an oversight of the "operator new/delete" system seeing that realloc is standard C feature. I think the main problem with this concept is that people would get confused and assume that this "operator renew" can handle non-POD, but there should be some wrapper that interfaces with the "operator new/delete" system that takes advantage of using realloc. Best, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Andrei Alexandrescu Guest Posted: Sat Feb 21, 2004 10:45 am    Post subject: Re: resizing memory "Francis Glassborow" <francis (AT) robinton (DOT) demon.co.uk> wrote Quote: But the specification of realloc has nothing to say on the subject of what happens to the excess memory. On many systems using it to attempt to reduce memory use does nothing other than put the extra out of reach of anything until free is called on the current pointer (returned by realloc). Try using realloc to reduce memory use and you will discover that on many systems it just returns the original pointer, on others it copies the original and returns the old memory to the programs own resource heap. What are those "many systems". My cell phone maybe. (It's the cheapest I could find. Actually the cost was negative - I got paid $100 to use it. No shrinking realloc for that money.) I know realloc does shrink memory on MSVC/Windows. Linux and cygwin (IIRC) use Lea's allocator, http://gee.cs.oswego.edu/dl/html/malloc.html, considered by many the fastest general-purpose allocator in the world. Lea's allocator does support in-place shrinking and I wouldn't think the Linux fellas just didn't want to use that. I got word from Metrowerks that their realloc does in-place shrinking, too. C made one little mistake in designing their allocator API, by not offering in-place reallocation functions and have realloc take care of two things at the same time; they didn't care much about it because bitwise copy was all anyone needed. C++ is dearly paying for that little mistake ever since. Andrei [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Sat Feb 21, 2004 4:23 pm    Post subject: Re: resizing memory In message <c15tqi$1ettme$1 (AT) ID-14036 (DOT) news.uni-berlin.de>, Andrei Alexandrescu <SeeWebsiteForEmail (AT) moderncppdesign (DOT) com> writes Quote: "Francis Glassborow" <francis (AT) robinton (DOT) demon.co.uk> wrote in message news:MCe60PDTMeNAFwzp (AT) robinton (DOT) demon.co.uk... But the specification of realloc has nothing to say on the subject of what happens to the excess memory. On many systems using it to attempt to reduce memory use does nothing other than put the extra out of reach of anything until free is called on the current pointer (returned by realloc). Try using realloc to reduce memory use and you will discover that on many systems it just returns the original pointer, on others it copies the original and returns the old memory to the programs own resource heap. What are those "many systems". My cell phone maybe. (It's the cheapest I could find. Actually the cost was negative - I got paid $100 to use it. No shrinking realloc for that money.) I know realloc does shrink memory on MSVC/Windows. Linux and cygwin (IIRC) use Lea's allocator, http://gee.cs.oswego.edu/dl/html/malloc.html, considered by many the fastest general-purpose allocator in the world. Lea's allocator does support in-place shrinking and I wouldn't think the Linux fellas just didn't want to use that. I got word from Metrowerks that their realloc does in-place shrinking, too. And none of that conflicts with what I wrote. I am very happy that the most recent C++ implementations provide in place shrinkage. However the programmer must check that the compiler in its current mode (switches set etc.) does provide in place shrinkage if that is important to the programmer. In addition, I can see nothing in the C++ Standard that prevents an implementation of std::vector from using in place shrinkage if it is available, it is just not something that is required so once again the programmer must read the documentation. Quote: C made one little mistake in designing their allocator API, by not offering in-place reallocation functions and have realloc take care of two things at the same time; they didn't care much about it because bitwise copy was all anyone needed. C++ is dearly paying for that little mistake ever since. It would, back in the 80s, have been quite unreasonable to require in place shrinkage semantics for realloc. That has nothing to do with whether or not bitwise copying was being used and a lot to do with the facilities then available. Of course any system that could implement in place shrinkage efficiently was entirely free to do so, it just wasn't and isn't required. AFAICS exactly the same applies to std::vector. There is nothing that I am aware of that forbids in place shrinkage, however requiring it would, IMHO, be a step too far (and how would you know that it had actually been accomplished? Just because the addresses remain the same does not mean that it has not been copied to a new block of memory that gets mapped in to replace the original. Actually we never know what is physically providing the resource that our process is using as memory and it is becoming increasingly irrelevant on desktops etc. Of course it matters more on small hand-helds (but for how much longer?). -- Francis Glassborow ACCU Author of 'You Can Do It!' see http://www.spellen.org/youcandoit For project ideas and contributions: http://www.spellen.org/youcandoit/projects [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Allan W Guest Posted: Sun Feb 22, 2004 1:41 am    Post subject: Re: resizing memory "Andrei Alexandrescu" <SeeWebsiteForEmail (AT) moderncppdesign (DOT) com> wrote Quote: I know realloc does shrink memory on MSVC/Windows. I stepped into free() and eventually got to this code: /* optionally reclaim memory */ if (!(_crtDbgFlag & _CRTDBG_DELAY_FREE_MEM_DF)) { if (pHead->pBlockHeaderNext) { pHead->pBlockHeaderNext->pBlockHeaderPrev = pHead->pBlockHeaderPrev; } else { _ASSERTE(_pLastBlock == pHead); _pLastBlock = pHead->pBlockHeaderPrev; } if (pHead->pBlockHeaderPrev) { pHead->pBlockHeaderPrev->pBlockHeaderNext = pHead->pBlockHeaderNext; } else { _ASSERTE(_pFirstBlock == pHead); _pFirstBlock = pHead->pBlockHeaderNext; } /* fill the entire block including header with dead-land-fill */ memset(pHead, _bDeadLandFill, sizeof(_CrtMemBlockHeader) + pHead->nDataSize + nNoMansLandSize); _free_base(pHead); } else { pHead->nBlockUse = _FREE_BLOCK; /* keep memory around as dead space */ memset(pbData(pHead), _bDeadLandFill, pHead->nDataSize); } MSVC 6.0 clearly does not return memory to the OS when you call free(). Doing the same thing with realloc() is left as an exercise. Disclaimer: I did this in debug mode. The results would be different in release mode but I'm pretty sure the total memory footprint wouldn't change there either. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.