 |
C++Talk.NET
C++ language newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
John Dill
Guest
|
 Posted: Sat Jan 31, 2004 1:41 am Post subject: soft memory limit |
 |
|
Is there a (easy) way for me to arbitrarily set the amount of memory
that my program can use to test memory allocation errors? I have a
3GB memory computer for image processing stuff that I develop on and
would like to set a much less arbitrary amount. Can you do this
through making your own allocator/heap and setting new and delete to
interface with that (just for testing purposes)?
Thanks,
John
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
 |
Dhruv
Guest
|
| Posted: Sat Jan 31, 2004 10:50 am Post subject: Re: soft memory limit |
|
|
On Fri, 30 Jan 2004 20:41:56 -0500, John Dill wrote:
| Quote: | Is there a (easy) way for me to arbitrarily set the amount of memory
that my program can use to test memory allocation errors? I have a
3GB memory computer for image processing stuff that I develop on and
would like to set a much less arbitrary amount. Can you do this
through making your own allocator/heap and setting new and delete to
interface with that (just for testing purposes)?
|
Try this. It is guaranteed to be non-portable!
#include <iostream>
#include <cstdlib>
#include <functional>
using namespace std;
namespace OP_New { char *Heap_Min = 0; char *Heap_Max = 0; }
void *operator new (size_t _sz) throw (std::bad_alloc)
{
using namespace OP_New;
const size_t Max_Limit = 1024*2; //Heap not greater than 2Kb.
if (!Heap_Min)
{
Heap_Min = (char*)malloc (_sz);
Heap_Max = Heap_Min;
if (!Heap_Min)
throw std::bad_alloc();
else
return (void*)Heap_Min;
}
if (std::less<char*>() (Heap_Max, Heap_Min))
throw std::bad_alloc ();
else
if (Heap_Max - Heap_Min > (ptrdiff_t)Max_Limit)
throw std::bad_alloc ();
else
{
Heap_Max = (char*) malloc (_sz);
return Heap_Max;
}
}
void operator delete (void* _ptr) { free (_ptr); }
int main ()
{
for (int i = 0; i < 100; ++i)
{
int *x = new int[10];
cout<<"i == "<
}
}
Regards,
-Dhruv.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Ivan Vecerina
Guest
|
| Posted: Sat Jan 31, 2004 10:52 am Post subject: Re: soft memory limit |
|
|
"John Dill" <john-dill (AT) uiowa (DOT) edu> wrote
| Quote: | Is there a (easy) way for me to arbitrarily set the amount of memory
that my program can use to test memory allocation errors? I have a
3GB memory computer for image processing stuff that I develop on and
would like to set a much less arbitrary amount. Can you do this
through making your own allocator/heap and setting new and delete to
interface with that (just for testing purposes)?
You can provide your own implementation of operators new, new[], |
delete and delete[] - in both throwing and non-throwing versions.
The standard allocator for containers is required to use these
functions.
A complete list of operator overloads is (std 17.4.3.4/2):
void* operator new(size_t);
void* operator new(size_t, const std::nothrow_t&);
void* operator new[](size_t);
void* operator new[](size_t, const std::nothrow_t&);
void operator delete(void*);
void operator delete(void*, const std::nothrow_t&);
void operator delete[](void*);
void operator delete[](void*, const std::nothrow_t&);
(note that the placement-delete overloads are implicitly
called upon failure of an object's construction that
followed the successful allocation of the matching 'new').
It is simple enough to implement these functions using
malloc/free, to track the total memory size, and to
make the allocation calls fail beyond any threshold.
This will intercept all standard memory allocations,
but not calls to malloc/free or other platform-specific calls.
(caveat: IIRC, operator new(0) shall return a non-null ptr
in C++, so you should call malloc(1) in this case).
Regards,
Ivan
--
http://ivan.vecerina.com/contact/?subject=NG_POST <- e-mail contact form
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Thomas Mang
Guest
|
| Posted: Mon Feb 02, 2004 5:37 pm Post subject: Re: soft memory limit |
|
|
Dhruv schrieb:
| Quote: | Try this. It is guaranteed to be non-portable!
#include <iostream
#include
#include
using namespace std;
namespace OP_New { char *Heap_Min = 0; char *Heap_Max = 0; }
void *operator new (size_t _sz) throw (std::bad_alloc)
{
using namespace OP_New;
const size_t Max_Limit = 1024*2; //Heap not greater than 2Kb.
if (!Heap_Min)
{
Heap_Min = (char*)malloc (_sz);
Heap_Max = Heap_Min;
if (!Heap_Min)
throw std::bad_alloc();
else
return (void*)Heap_Min;
}
if (std::less
throw std::bad_alloc ();
else
if (Heap_Max - Heap_Min > (ptrdiff_t)Max_Limit)
throw std::bad_alloc ();
else
{
Heap_Max = (char*) malloc (_sz);
return Heap_Max;
}
}
void operator delete (void* _ptr) { free (_ptr); }
int main ()
{
for (int i = 0; i < 100; ++i)
{
int *x = new int[10];
cout<<"i == "<
}
}
|
Dhruv, why did you replace the global operator new, when never calling it?
Remember, this expression
int *x = new int[10];
calls operator new[], not operator new.
Maybe you confused them?
regards,
Thomas
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
kanze@gabi-soft.fr
Guest
|
| Posted: Mon Feb 02, 2004 5:48 pm Post subject: Re: soft memory limit |
|
|
[email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote in message
news:<302c79f4.0401300803.17490079 (AT) posting (DOT) google.com>...
| Quote: | Is there a (easy) way for me to arbitrarily set the amount of memory
that my program can use to test memory allocation errors? I have a
3GB memory computer for image processing stuff that I develop on and
would like to set a much less arbitrary amount. Can you do this
through making your own allocator/heap and setting new and delete to
interface with that (just for testing purposes)?
|
If you'll check the test harness stuff at my site ([url]www.gabi-soft.fr)[/url],
you'll find a custom memory allocator for verifying a number of
conditions. It was originally developped because I couldn't afford
Purify for my home site, but one of the features is a "setErrorTrigger"
function which will cause the nth following allocation to fail.
For the moment, there's no support for the nothrow allocators in my
stuff, but if you needed it, it wouldn't be to difficult to add.
--
James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Dhruv
Guest
|
| Posted: Tue Feb 03, 2004 6:44 pm Post subject: Re: soft memory limit |
|
|
On Mon, 02 Feb 2004 12:37:30 -0500, Thomas Mang wrote:
[...]
| Quote: | Dhruv schrieb:
Try this. It is guaranteed to be non-portable!
[...] |
| Quote: | int main ()
{
for (int i = 0; i < 100; ++i)
{
int *x = new int[10];
cout<<"i == "<
}
}
Dhruv, why did you replace the global operator new, when never calling it?
Remember, this expression
int *x = new int[10];
calls operator new[], not operator new.
Maybe you confused them?
|
As I said earlier, 'this is guaranteed to be non-portable!' I assumed
that the final user will:
1. Override the array new operator on his/her own, or:
2. Like libstdc++-v3, array new implicitly calls operator new for it's
memory requests.
Do not simply copy paste the above code, and expect it to work on your
platform/compiler. Please modify it to suit your needs.
Regards,
-Dhruv.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
John Dill
Guest
|
| Posted: Thu Feb 05, 2004 3:30 am Post subject: Re: soft memory limit |
|
|
I understand a bit better about what I need to do, but have a couple
of questions. One thing is if I want to redirect the memory
allocations to a contiguous memory block that I define, would I use
placement new within my operator new calls? I suppose that this might
turn into creating a memory allocator algorithm for finding where to
put my allocated memory. If I don't want it all in a contiguous
memory block, then I think the implementation is pretty simple.
Next, where should I place this memory block? In its own static
class, a singleton, in the global space?
I'm leaning towards encapsulating these soft limit operator new/delete
types in policy classes with static member functions where the class
defines static operator new/delete according to its memory soft limit
strategy. This way I can more easily interchange the global operators
with the different memory soft limit strategies.
John
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
kanze@gabi-soft.fr
Guest
|
| Posted: Thu Feb 05, 2004 3:52 am Post subject: Re: soft memory limit |
|
|
Thomas Mang <a9804814 (AT) unet (DOT) univie.ac.at> wrote
[...]
| Quote: | Dhruv, why did you replace the global operator new, when never calling
it? Remember, this expression
int *x = new int[10];
calls operator new[], not operator new.
|
and the default operator new[] is guaranteed to call operator new().
(See §18.4.1.2/4.)
--
James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
kanze@gabi-soft.fr
Guest
|
| Posted: Thu Feb 05, 2004 11:30 pm Post subject: Re: soft memory limit |
|
|
[email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote in message
news:<302c79f4.0402031443.18df39dd (AT) posting (DOT) google.com>...
| Quote: | I understand a bit better about what I need to do, but have a couple
of questions. One thing is if I want to redirect the memory
allocations to a contiguous memory block that I define, would I use
placement new within my operator new calls?
|
Why? The placement placement new (defined in <new>) is a way of calling
the constructor without allocating memory. In an operator new function,
you shouldn't call the constructor.
| Quote: | I suppose that this might turn into creating a memory allocator
algorithm for finding where to put my allocated memory. If I don't
want it all in a contiguous memory block, then I think the
implementation is pretty simple.
|
If you aren't restricted to a contiguous block, the implementation is
even simpler: malloc() and free().
| Quote: | Next, where should I place this memory block? In its own static
class, a singleton, in the global space?
|
First, what are you trying to do exactly?
If the goal is to test memory allocation errors, an operator new that
generates such an error on command is a better solution than a limited
quantity of memory (which, at least on Unix-like systems, is easiest
simulated by means of a ulinit command before starting the program) --
you can force an error on absolutely any allocation request. The code
to do so is available at my site. (Of all of the code available at my
site, I am convinced that the test harness code, of which MemoryCheck is
a part, is by far the most useful.)
If all you want to do is to impose an artificial upper limit on use, and
for some reason, ulimit (or its equivalent under Windows) is not an
acceptable solution, it is fairly simple to forward to malloc and free,
and simply keep track of the allocated amount in a single, static
size_t.
If the goal is to have all of the memory allocated in a separate,
contiguous block (but why?), then you will have to write your own
allocator for the block.
| Quote: | I'm leaning towards encapsulating these soft limit operator new/delete
types in policy classes with static member functions where the class
defines static operator new/delete according to its memory soft limit
strategy. This way I can more easily interchange the global operators
with the different memory soft limit strategies.
|
And the question is: why? It sounds to me like a blatant case of
premature generization.
--
James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Ivan Vecerina
Guest
|
| Posted: Thu Feb 05, 2004 11:44 pm Post subject: Re: soft memory limit |
|
|
"John Dill" <john-dill (AT) uiowa (DOT) edu> wrote
| Quote: | I understand a bit better about what I need to do, but have a couple
of questions. One thing is if I want to redirect the memory
allocations to a contiguous memory block that I define, would I use
placement new within my operator new calls? I suppose that this might
turn into creating a memory allocator algorithm for finding where to
put my allocated memory. If I don't want it all in a contiguous
memory block, then I think the implementation is pretty simple.
|
Things to be think of when implementing a memory allocator:
- alignment: system performance can be very dependent on proper
alignment of allocated memory, and in a platform-specific
way (could be 4, 8 or 16-bytes on platforms I use)
- malloc/free performance: many approaches exist, each has trade-offs.
This said, doing sub-allocations within a contiguous memory block
isn't all that difficult if you are ok with linear worst-case time
for memory allocation ( constant-time deletion is then possible ).
| Quote: | Next, where should I place this memory block? In its own static
class, a singleton, in the global space?
I'm leaning towards encapsulating these soft limit operator new/delete
types in policy classes with static member functions where the class
defines static operator new/delete according to its memory soft limit
strategy. This way I can more easily interchange the global operators
with the different memory soft limit strategies.
|
For some graphical programming (where a specially allocated memory
range needs to be sub-divided to allocated graphical resources such
as vertex arrays), I have once used a memory allocator very similar
to what you are looking for (on a personal pet project).
I sometimes needed multiple such graphic buffers, so each memory
allocator was an individual class instance.
In case it helps, I have posted two versions of this memory
allocator ( != std::allocator ) here:
http://ivan.vecerina.com/temp/
Note that these files haven't been thoroughly tested, but
I remember them working fine with some test cases.
The SequentialAllocator has worst-case O(N) allocation time,
and works by inserting a memory header in front of each of
its blocks (deletion is always O(1), allocation often is too).
It should work well for what you want to do.
The IndirectAllocator uses a separate std::map to list used
and unused memory segments. If you want to implement
::operator new with this, you would have to modify the
container declarations to use a malloc()-based std::allocator.
(infinite recursion will occur otherwise). About O(lgN) time.
Personally I would use a static instance of either of these
objects in the file where all global new/delete operators are
defined.
Another source of inspiration would be the C library of your
platform (the SequentialAllocator above is inspired by a
circa-1995 implementation of malloc in a Metrowerks compiler).
I hope the linked files will be helpful to you.
Please keep me/us posted on your fixes to them, or whatever
final allocator implementation you use.
Regards,
Ivan
--
http://ivan.vecerina.com/contact/?subject=NG_POST <- e-mail contact form
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Martijn Lievaart
Guest
|
| Posted: Thu Feb 05, 2004 11:53 pm Post subject: Re: soft memory limit |
|
|
On Wed, 04 Feb 2004 22:30:13 -0500, John Dill wrote:
| Quote: | I understand a bit better about what I need to do, but have a couple
of questions. One thing is if I want to redirect the memory
allocations to a contiguous memory block that I define, would I use
placement new within my operator new calls? I suppose that this might
|
You cannot overload operator new, but you can replace the global
allocation function or a class specific one. This directly answers your
question, as these return raw memory. So don't do placement new, just
return a pointer to a block of the requested size.
| Quote: | turn into creating a memory allocator algorithm for finding where to
put my allocated memory. If I don't want it all in a contiguous
memory block, then I think the implementation is pretty simple.
|
It is, just remember to replace both the allocation functions, new and
new[]. And obviously the corresponding de-allocation functions.
| Quote: |
Next, where should I place this memory block? In its own static class,
a singleton, in the global space?
|
It's up to you, but a singleton sounds right to me.
| Quote: | I'm leaning towards encapsulating these soft limit operator new/delete
types in policy classes with static member functions where the class
defines static operator new/delete according to its memory soft limit
strategy. This way I can more easily interchange the global operators
with the different memory soft limit strategies.
|
Makes sense, but in that case also have a look at STL-compatible
allocators. Using them outside of STL containers is ugly, but it is a
standard interface. If your calls to new tend to be encapsulated anyhow,
this is not a great drawback.
It mostly depend if you want to be able to switch allocation policies as a
whole, or at ad-hoc basis. As a whole, then #ifdef would work just as
well, more a matter of style. Ad-hoc, then you cannot (easily) overload
the global allocation function, so you could have a look at placement new
and/or allocators.
Also have a look at "Modern C++ Design", it's treatment of small-object
allocaters, while probably not directly relevant for your problem, has an
in depth discussion about allocation stratedies, implementations and
replacing allocation functions. IIRC Stroustrup goes into consideral
detail as well, but does not cover allocators in enough detail.
HTH,
M4
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
John Dill
Guest
|
| Posted: Fri Feb 06, 2004 12:34 pm Post subject: Re: soft memory limit |
|
|
I have got some basic memory limit stuff working (thanks everyone).
I'm now trying to determine how to incorporate this into my particular
test programs. Should I write global operator new/delete in a header
file which is included for any memory alloc testing that I do?
Also, I would like to apply this not only to the global allocator
operators, but also on a container/class wide basis. I would like to
be able to apply these memory limit ideas to the std containers. It
would be a great test for ensuring my memory allocation errors are
being handled correctly by my classes which use std containers. Is
writing my own allocator the way to go? The main thing will be that
I'll probably have to have a non-singleton version similar to what I
currently have for implementing std container memory limited
allocators.
Many thanks.
John
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Dhruv Matani
Guest
|
| Posted: Fri Feb 06, 2004 12:35 pm Post subject: Re: soft memory limit |
|
|
On Wed, 04 Feb 2004 22:52:32 -0500, kanz wrote:
| Quote: | Thomas Mang <a9804814 (AT) unet (DOT) univie.ac.at> wrote in message
news:<401E3A79.F7B0E0DD (AT) unet (DOT) univie.ac.at>...
Dhruv schrieb:
[...]
Dhruv, why did you replace the global operator new, when never calling
it? Remember, this expression
int *x = new int[10];
calls operator new[], not operator new.
and the default operator new[] is guaranteed to call operator new().
(See §18.4.1.2/4.)
|
Thanks! So, I was unknowingly right! While I thought that there was no
such guarantee.
Regards,
-Dhruv.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Dhruv Matani
Guest
|
| Posted: Fri Feb 06, 2004 12:35 pm Post subject: Re: soft memory limit |
|
|
On Wed, 04 Feb 2004 22:30:13 -0500, John Dill wrote:
| Quote: | I understand a bit better about what I need to do, but have a couple
of questions. One thing is if I want to redirect the memory
allocations to a contiguous memory block that I define, would I use
placement new within my operator new calls?
|
Definitely not! Placement new is well defined to do nothing.
| Quote: | I suppose that this might
turn into creating a memory allocator algorithm for finding where to
put my allocated memory. If I don't want it all in a contiguous
memory block, then I think the implementation is pretty simple.
|
I didn't quite get you here.
| Quote: | Next, where should I place this memory block? In its own static
class, a singleton, in the global space?
|
You mean logically right? because phycally the block will be placed in the
free-store or heap memory or data segment or whatever other name exists.
That's an implementation detail. It's up to you to decide a nice safe
place for the block ;-)
| Quote: | I'm leaning towards encapsulating these soft limit operator new/delete
types in policy classes with static member functions where the class
defines static operator new/delete according to its memory soft limit
strategy. This way I can more easily interchange the global operators
with the different memory soft limit strategies.
|
Looks good to me, but I can't really say what you want to use it for and
how good it will look in the big picture (ie. how well the approach will
scale as time progresses). Some more information would help!
Regards,
-Dhruv.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
John Dill
Guest
|
| Posted: Fri Feb 06, 2004 12:43 pm Post subject: Re: soft memory limit |
|
|
I have implemented something like your example. I also would like to
implement a soft memory limit such that deallocation increased the
memory available so that the total free store is less than a certain
amount. It appears that I am out of luck for implementing this
behavior (just using the global operator new/delete interface).
I can use operator delete( void*, std::size_t ) only at class level
correct? This would mean that I can maintain the memory limit where
deallocation returns available memory back if I point the class
operator new/delete to interface with my memory limiter, but it
wouldn't work for built in types.
It's not too big of a deal, but curious if this could be done on the
global level. As long as I have something to trigger bad_alloc
arbitrarily :)
It seems the best I can do is maintain the count of
allocated/deallocated objects to approximate this soft memory limit.
Best,
John
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
