 |
C++Talk.NET
C++ language newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
Tony
Guest
|
 Posted: Sat Apr 10, 2004 12:25 pm Post subject: How to know what is returned by new |
 |
|
Dear C++ Gurus,
It is well known that new returns the starting address of the "memory"
alloctated. If the required memory is very large and there is not enough
memory, many operation systems will use the virtual memory (hard disk)
instead. I have two questions.
1. Is it possible that a part of the "memory" found by new is
true memory and other part is on hard disk ?
2. How to know the "memory" found by new is all is true memory,
a part is true memory, all is virtual memory ?
My purpose is to have more control on the virtual memory.
Suppose I have a large amount of data need to be stored in memory.
If the memory is not enough, I need to store a part of it on the hard
disk (as file). But I need to know when I should really write file.
To speed up the data processing, I may need to swap the data in memory
and data in the file. If I don't know what the new returns, I cannot
decide when I should use file. If a part or all of "memory" returned
by new is actually virtual memory, the data processing using
virtual memory will dramatically slow down.
Best regards
Tony
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
 |
Francis Glassborow
Guest
|
| Posted: Sat Apr 10, 2004 2:43 pm Post subject: Re: How to know what is returned by new |
|
|
In message <9396fe12.0404091507.22e02da0 (AT) posting (DOT) google.com>, Tony
<tongru_huo (AT) netzero (DOT) com> writes
| Quote: | It is well known that new returns the starting address of the "memory"
alloctated.
|
Let us be absolutely clear about this, operator new returns the start
address of the allocated block of memory, the new expression returns the
address of the constructed object. Those two addresses are not required
to be equal.
| Quote: | If the required memory is very large and there is not enough
memory, many operation systems will use the virtual memory (hard disk)
instead. I have two questions.
|
You are confusing the address space with the actual RAM. Most modern
operating systems use RAM as a temporary resource that is passed around
from process to process. When processes are paged out, or when a process
is not fully resident in RAM some other resource such as a hard disc is
used to save the state of the process.
| Quote: |
1. Is it possible that a part of the "memory" found by new is
true memory and other part is on hard disk ?
|
On modern systems that is a meaningless question. operator new finds a
block of addresses from the address space available to the process. How
that is implemented is outside the scope of C++.
| Quote: |
2. How to know the "memory" found by new is all is true memory,
a part is true memory, all is virtual memory ?
|
Impossible because it may vary from moment to moment.
| Quote: |
My purpose is to have more control on the virtual memory.
|
If you want that you must work with the OS directly.
| Quote: | Suppose I have a large amount of data need to be stored in memory.
If the memory is not enough, I need to store a part of it on the hard
disk (as file). But I need to know when I should really write file.
|
Why? The OS will handle it for you.
| Quote: | To speed up the data processing, I may need to swap the data in memory
and data in the file.
|
Why? Modern OSs do that for you by using virtual memory and paging
things in and out.
| Quote: | If I don't know what the new returns, I cannot
decide when I should use file. If a part or all of "memory" returned
by new is actually virtual memory, the data processing using
virtual memory will dramatically slow down.
|
What is far more important for the purposes of optimisation is the
concept of 'locality' You want to maximise the possibility that all the
data you are currently processing resides in RAM (or even in cache of as
high a level as possible) at the same time. But doing this requires a
detailed understanding of data structures rather than knowledge of where
your data may currently reside.
--
Francis Glassborow ACCU
Author of 'You Can Do It!' see http://www.spellen.org/youcandoit
For project ideas and contributions: http://www.spellen.org/youcandoit/projects
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Bill Wade
Guest
|
| Posted: Mon Apr 12, 2004 6:47 pm Post subject: Re: How to know what is returned by new |
|
|
Francis Glassborow <francis (AT) robinton (DOT) demon.co.uk> wrote
| Quote: | Let us be absolutely clear about this, operator new returns the start
address of the allocated block of memory, the new expression returns the
address of the constructed object. Those two addresses are not required
to be equal.
|
They are required to be equal except when the object is an array.
Combining 3.8p5 with 4.10p2, I believe the following is well defined.
T* tp = new T; // calls operator new(sizeof(T)) (5.3.4p10)
void* vp = tp;
tp->~T();
memset(vp, 2, sizeof(T));
new(vp) T;
delete tp;
If vp were not the result of the call to operator new, the call to
memset() would be UB.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Tony
Guest
|
| Posted: Tue Apr 13, 2004 7:59 am Post subject: Re: How to know what is returned by new |
|
|
The problem I am having is the efficiency. The OS level of paging in
and out
is too general and thus not efficient for me. Because I use new to
allocate memory, I need to know how much memory is available by new.
To allocate memory
is OS level work, but in any way does new have any information about
what "memory" resources it finds ?
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Rakesh
Guest
|
| Posted: Tue Apr 13, 2004 8:03 am Post subject: Re: How to know what is returned by new |
|
|
[email]wade (AT) stoner (DOT) com[/email] (Bill Wade) wrote in message news:<2bbfa355.0404120730.32a73e30 (AT) posting (DOT) google.com>...
| Quote: | Combining 3.8p5 with 4.10p2, I believe the following is well defined.
T* tp = new T; // calls operator new(sizeof(T)) (5.3.4p10)
void* vp = tp;
tp->~T();
I am a little bit confused by the steps that follow here. Can you |
please explain more.
Why do we call the destructor explicitly ? ( How safe is that ? When
are the special occasions when we need to do that ? ) .
| Quote: | memset(vp, 2, sizeof(T));
new(vp) T;
delete tp;
If vp were not the result of the call to operator new, the call to
memset() would be UB.
|
UB ? I am sorry I didn't understand this.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Francis Glassborow
Guest
|
| Posted: Tue Apr 13, 2004 3:37 pm Post subject: Re: How to know what is returned by new |
|
|
In message <9396fe12.0404121038.3fbd1756 (AT) posting (DOT) google.com>, Tony
<tongru_huo (AT) netzero (DOT) com> writes
| Quote: | The problem I am having is the efficiency. The OS level of paging in
and out
is too general and thus not efficient for me. Because I use new to
allocate memory, I need to know how much memory is available by new.
To allocate memory
is OS level work, but in any way does new have any information about
what "memory" resources it finds ?
|
NO, how could it? Paging is entirely in the hands of the OS. Of course
some OSs might provide a special memory allocation of 'allocate and stay
resident' but that would be uncommon these days as it blocks the ability
of the OS to handle multiple processes efficiently.
In ancient times I came across a greedy program which enquired from the
'OS' as to how much memory was available and then took it all. The
program certainly worked more efficiently if it had all the memory it
could lay its hands on, but also had the unfortunate side-effect of
stopping any other new process from starting up. The only way we could
fix the program was by writing another that allocated to itself all but
x Kbytes of the available RAM. The procedure was then to start the
latter program, start the greedy program and then close the memory
holding program.
Hardware gets faster and OSs adapt to use that, good programs generally
stay out of the way and so can work along side each other.
--
Francis Glassborow ACCU
Author of 'You Can Do It!' see http://www.spellen.org/youcandoit
For project ideas and contributions: http://www.spellen.org/youcandoit/projects
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Bill Wade
Guest
|
| Posted: Tue Apr 13, 2004 4:17 pm Post subject: Re: How to know what is returned by new |
|
|
We're drifting from the original poster's question (dealing with
virtual memory (VM) management). As FG pointed out, neither the C++
language, nor the standard library provide any direct access to
whatever VM system may or may not be present.
FG made the strictly true statement that the void* address returned by
operator new was not necessarily the same [I would say does not
compare equal to] the T* address returned by the new expression.
I wanted to clarify by saying that FG's statement was only true in the
case of an array new. My example intended to show code that was well
defined, according to the standard, but would have undefined behavior
if operator new and the new expression were allowed to return
different addresses for a non-array object.
Now on to your specific questions:
| Quote: | Why do we call the destructor explicitly?
|
For the specific example, my intent was to use the object's memory (by
stepping on it with memset()) after the life of the object was over
(it is certainly over after the explicit call to the destructor).
In general it is very dangerous. Once the destructor has been called,
you are no longer allowed to treat the memory as much more than a
character array. However you may still convert a pointer to the old
object down to a void*. In particular if an implicit call to the
destructor occurs (perhaps because you called delete, or an auto
variable went out of scope) the behavior is undefined.
| Quote: | When are the special occasions when we need to do that ? ) .
|
This is must commonly done for container classes (in std::, this is
usually delegated to allocators). If I have a container (perhaps a
vector) with five "living" objects, and then change the size to four,
I might want to keep the same memory, but destroy the fifth element
with an explicit call to its destructor.
Usually (as in my example), explicit calls to destructors are balanced
by calls to placement new. In most cases (not in my example) the call
to placement new occurs before the explicit call to the destructor.
| Quote: | UB ? I am sorry I didn't understand this.
|
Undefined Behavior
Regards
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Michiel Salters
Guest
|
| Posted: Wed Apr 14, 2004 6:17 am Post subject: Re: How to know what is returned by new |
|
|
[email]dreamzlion (AT) yahoo (DOT) com[/email] (Rakesh) wrote in message news:<3e744c46.0404121437.59c4672e (AT) posting (DOT) google.com>...
| Quote: | wade (AT) stoner (DOT) com (Bill Wade) wrote in message news:<2bbfa355.0404120730.32a73e30 (AT) posting (DOT) google.com>...
Combining 3.8p5 with 4.10p2, I believe the following is well defined.
T* tp = new T; // calls operator new(sizeof(T)) (5.3.4p10)
void* vp = tp;
tp->~T();
I am a little bit confused by the steps that follow here. Can you
please explain more.
Why do we call the destructor explicitly ? ( How safe is that ? When
are the special occasions when we need to do that ? ) .
|
The code above isn't trying to do anything sensible. It is code to test
a compiler. The reason this is done is to get access to the raw memory
used by the T object. For instance, a std::string object may hold an
std::string_implementation object or when the string is small, reuse
that storage to store the characters directly. std::string operator=
may then need to destroy the string_implementation object without
releasing its memory, and thus call impl->~string_implementation();
This would be a typical case of advanced memory management, where these
calls are usually found. You rarely need it directly, because you can
often use the more friendly std:: classes to manage memory.
| Quote: | memset(vp, 2, sizeof(T));
new(vp) T;
delete tp;
If vp were not the result of the call to operator new, the call to
memset() would be UB.
UB ? I am sorry I didn't understand this.
|
Undefined behavior. The technical term for a class of very bad things.
In this case, writing to memory which isn't properly allocated would
be Undefined Behavior. Because we know that the memset must be legal,
we know that vp must have been properly allocated by the first 'new'.
Regards,
Michiel Salters
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Derek
Guest
|
| Posted: Wed Apr 14, 2004 6:31 am Post subject: Re: How to know what is returned by new |
|
|
Tony wrote:
| Quote: | My purpose is to have more control on the virtual memory.
Suppose I have a large amount of data need to be stored
in memory. If the memory is not enough, I need to store
a part of it on the hard disk (as file). But I need
to know when I should really write file. To speed up
the data processing, I may need to swap the data in
memory and data in the file. If I don't know what the
new returns, I cannot decide when I should use file. If
a part or all of "memory" returned by new is actually
virtual memory, the data processing using virtual memory
will dramatically slow down.
|
In short, standard C++ has no way to query the OS about
anything, let alone tell it how to do its job. I work
with applications that work on datasets much larger than
available memory and I write my own paging code. After
all, I understand my application's memory use patterns
better than any compiler or operating system.
As has already been said, it's important to organize your
application to maximize locality.
To give you an example, my application typically needs
to access thousands of images. But I happen to know it
only needs a few pieces of a few images at any one time.
Consequently I write my code to take advantage of this
instead of reading thousands of images into memory and
hoping the OS can page to disk intelligently. I can do a
much better job and keep my working dataset down to a few
MB out of 200GB, even though my images are being accessed
at random (though coherently).
Your OS may have system-specific calls to help you determine
the overall state of memory, like how much physical memory
is available for your process, but this is outside the scope
of standard C++.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Tony
Guest
|
| Posted: Wed Apr 14, 2004 6:33 am Post subject: Re: How to know what is returned by new |
|
|
Let us take extremely very large linear equation solution as an
example.
The matrix is huge, the memory is not able to hold it.
In almost all algorithms, at a certrain time only a part of the matrix
enters the operation. So the whole matrix is huge, but at any time
only
a small part of the matrix needs to be in memory. (If not, we cannot
solve
the huge equation at all.) For the efficiency, it is best to keep the
matrix as large as possible in memory. The getting in and out of the
memory
also needs a lot of dedicate considerations. The OS level paging
doesn't
meet the requirement at all. If it does, the huge equation solution
will be
an easy business.
I need to know how much memory can be found by new. Then I can decide
how much data I can feed in memory and how much I have to feed in
virtual memory, and also how to swap data between memory and virtual
memory.
Regards
Tony
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Andrea Griffini
Guest
|
| Posted: Wed Apr 14, 2004 6:40 am Post subject: Re: How to know what is returned by new |
|
|
On 13 Apr 2004 11:37:36 -0400, Francis Glassborow
<francis (AT) robinton (DOT) demon.co.uk> wrote:
| Quote: | Hardware gets faster and OSs adapt to use that, good programs generally
stay out of the way and so can work along side each other.
|
Good hardware killed good software, but even
nowdays the OS doesn't really have enough information
about what you want to do with the memory to pretend
it's going to do the best possible job by just blindly
paging out to disk.
Let alone that not long ago the most diffused OS
on PC had such a terrible virtual memory management
that once the heavy swapping started there was no
way but a cold reboot to get the system back to an
usable state (no, freeing the memory and killing the
processes that allocated it is not enough... the
system just keeps swapping and swapping forever)
So if you wanna say that you can't know in C++
then fine... but please don't pretend it's not
needed; IMO that's a very poor argument.
Andrea
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Clem Dickey
Guest
|
| Posted: Wed Apr 14, 2004 4:37 pm Post subject: Re: How to know what is returned by new |
|
|
[email]tongru_huo (AT) netzero (DOT) com[/email] (Tony) wrote in message news:<9396fe12.0404091507.22e02da0 (AT) posting (DOT) google.com>...
| Quote: | 2. How to know the "memory" found by new is all is true memory,
a part is true memory, all is virtual memory ?
|
Nothing to do with C++, but there are a few ways to determine "true"
memory:
1. In BSD, Linux or MacOS X, use the mincore function to determine
whether a page is in core. As others have said, this can change over
time.
2. Turn off paging. For linux the "swapoff" command will do this. Then
all memory will be "true" memory.
3. Write a device driver which can "pin" memory pages, i.e. it
instructs the OS to keep them in "true" memory.
Finally, since speed is a concern, you could stop allocating memory
when your program began to run too slowly. Use the C++ library
functions time() and difftime() to determine that.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Michiel Salters
Guest
|
| Posted: Thu Apr 15, 2004 12:51 am Post subject: Re: How to know what is returned by new |
|
|
[email]tongru_huo (AT) netzero (DOT) com[/email] (Tony) wrote in message news:<9396fe12.0404121038.3fbd1756 (AT) posting (DOT) google.com>...
| Quote: | The problem I am having is the efficiency. The OS level of paging in
and out is too general and thus not efficient for me. Because I
use new to allocate memory, I need to know how much memory is
available by new. To allocate memory is OS level work, but in
any way does new have any information about what "memory"
resources it finds ?
|
The default new is part of the implementation, so it internally knows
what it gets. Usually, this will be OS-managed virtual memory which
is dynamically swapped between disk and RAM, depending on the load.
It is possible to provide your own operator new, which allocates
memory from other sources (e.g. Non-paged pool of your OS). However,
developing this will take time. With current 64-bit systems and
memory prices, throwing hardware at the problem is often sufficient.
In addition, since usually only a fraction of the memory is
non-pageable, allocating only non-pageable memory will mean that
you can't use all the RAM in your machine. This will slow things.
I've had similar problems, but since my program needed to run on 1
PC only I could simply buy 2Gb and create an optimized string class
(my strings were <255 chars, with 50% <7 chars, and I had ~60 million)
Sure, if I started any other program, this would effectively block
progress, but since the PC was dedicated to this task I didn't care.
Regards,
Michiel Salters
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Dave Boyle
Guest
|
| Posted: Thu Apr 15, 2004 6:43 am Post subject: Re: How to know what is returned by new |
|
|
<snip>
| Quote: | So if you wanna say that you can't know in C++
then fine... but please don't pretend it's not
needed; IMO that's a very poor argument.
|
<snip>
The OP requires functionality that is at variance with a fundamental
principle of multi-tasking OS's. It is not up to a process to control
the allocation of resources that it requires, that is the job of the
OS.
Imagine 'new' could return some kind of information related to how
much free physical memory there was. Where would that get us? In a
multi-tasking OS it gets us nowhere as the OS can page out a load of
RAM to the page file 10 millseconds after a 'new' call from your
process. What if the process could 'pin' memory in RAM the way some
OS's kernal code uses a non-paged pool of memory? This would
obviously negatively impact other processes that required memory (as
Francis has pointed out).
In short, you may think it useful to have this facility but it simply
isn't possible in most multi-tasking OS's and most programmers (I
would guess) dislike the idea of some user process being able to
effectively disable paging for some arbitrary amount of RAM.
Cheers,
Dave
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Tony
Guest
|
| Posted: Thu Apr 15, 2004 6:54 am Post subject: Re: How to know what is returned by new |
|
|
To keep the complete data, I need to have HUGE bytes of memory.
The actual memory may be smaller than HUGE. So to be on the
concervative side,
T *part_of_data = new T [0.1*HUGE]; //I bet all of this in actual memory.
//This is the best I can do.
I need to keep 0.9*HUGE of data on hard disk.
In short, 0.1*HUGE (the actual memory available) is by guess or some
other system call (out side of C++ new). This is the best I can do, right?
Regards
Tony
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
