C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Using vector<unsigned char> as raw memory Goto page 1, 2, 3, 4, 5  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message kanze@gabi-soft.fr Guest Posted: Thu Feb 05, 2004 11:45 am    Post subject: Using vector<unsigned char> as raw memory Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). -- James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr Conseils en informatique orientée objet/ http://www.gabi-soft.fr Beratung in objektorientierter Datenverarbeitung 11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Thomas Mang Guest Posted: Fri Feb 06, 2004 10:15 am    Post subject: Re: Using vector<unsigned char> as raw memory [email]kanze (AT) gabi-soft (DOT) fr[/email] schrieb: Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) Does the Standard somewhere require vector to obtain its memory always via operator new? Wouldn't it be possible that class vector, especially when optimized for small classes (say, the built-in types), reserves internally some raw memory for a limited number of T-objects, to avoid allocating memory from the heap (the compiler could, of course, easily find out how this raw memory would have to be aligned at instantiation time)? I remember having heard of a string implementation (Intels?) that was optimized for small strings and had a charT[16] - array as data member, exactly to avoid operator new. Couldn't the same issue apply to vector? In case this is true, then I think your question can be easily answered that there is no alignment guarantee. regards, Thomas [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Joe Guest Posted: Fri Feb 06, 2004 10:31 am    Post subject: Re: Using vector<unsigned char> as raw memory I am no expert on the STL or the thorny issues of data alignment, but are you sure that vector<T> will use new to obtain memory for data storage? Although I know of no implementation that does, could not vector<T> (like some string implementations) use some internal space to store "T" for small vector<T> sizes and then use new as needed for larger sizes. Would be a faster implementation for small vector<T> sizes. In addition, I seem to remember a post some time of go about a standards proposal that would require all elements of a vector<T> to occupy contiguous memory locations -- thus implying that the current standard does not require it. Although I strongly doubt that any implementation would not have contiguous elements. Joe <kanze (AT) gabi-soft (DOT) fr> wrote Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Hendrik Belitz Guest Posted: Fri Feb 06, 2004 12:31 pm    Post subject: Re: Using vector<unsigned char> as raw memory [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. For the most implementations of std::vector you will get the desired behaviour. But since the standard not forces a specific implementation of std::vector, this cannot be guaranteed. Also using another allocator for the vector may change this behaviour too. -- To get my real email adress, remove the two onkas -- Dipl.-Inform. Hendrik Belitz Central Institute of Electronics Research Center Juelich [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bogdan Guest Posted: Fri Feb 06, 2004 12:32 pm    Post subject: Re: Using vector<unsigned char> as raw memory [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote in message news:<d6652001.0402040847.49b9669 (AT) posting (DOT) google.com>... Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). I don't know what standard says about &v[0]. Maybe you could use std::basic_string<unsigned char>. I know that data() return a const pointer to an array owned by the string, but if no non-const methods are called on the string that array could be used. Anyway, in this case maybe it worth to write a simplified version of boost::scoped_array Best regards, Bogdan Sintoma [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top tom_usenet Guest Posted: Fri Feb 06, 2004 12:32 pm    Post subject: Re: Using vector<unsigned char> as raw memory On 5 Feb 2004 06:45:05 -0500, [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) Looking at 20.4.1.1, std::allocator<T>::allocate allocates memory suitably aligned for T, so memory returned by std::allocator<unsigned char>::allocate is only guaranteed to have an alignment of 1. In addition, vector doesn't necessarily have to use the memory returned by the allocator directly. e.g. one could envisage a specialization of vector for unsigned char that stored the size of the vector at the start of the allocation (using knowledge of std::allocator to determine the capacity). e.g. template<> class vector<unsigned char, std::allocator { unsigned char* m_data; //only data member, never null public: //... size_type size() const { return *reinterpret_cast<size_type*>(m_data); } size_type capacity() const { //using knowledge of operator new. return *reinterpret_cast<size_t*>(m_data - 4) - sizeof(size_type); } reference operator[](size_type t) { return m_data[t + sizeof(size_type)]; } }; So even if std::allocator returned fully aligned memory, vector<unsigned char> could make v[0] unaligned. Tom C++ FAQ: http://www.parashift.com/c++-faq-lite/ C FAQ: http://www.eskimo.com/~scs/C-faq/top.html [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Vladimir Kouznetsov Guest Posted: Fri Feb 06, 2004 12:40 pm    Post subject: Re: Using vector<unsigned char> as raw memory <kanze (AT) gabi-soft (DOT) fr> wrote Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) I think to be on the safe side you better to implement your own allocator. There are no any requirements for the standard allocator on how data are laid out inside the ::operator new() allocated storage except they should be T-aligned. But I believe you know that already. Quote: James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr thanks, v [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Balog Pal Guest Posted: Fri Feb 06, 2004 12:42 pm    Post subject: Re: Using vector<unsigned char> as raw memory <kanze (AT) gabi-soft (DOT) fr> wrote Quote: For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). Come on, James do you say you can't write a wrapper to a raw memblock handler class in 20 minutes? If you'd use scoped_array or some other boost class unless your compiler ptoblems, you can just delete the template<> part and write a typedef 2 lines below it -- creating a nontemplate instance with the same functionality. Why not do it the way it worked 10 years back if it does the job? Paul [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Chris Theis Guest Posted: Fri Feb 06, 2004 7:34 pm    Post subject: Re: Using vector<unsigned char> as raw memory "Thomas Mang" <a9804814 (AT) unet (DOT) univie.ac.at> wrote Quote: [email]kanze (AT) gabi-soft (DOT) fr[/email] schrieb: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) Does the Standard somewhere require vector to obtain its memory always via operator new? The containers allocate the memory via the allocators and the implementation of the default allocator uses global new & delete. However, AFAIK the standard does not require other optimized/user implemented allocators to do so. Til now I assumed that this was the idea behind the allocator approach, so that this is not necessary. [SNIP] Chris [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Maciej Sobczak Guest Posted: Fri Feb 06, 2004 7:58 pm    Post subject: Re: Using vector<unsigned char> as raw memory Hi, [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: Is it possible to use std::vector< unsigned char > as raw memory? I cannot answer your specific question (I can only share your assumptions that it is OK), but there is a place for spinning off a related question. Why do you want to use unsigned char as the underlying type? Is it any better than plain char when used as "raw memory" (where by "raw memory" I mean that the only later use will involve reinterpret casts or object copy). Consider: 3.9/2 states that POD can be copied back and forth using array of char OR unsigned char, preserving its value. 3.9.1/1: "A char [...] AND unsigned char [...] have the same object representation." There are other places where such properties are defined to be the same for both char and unsigned char. The only relevant place that shows some assymetry is 3.9/4: "The object representation of an object of type T is the sequence of N unsigned char objects taken up by the object of type T, where N equals sizeof(T)." This would state that unsigned char is better than plain char for "raw memory" uses, but somehow I cannot believe it due to the 3.9.1/1 cited above. It was my habit to use unsigned char, but I resigned from it and now consequently use char buffers when the "raw memory" is what I need. I just found it more consistent with various API functions, where pointer to char is expected as a buffer parameter. I will be glad to know your opinion on this. -- Maciej Sobczak : http://www.msobczak.com/ Programming : http://www.msobczak.com/prog/ [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Hendrik Belitz Guest Posted: Fri Feb 06, 2004 8:12 pm    Post subject: Re: Using vector<unsigned char> as raw memory Thomas Mang wrote: Quote: [email]kanze (AT) gabi-soft (DOT) fr[/email] schrieb: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) Does the Standard somewhere require vector to obtain its memory always via operator new? Wouldn't it be possible that class vector, especially when optimized for small classes (say, the built-in types), reserves internally some raw memory for a limited number of T-objects, to avoid allocating memory from the heap (the compiler could, of course, easily find out how this raw memory would have to be aligned at instantiation time)? I remember having heard of a string implementation (Intels?) that was optimized for small strings and had a charT[16] - array as data member, exactly to avoid operator new. Couldn't the same issue apply to vector? In case this is true, then I think your question can be easily answered that there is no alignment guarantee. std::vector uses an allocator to obtain memory. So even if the standard vector does not call new, you're always able to write your own allocator and use it with the vector class without major modifications of your code. -- To get my real email adress, remove the two onkas -- Dipl.-Inform. Hendrik Belitz Central Institute of Electronics Research Center Juelich [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Fri Feb 06, 2004 8:16 pm    Post subject: Re: Using vector<unsigned char> as raw memory In message <d6652001.0402040847.49b9669 (AT) posting (DOT) google.com>, [email]kanze (AT) gabi-soft (DOT) fr[/email] writes Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). what I do not understand is why you do not just use operator new direct: unsigned char * ptr = (unsigned char) operator new(requirement); -- Francis Glassborow ACCU Author of 'You Can Do It!' see http://www.spellen.org/youcandoit For project ideas and contributions: http://www.spellen.org/youcandoit/projects [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Graeme Prentice Guest Posted: Fri Feb 06, 2004 8:18 pm    Post subject: Re: Using vector<unsigned char> as raw memory On 5 Feb 2004 06:45:05 -0500, [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) It does guarantee it. As you probably know, 20.4.1.1 requires operator new(size_t) to be used by the default allocator. 3.7.3.1 para 2 requires the address returned by new to be suitably aligned so that it can be converted to a pointer of any complete object type. You also probably know that you can't use v[0] until there is at least one element in the vector even if memory has been allocated/reserved for one or more elements. One more thing, for the memory returned by operator new, the address of one past the end of that memory has to be a valid address because new has to allow for the fact that the memory might be being used for an array, which requires one past the end to be a valid address. Graeme [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Graeme Prentice Guest Posted: Fri Feb 06, 2004 8:21 pm    Post subject: Re: Using vector<unsigned char> as raw memory On 5 Feb 2004 06:45:05 -0500, [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: Is it possible to use std::vector< unsigned char > as raw memory? Or, more specifically, if I have a std::vector< unsigned char > v, is &v[0] guaranteed to be aligned for all possible data types. (Since an implementation is required ultimately to use operator new to obtain the buffer, I can't see how it couldn't be in practice, but I rather doubt that the standard gives me this guarantee, even indirectly. But maybe this was the intent.) I take back what I said in another reply to this (which may or may not have turned up yet). Even though operator new is required to be used, allocator<T> returns memory suitably aligned for an object of type T only, so if the default allocator breaks up the memory returned by operator new (which it's allowed to do) then there's no alignment guarantee for non char objects. Graeme [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dhruv Matani Guest Posted: Fri Feb 06, 2004 8:24 pm    Post subject: Re: Using vector<unsigned char> as raw memory On Thu, 05 Feb 2004 06:45:05 -0500, kanz wrote: Quote: For those who are curious: certain Posix functions require some very strange memory tricks. The second parameter of readdir_r is the one that's giving me the problems -- and the actual size needed isn't known until runtime, because it depends on the filesystem where the directory is hosted. So I need to allocate dynamcally, and I need RAII (and my compiler is too old to support Boost, so scoped_array isn't an option). What about std::auto_ptr<>? Regards, -Dhruv. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2, 3, 4, 5  Next Page 1 of 5   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.