C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Packing structs into untyped memory with sizeof() Goto page 1, 2  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message NJS Guest Posted: Tue Oct 21, 2003 3:21 pm    Post subject: Packing structs into untyped memory with sizeof() Hello, I'm trying to pack a chunk of raw, allocated memory with a bunch of structs of different types. I'm a bit concerned about alignment issues, and I'm wondering if sizeof() will always return the aligned-size of the structure. If I'm interpreting the ARM (5.3.2) correctly, the answer is yes: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing such objects in an array." So given the starting memory address of a chunk of recently allocated memory, foo, I would pack the structs as follows: A starts at foo B starts at foo + sizeof( A ) C starts at foo + sizeof( A ) + sizeof( B ) This general concept should work, right? - Niek Sanders [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bo Persson Guest Posted: Wed Oct 22, 2003 1:28 am    Post subject: Re: Packing structs into untyped memory with sizeof() "NJS" <njs8030 (AT) yahoo (DOT) com> skrev i meddelandet news:c57a219f.0310200654.2e0d9a8c (AT) posting (DOT) google.com... Quote: Hello, I'm trying to pack a chunk of raw, allocated memory with a bunch of structs of different types. I'm a bit concerned about alignment issues, and I'm wondering if sizeof() will always return the aligned-size of the structure. If I'm interpreting the ARM (5.3.2) correctly, the answer is yes: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing such objects in an array." So given the starting memory address of a chunk of recently allocated memory, foo, I would pack the structs as follows: A starts at foo B starts at foo + sizeof( A ) No, there is no guarantee that B's alignment has anything to do with sizeof(A). B could start at foo, foo + sizeof(B), etc. Quote: C starts at foo + sizeof( A ) + sizeof( B ) No. Quote: This general concept should work, right? Wrong. Bo Persson [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Antoun Kanawati Guest Posted: Wed Oct 22, 2003 1:29 am    Post subject: Re: Packing structs into untyped memory with sizeof() NJS wrote: Quote: Hello, I'm trying to pack a chunk of raw, allocated memory with a bunch of structs of different types. I'm a bit concerned about alignment issues, and I'm wondering if sizeof() will always return the aligned-size of the structure. If I'm interpreting the ARM (5.3.2) correctly, the answer is yes: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing such objects in an array." So given the starting memory address of a chunk of recently allocated memory, foo, I would pack the structs as follows: A starts at foo B starts at foo + sizeof( A ) C starts at foo + sizeof( A ) + sizeof( B ) This general concept should work, right? No, it wouldn't. The correct interpretation is: given a correctly aligned starting address A for type X, then A+sizeof(X) will also be correctly aligned. However, if the starting address is not aligned, there is no guarantee that adding one sizeof(X) will align it. In your example, there is no guarantee that foo is aligned to any suitable restrictions, and even if it were, there is no guarantee that foo+sizeof(A) will align to accomodate B, and so on. What you need to do is create structs of the following form: struct Foo { char _t; YourType x; }; and then study the distance between Foo: and Foo::_t; that will give you an idea about the alignment contraints. Alternatively, you can what malloc() does, and align on the strictest limit (usually double). Generally, the first member of a plain-data struct sits at offset 0, and padding is inserted between members and the tail of the struct. So, you usually have to worry about aligning the first member of the struct (apply rule recursively if the first member is also a struct type). However, about 15 years ago, I ran into code where some programmer felt that the base address of a struct was &foo.firstmember, and wrote all his code accordingly. The reason was not explained anywhere. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Ulrich Eckhardt Guest Posted: Wed Oct 22, 2003 4:05 pm    Post subject: Re: Packing structs into untyped memory with sizeof() NJS wrote: Quote: So given the starting memory address of a chunk of recently allocated memory, foo, I would pack the structs as follows: A starts at foo B starts at foo + sizeof( A ) C starts at foo + sizeof( A ) + sizeof( B ) Apart from what was already mentioned (i.e. that the above deosn't fullfill alignment requirements), consider this trick: struct tmp { A anA; B aB; C aC; }; Here you have them suitably laid out in memory. Uli -- Questions ? see C++-FAQ Lite: http://parashift.com/c++-faq-lite/ first ! [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Stephen Howe Guest Posted: Wed Oct 22, 2003 6:37 pm    Post subject: Re: Packing structs into untyped memory with sizeof() Quote: A starts at foo B starts at foo + sizeof( A ) C starts at foo + sizeof( A ) + sizeof( B ) This general concept should work, right? No. Imagine you have struct like so struct foobar { A m1; B m2; C m2; }; The compiler could arrange that there is padding between m1 and m2 and between m2 and m3 in order that m2 and m3 start on a appropriate boundary. For example sizeof(A) could be odd and it may be important that B starts on an even boundary. Padding at the end of A would be required to achieve that. Stephen Howe Quote: - Niek Sanders [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top NJS Guest Posted: Thu Oct 23, 2003 2:32 pm    Post subject: Re: Packing structs into untyped memory with sizeof() This seems like the best general approach, but I don't think it will work in my specific case. [snip] Quote: struct foobar { A m1; B m2; C m2; }; The compiler could arrange that there is padding between m1 and m2 and between m2 and m3 in order that m2 and m3 start on a appropriate boundary. [snip] I know that I won't have to pack any struct more than once, but I don't know what I'm packing for each object until runtime. I suppose I could brute-force enumerate all the possible struct combinations and do a sizeof() for each, but that's 2^n cases. In this case, n is currently around five, with the potential of expanding to perhaps fifteen or so in the future. I would really like to pack the structs into one contiguous chunk of heap allocated memory. In the worse case, I could guarantee portability by using a linked list structure with each struct in its own chunk of heap memory, and have a pointer to the next struct. This way I could let the OS worry about the alignment issues. OBJ -> 1st struct -> 2nd struct -> 3rd struct That leaves me with one pointer in the object, where I'm most concerned about space bloat, so it's not an unreasonable approach. However, I'm in a performance situation where I would like to keep the structs together if at all possible. - Niek [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Le Chaud Lapin Guest Posted: Fri Oct 24, 2003 3:19 pm    Post subject: Re: Packing structs into untyped memory with sizeof() [email]njs8030 (AT) yahoo (DOT) com[/email] (NJS) wrote in message news:<c57a219f.0310200654.2e0d9a8c (AT) posting (DOT) google.com>... Quote: Hello, I'm trying to pack a chunk of raw, allocated memory with a bunch of structs of different types. I'm a bit concerned about alignment issues, and I'm wondering if sizeof() will always return the aligned-size of the structure. If I'm interpreting the ARM (5.3.2) correctly, the answer is yes: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing such objects in an array." So given the starting memory address of a chunk of recently allocated memory, foo, I would pack the structs as follows: A starts at foo B starts at foo + sizeof( A ) C starts at foo + sizeof( A ) + sizeof( B ) This general concept should work, right? http://lists.debian.org/debian-gcc/2001/debian-gcc-200104/msg00102.html According to that link, the compiler supplies two facilities for tight packing, one for intra-structure, and the other for inter-structure, and #pragma pack will only handle intra-structure packing. To get inter-structure packing [finding true size using sizeof()], it seems that you have to write: __attribute__ ((packed)) // on a per structure basis However, if you are unable to modify the declarations of the structures, there might be a second solution depending on how tightly the compiler packs aggregates form through inheritance. If #pragma packs such aggregates tightly, then you could form apply a 1-member char-type wrapper around your structure and caculate the offset of the the member from the base of the aggregate. Then you could substitute this template in place of sizeof(): template <typename X> struct Sizer : X { char last_member; // Hopefully packed tightly against inherited struct static unsigned int size () { return (unsigned) &reinterpret_cast<Sizer *>(0)->last_member; } } ; struct Foo { unsigned int a; unsigned int b; char c; } foo; int main () { cout << "The size of a Foo is " << Sizer return 0; } Note that I left out the #pragma directive (don't know syntax) -Chaud Lapin- [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dhruv Guest Posted: Tue Nov 04, 2003 5:10 pm    Post subject: Re: Packing structs into untyped memory with sizeof() On Tue, 21 Oct 2003 11:21:22 -0400, NJS wrote: Well, the general rule would be to place the member with the largest size first, and then place objects in decreasing order of size, which in most cases results in optimally aligned data variables. This is by no means a rule, and you do have cases which do not atisfy this requirement, and which occupy greater space when layed out like this, for example: char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. So, I decided to write some code that would create a structure that aligns members like how I just mentioned. I ran into quite a few problems myself, and without the help of others, it probably would have still not been working. You can find the code in the thread "What's wrong here (Templates)". Here is the final code, which should work as expected: #include <iostream> template <class T1, class T2> struct same_type { static const bool result = false; }; template <class T1> struct same_type<T1, T1> { static const bool result = true; }; template <bool Condition, class FT, class ST> struct IF_ { typedef ST type; }; template <class FT, class ST> struct IF_<true, FT, ST> { typedef FT type; }; template <class A, class B> struct smaller_2 { typedef typename IF_< (sizeof(A) < sizeof(B)), A, B>::type type; }; template <class A, class B> struct larger_size_2 { typedef typename IF_< (sizeof(A) > sizeof(B)), A, B>::type type; }; template <class A, class B, class C> struct larger_size_3 { typedef typename IF_< (sizeof (typename larger_size_2 sizeof (C)), typename larger_size_2<A, B>::type, C>::type type; }; template <class A, class B, class C> struct second_largest_3 { typedef typename larger_size_3 <A, B, C>::type Exclude; typedef typename IF_ <same_type ::type first_type; typedef typename IF_ <same_type ::type, C>::type second_type; typedef typename larger_size_2 <first_type, second_type>::type type; }; template <class A, class B, class C> struct smallest_3 { typedef typename smaller_2 <typename second_largest_3 typename second_largest_3 <A, B, C>::first_type>::type type; }; template <class A, class B, class C> struct foo { typename larger_size_3 <A, B, C>::type var1; typename second_largest_3 <A, B, C>::type var2; typename smallest_3 <A, B, C>::type var3; // foo() { } foo () { var1 = "dhruv"; var2 = "dd"; var3 = "445"; } }; struct Foo {}; struct bar {}; struct hhh {}; struct Ua { }; struct Ub { }; struct Uc { }; struct Ud { int d; }; struct Ue { int e; }; int main () { foo <Foo, bar, hhh> ofoo; foo<Ua, Ub, Uc> a; foo<Ua, Ub, Ud> b; foo<Ua, Ud, Uc> c; foo<Ud, Ub, Uc> d; foo<Ua, Ud, Ue> d; foo<Ud, Ub, Ue> e; foo<Ud, Ue, Uc> f; } Regards, -Dhruv. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Raoul Gough Guest Posted: Fri Nov 07, 2003 2:30 pm    Post subject: Re: Packing structs into untyped memory with sizeof() "Dhruv" <dhruvbird (AT) gmx (DOT) net> writes: Quote: On Tue, 21 Oct 2003 11:21:22 -0400, NJS wrote: Well, the general rule would be to place the member with the largest size first, and then place objects in decreasing order of size, which in most cases results in optimally aligned data variables. This is by no means a rule, and you do have cases which do not atisfy this requirement, and which occupy greater space when layed out like this, for example: char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. I don't think the ordering makes any difference in this case, since sizeof() must include any tail-padding necessary for storing the object in an array. Certainly my compiler gives identical sizes for structs with either of the two orderings you mentioned: #include <cstdio> struct foo { char a[5]; int x; int y; }; struct bar { int x; int y; char a[5]; }; int main () { printf ("%u %un", sizeof (foo), sizeof(bar)); } my output: 16 16 -- Raoul Gough. (setq dabbrev-case-fold-search nil) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Francis Glassborow Guest Posted: Sat Nov 08, 2003 1:00 pm    Post subject: Re: Packing structs into untyped memory with sizeof() In article <uwuadcoz1.fsf (AT) yahoo (DOT) co.uk>, Raoul Gough <RaoulGough (AT) yahoo (DOT) co.uk> writes Quote: char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. I don't think the ordering makes any difference in this case, since sizeof() must include any tail-padding necessary for storing the object in an array. Certainly my compiler gives identical sizes for structs with either of the two orderings you mentioned: #include struct foo { char a[5]; int x; int y; }; struct bar { int x; int y; char a[5]; }; int main () { printf ("%u %un", sizeof (foo), sizeof(bar)); } However: struct foo { char a[5]; int x; int y; char c }; struct bar { int x; int y; char a[5]; char c }; would often be different sizes. If size matters, then you may need more care and platform specific arrangement than if it is irrelevant. -- Francis Glassborow ACCU If you are not using up-to-date virus protection you should not be reading this. Viruses do not just hurt the infected but the whole community. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dhruv Guest Posted: Sat Nov 08, 2003 1:33 pm    Post subject: Re: Packing structs into untyped memory with sizeof() On Fri, 07 Nov 2003 09:30:12 -0500, Raoul Gough wrote: Quote: "Dhruv" <dhruvbird (AT) gmx (DOT) net> writes: On Tue, 21 Oct 2003 11:21:22 -0400, NJS wrote: Well, the general rule would be to place the member with the largest size first, and then place objects in decreasing order of size, which in most cases results in optimally aligned data variables. This is by no means a rule, and you do have cases which do not atisfy this requirement, and which occupy greater space when layed out like this, for example: char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. I don't think the ordering makes any difference in this case, since sizeof() must include any tail-padding necessary for storing the object in an array. Certainly my compiler gives identical sizes for structs with either of the two orderings you mentioned: #include struct foo { char a[5]; int x; int y; }; struct bar { int x; int y; char a[5]; }; int main () { printf ("%u %un", sizeof (foo), sizeof(bar)); } my output: 16 16 Again, this padding thing varies form compiler to compiler, and there is nothing standard about it, so it's alright... Regards, -Dhruv. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Raoul Gough Guest Posted: Sun Nov 09, 2003 10:51 pm    Post subject: Re: Packing structs into untyped memory with sizeof() "Dhruv" <dhruvbird (AT) gmx (DOT) net> writes: Quote: On Fri, 07 Nov 2003 09:30:12 -0500, Raoul Gough wrote: "Dhruv" <dhruvbird (AT) gmx (DOT) net> writes: [snip] char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. I don't think the ordering makes any difference in this case, since sizeof() must include any tail-padding necessary for storing the object in an array. Certainly my compiler gives identical sizes for structs with either of the two orderings you mentioned: #include struct foo { char a[5]; int x; int y; }; struct bar { int x; int y; char a[5]; }; int main () { printf ("%u %un", sizeof (foo), sizeof(bar)); } my output: 16 16 Again, this padding thing varies form compiler to compiler, and there is nothing standard about it, so it's alright... The C++ standard, section 5.3.3/2 includes this about sizeof: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing objects of that type in an array." You seemed to be suggesting that putting the char[5] at the end would prevent any padding from being necessary, when it was necessary with the char[5] at the start. This is very unlikely because of the array packing issue - if the compiler would have to pad the first version in the middle then it would also have to pad the second version at the end. So I think if you actually try it out on your compiler you will also get the same size for both versions. -- Raoul Gough. (setq dabbrev-case-fold-search nil) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Raoul Gough Guest Posted: Sun Nov 09, 2003 10:52 pm    Post subject: Re: Packing structs into untyped memory with sizeof() Francis Glassborow <francis (AT) robinton (DOT) demon.co.uk> writes: Quote: In article <uwuadcoz1.fsf (AT) yahoo (DOT) co.uk>, Raoul Gough [email]RaoulGough (AT) yahoo (DOT) co.uk[/email]> writes char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. I don't think the ordering makes any difference in this case, since sizeof() must include any tail-padding necessary for storing the object in an array. Certainly my compiler gives identical sizes for structs with either of the two orderings you mentioned: #include <cstdio struct foo { char a[5]; int x; int y; }; struct bar { int x; int y; char a[5]; }; int main () { printf ("%u %un", sizeof (foo), sizeof(bar)); } However: struct foo { char a[5]; int x; int y; char c }; struct bar { int x; int y; char a[5]; char c }; would often be different sizes. If size matters, then you may need more care and platform specific arrangement than if it is irrelevant. Exactly - there's more to it than sorting the members by size and assuming that gives the best packing. I seem to remember using a C compiler once that could reorder structure elements automatically to achieve the smallest size, which must be kind of handy in some circumstances. IIRC, it was Watcom C for QNX, circa 1994 but I can't find any documentation for it now. BTW, does anyone know of any C++ compilers that actually take advantage of the freedom to reorder class members with intervening access specifiers (section 9.2/12)? e.g. #include struct a { char a[2]; int x; char b[2]; }; struct b { char a[2]; public: int x; public: char b[2]; }; struct c { char a[2]; char b[2]; int x; }; int main () { printf ("%u %u %un", sizeof (a), sizeof (b), sizeof (c)); } With MinGW g++ 3.3.1, output is 12 12 8, whereas a reordering of struct b for size efficiency would have produced 12 8 8. -- Raoul Gough. (setq dabbrev-case-fold-search nil) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dhruv Guest Posted: Fri Nov 14, 2003 10:19 am    Post subject: Re: Packing structs into untyped memory with sizeof() On Sun, 09 Nov 2003 17:51:42 -0500, Raoul Gough wrote: Quote: The C++ standard, section 5.3.3/2 includes this about sizeof: "When applied to a class, the result is the number of bytes in an object of that class including any padding required for placing objects of that type in an array." You seemed to be suggesting that putting the char[5] at the end would prevent any padding from being necessary, when it was necessary with the char[5] at the start. This is very unlikely because of the array packing issue - if the compiler would have to pad the first version in the middle then it would also have to pad the second version at the end. So I think if you actually try it out on your compiler you will also get the same size for both versions. As I said, all this is compiler specific, so I said probebly when I first wrote the statement, so all compilers might not give the same result. Regards, -Dhruv. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Raoul Gough Guest Posted: Sat Nov 15, 2003 9:56 am    Post subject: Re: Packing structs into untyped memory with sizeof() on 4 Nov 2003 12:10:30 -0500 "Dhruv" <dhruvbird (AT) gmx (DOT) net> wrote: Quote: char a[5]; int x; int y; would probably occupy 5+3+4+4 bytes instead of: int x; int y; char a[5]; 4+4+5 bytes. on 14 Nov 2003 05:19:34 -0500 "Dhruv" <dhruvbird (AT) gmx (DOT) net> wrote: Quote: On Sun, 09 Nov 2003 17:51:42 -0500, Raoul Gough wrote: [snip] This is very unlikely because of the array packing issue - if the compiler would have to pad the first version in the middle then it would also have to pad the second version at the end. So I think if you actually try it out on your compiler you will also get the same size for both versions. As I said, all this is compiler specific, so I said probebly when I first wrote the statement, so all compilers might not give the same result. OK - you're saying "probably", and I'm saying "very unlikely". I thought it was worth detailing the array packing issue, but maybe we'll just have to agree to disagree. -- Raoul Gough. (setq dabbrev-case-fold-search nil) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.