C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Passing strings to APIs Goto page 1, 2, 3  Next          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message Bartosz Milewski Guest Posted: Sun Oct 09, 2005 8:30 pm    Post subject: Passing strings to APIs When encapsulating C-language APIs in C++ wrappers one often has to convert between std::strings and C-strings. One such easy case is when the API requires a char const * --you can just pass it the result of std::string::c_str (). But then there is the case where the API "returns" a string. It takes a char * buffer together with its length. There either is the maximum length specified, or the client is supposed to make another call to find out the length. My solution was always to create a std::string of appropriate length and then pass the buffer to the API as &str[0]. But according to Alexandrescu & Sutter's "C++ Coding Standards" this is incorrect, since there is no guarantee that the string storage be contiguous. Technically this is true; which would force me to use a std::vector<char> (which is guaranteed to be contiguous!) instead of std::string, call the API with &vec[0], and then copy it to a string. I find this solution awkward and unnecessarily complex (and complexity always reduces maintainability). On the other hand I can't think of any sane implementation of std::string that wouldn't use contiguous storage. Just think how to implement c_str() (which is const) in such a case. Shouldn't the standard require std::string storage to be contiguous? Unfortunately, to be completely safe in using &str[0] for writing, one would also need the guarantee that there is room for the terminating null character at str[str.size()]. BTW, these conditions are met by every implementation known to me. Is there any advantage to not imposing them, other than for fear of missing some future cool optimizing trick? Bartosz [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bo Persson Guest Posted: Mon Oct 10, 2005 8:01 pm    Post subject: Re: Passing strings to APIs "Bartosz Milewski" <bartosz (AT) relisoft (DOT) com> skrev i meddelandet news:di9ao8$8jp$1 (AT) nwnexus-news (DOT) nwnexus.com... Quote: When encapsulating C-language APIs in C++ wrappers one often has to convert between std::strings and C-strings. One such easy case is when the API requires a char const * --you can just pass it the result of std::string::c_str (). But then there is the case where the API "returns" a string. It takes a char * buffer together with its length. There either is the maximum length specified, or the client is supposed to make another call to find out the length. My solution was always to create a std::string of appropriate length and then pass the buffer to the API as &str[0]. But according to Alexandrescu & Sutter's "C++ Coding Standards" this is incorrect, since there is no guarantee that the string storage be contiguous. Technically this is true; which would force me to use a std::vector (which is guaranteed to be contiguous!) instead of std::string, call the API with &vec[0], and then copy it to a string. I find this solution awkward and unnecessarily complex (and complexity always reduces maintainability). But does this occur frequently? Could you possibly encapsulate it reasonably well? Quote: On the other hand I can't think of any sane implementation of std::string that wouldn't use contiguous storage. Just think how to implement c_str() (which is const) in such a case. It is possible to store strings in segments, perhaps not performing some operations immediately. If the string is stored in segments, it would have to be concatenated at the first call to c_str(). The functions is allowed to return a temporary copy of the string content. That is one reason for the return value to be const. The major reason is probably to allow strings to share contents (reference counting). Quote: Shouldn't the standard require std::string storage to be contiguous? Just to solve interfacing old C code? Maybe, maybe not. Quote: Unfortunately, to be completely safe in using &str[0] for writing, one would also need the guarantee that there is room for the terminating null character at str[str.size()]. Which isn't guaranteed either, of course. There is no requirement to have a null stored in the string, it just has to be present in the c_str() return value. Quote: BTW, these conditions are met by every implementation known to me. Is there any advantage to not imposing them, other than for fear of missing some future cool optimizing trick? The only problem with using undefined behaviour, is that it sometimes doesn't work. :-) Bo Persson [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bartosz Milewski Guest Posted: Wed Oct 12, 2005 12:59 am    Post subject: Re: Passing strings to APIs Quote: It is possible to store strings in segments, perhaps not performing some operations immediately. If the string is stored in segments, it would have to be concatenated at the first call to c_str(). The functions is allowed to return a temporary copy of the string content. That is one reason for the return value to be const. The only way one could do it is to cheat on the constness of the method c_str(). It would not only have to allocate a buffer for the contiguous representation of the string (including the terminating null), but it would have to keep an internal pointer to it (you can't return a pointer to a temporary). The assignment to this internal pointer breaks the constness. (Granted, one could make this pointer mutable.) I guess one could imagine a special-purpose string implementation that would be optimized towards lots of concatentions, at the cost of more complex iteration, indexed access, and C-string conversion. Quote: The major reason is probably to allow strings to share contents (reference counting). It seems like in most scenarios that would be an orthogonal issue. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze Guest Posted: Wed Oct 12, 2005 8:49 am    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: It is possible to store strings in segments, perhaps not performing some operations immediately. If the string is stored in segments, it would have to be concatenated at the first call to c_str(). The functions is allowed to return a temporary copy of the string content. That is one reason for the return value to be const. The only way one could do it is to cheat on the constness of the method c_str(). I'm not sure what you mean by "cheat on the constness". The standard is emminately clear that within the library, const is logical const, not bit-wise const. Note too §21.3/5: "References, pointers, and iterators referring to the elements of a basic_string may be invalidated by the following uses of that basic_string object: [...] -- Calling data() and c_string() member functions". Quote: It would not only have to allocate a buffer for the contiguous representation of the string (including the terminating null), but it would have to keep an internal pointer to it (you can't return a pointer to a temporary). The assignment to this internal pointer breaks the constness. (Granted, one could make this pointer mutable.) For example. Or keep it somewhere else. Or whatever. It's all implementation details, which aren't addressed by the standard. Quote: I guess one could imagine a special-purpose string implementation that would be optimized towards lots of concatentions, at the cost of more complex iteration, indexed access, and C-string conversion. SGI has a class, rope, which does just this. Amongst other things, it supports inserting into the middle of a string in (I think) constant time. The intent of the standard was to allow such implementations. Quote: The major reason is probably to allow strings to share contents (reference counting). It seems like in most scenarios that would be an orthogonal issue. Sort of. The fundamental reason was to allow as much liberty as possible to the implementation, so that the implementers could optimize for what they considered most important for their users. If they thought that copying very long strings was critical, they could use reference counting. If they thought that insertion into an arbitrary location in very long strings was critical, they could use a segmented representation. Note that casting away const on the return value of c_str() will break most reference counted implementations (unless you take steps to ensure that the image is not shared before doing so). -- James Kanze GABI Software Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Greg Herlihy Guest Posted: Wed Oct 12, 2005 8:50 am    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: When encapsulating C-language APIs in C++ wrappers one often has to convert between std::strings and C-strings. One such easy case is when the API requires a char const * --you can just pass it the result of std::string::c_str (). But then there is the case where the API "returns" a string. It takes a char * buffer together with its length. There either is the maximum length specified, or the client is supposed to make another call to find out the length. My solution was always to create a std::string of appropriate length and then pass the buffer to the API as &str[0]. But according to Alexandrescu & Sutter's "C++ Coding Standards" this is incorrect, since there is no guarantee that the string storage be contiguous. Technically this is true; which would force me to use a std::vector<char (which is guaranteed to be contiguous!) instead of std::string, call the API with &vec[0], and then copy it to a string. I find this solution awkward and unnecessarily complex (and complexity always reduces maintainability). On the other hand I can't think of any sane implementation of std::string that wouldn't use contiguous storage. Just think how to implement c_str() (which is const) in such a case. Shouldn't the standard require std::string storage to be contiguous? Unfortunately, to be completely safe in using &str[0] for writing, one would also need the guarantee that there is room for the terminating null character at str[str.size()]. BTW, these conditions are met by every implementation known to me. Is there any advantage to not imposing them, other than for fear of missing some future cool optimizing trick? Bartosz The program, like any program, should do the "right thing." If std::string's buffer is not guaranteed to be contiguous then the program should not pass it to a function expecting a contiguous block of memory. In fact, I'm not even sure that std::string's buffer is even certain to be writable, even if it is known to be contiguous. The by-the-book solution need not be awkward or complex. It should be possible to implement a solution that is not only safe, but almost completely transparent. For example, consider this program: void GetCString( char * buffer, int len); { std::strncpy(buffer, "some text", len); } int main() { std::string s; // "reserve" space in s for the return string s.reserve(16); // pass s (suitably wrapped) as a char * parameter GetCString( writable_string(s), s.capacity() ); // s contains the string that GetCString wrote to the buffer std::cout << "s contains "" s << """ << std::endl; } Output: s contains "some text" In fact, it is hard to imagine a more streamlined way for s to hold the string that GetCString copied into buffer passed to it as a parameter. There is just one visible sign of the implementation - the writable_string that wraps the std::string instance. In fact this explicit conversion is safer than an implicit conversion because implicit conversions may occur in other situations in which such a conversion has not been anticpated. The only other task is implement writable_string: class writable_string { public: writable_string( std::string& s) : v_( s.capacity()), s_( s) { } ~writable_string() { s_.assign( &v_[0]); } operator char *() const { return (char *) &v_[0]; } private: std::vector std::string& s_; }; Otherwise there is nothing more needed. Greg [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top ben Guest Posted: Wed Oct 12, 2005 8:52 am    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: When encapsulating C-language APIs in C++ wrappers one often has to convert between std::strings and C-strings. One such easy case is when the API requires a char const * --you can just pass it the result of std::string::c_str (). Okay. Quote: But then there is the case where the API "returns" a string. It takes a char * buffer together with its length. There either is the maximum length specified, or the client is supposed to make another call to find out the length. Indeed. Quote: My solution was always to create a std::string of appropriate length and then pass the buffer to the API as &str[0]. But according to Alexandrescu & Sutter's "C++ Coding Standards" this is incorrect, since there is no guarantee that the string storage be contiguous. Don't do it. C string is a physical representation, C++ string is an abstract storage concept. They are different, just don't do one as the other. Quote: Technically this is true; which would force me to use a std::vector (which is guaranteed to be contiguous!) instead of std::string, call the API with &vec[0], and then copy it to a string. I find this solution awkward and unnecessarily complex (and complexity always reduces maintainability). On the other hand I can't think of any sane implementation of std::string that wouldn't use contiguous storage. Just think how to implement c_str() (which is const) in such a case. You don't need a vector. You can use a statically allocated (or globally allocated) char array for the buff. Just make it reasonably large so that the runtime length can never pass over the size. Then you can either use the buffer to construct a C++ string, or just use it as is. Quote: Shouldn't the standard require std::string storage to be contiguous? As I said before, std::string is an abstract storage concept. If you really need a string class that has contiguous internal storage, i'm sure you can find one or easily write your own. After all, std::string doesn't come with a horribly lot of string operations; and STL operations are guaranteed to work on arrays, don't they. Quote: Unfortunately, to be completely safe in using &str[0] for writing, one would also need the guarantee that there is room for the terminating null character at str[str.size()]. Repeat, std::string is not a physical storage concept. Quote: BTW, these conditions are met by every implementation known to me. Is there any advantage to not imposing them, other than for fear of missing some future cool optimizing trick? If they update their libraries in the future taking advantage of the current standard, it will break your code--and that is not going to be fun. Quote: Bartosz Regards, Ben [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bartosz Milewski Guest Posted: Wed Oct 12, 2005 10:50 pm    Post subject: Re: Passing strings to APIs Quote: int main() { std::string s; // "reserve" space in s for the return string s.reserve(16); // pass s (suitably wrapped) as a char * parameter GetCString( writable_string(s), s.capacity() ); // s contains the string that GetCString wrote to the buffer std::cout << "s contains "" s << """ << std::endl; } This is indeed an elegant solution. The only objection is performance. You are making one unnecessary copy in ~writable_string() { s_.assign( &v_[0]); } This could be a substantial hit if the API is to return a few megabytes of text. BTW, the use of reserve and capacity is very clever. I only realized that when looking at the destructor. It won't throw an exception! [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top wizofaus@hotmail.com Guest Posted: Thu Oct 13, 2005 1:00 pm    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: ~writable_string() { s_.assign( &v_[0]); } This could be a substantial hit if the API is to return a few megabytes of text. I would suggest that if that was likely, just keep the data in a vector until you absolute need (some portion of) it as a string. After all, there's not likely to be that many "stringy" type operations you'd want to perform a few megabytes of text in one go. About the most likely case is to search for a substring, which can be done with either std::search() or of course std::strstr() directly on the vector. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Greg Herlihy Guest Posted: Thu Oct 13, 2005 1:10 pm    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: int main() { std::string s; // "reserve" space in s for the return string s.reserve(16); // pass s (suitably wrapped) as a char * parameter GetCString( writable_string(s), s.capacity() ); // s contains the string that GetCString wrote to the buffer std::cout << "s contains "" s << """ << std::endl; } This is indeed an elegant solution. The only objection is performance. You are making one unnecessary copy in ~writable_string() { s_.assign( &v_[0]); } In terms of efficiency, this copy operation is in fact suboptimal - but the operation itself is nevertheless necessary because the program has chosen to use a std::string instance to hold the string returned by the function call. Having chosen std::string the program has no other option than to adhere to std::string's interface for manipulating its contents. And as far as I can tell, there is no better routine in std::string's interface that could be used here. So the program makes the best choice from the options that std::string has made available. In particular, the program is not entitled to rewrite std::string's interface in an effort to turn std::string into the string class that it wishes it were using. To do so is to make the program incorrect. Even if the attempt works, the program is still incorrect. And the measure of quality for any software program is not simply that it produce correct results - quality is attained only when the program itself is correct. Therefore, should this copy operation turn out to have an unacceptably high performance cost, the only recourse would be to replace std::string with a different string class that offers better performance characteristics - namely one whose interface allows direct access to its contents. An obvious candidate would be std::stringstream, since it offers both the std::string interface and a std::iostream interface for efficient input and output operations. std::vector Greg [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bartosz Milewski Guest Posted: Thu Oct 13, 2005 1:11 pm    Post subject: Re: Passing strings to APIs Quote: Note too §21.3/5: "References, pointers, and iterators referring to the elements of a basic_string may be invalidated by the following uses of that basic_string object: [...] -- Calling data() and c_string() member functions". Thanks for this reference. This is even worse than what I was proposing (using a volatile cache for the result of c_str). Doesn't this sort of make the const modifier meaningless? Has anyone defined what "logical constness" is? Quote: SGI has a class, rope, which does just this. Amongst other things, it supports inserting into the middle of a string in (I think) constant time. The intent of the standard was to allow such implementations. Note however that the class is called "rope" and not std::string. Imagine using rope as a genaral purpose string. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Bartosz Milewski Guest Posted: Thu Oct 13, 2005 1:17 pm    Post subject: Re: Passing strings to APIs Quote: You don't need a vector. You can use a statically allocated (or globally allocated) char array for the buff. Just make it reasonably large so that the runtime length can never pass over the size. Then you can either use the buffer to construct a C++ string, or just use it as is. It wouldn't work in a multithreaded environment. Quote: As I said before, std::string is an abstract storage concept. If you really need a string class that has contiguous internal storage, i'm sure you can find one or easily write your own. After all, std::string doesn't come with a horribly lot of string operations; and STL operations are guaranteed to work on arrays, don't they. Actually, std::string does come with quite a lot of functionality (both methods and free functions). Also, my encapsulation of the API is meant to be part of a library. I don't want to force clients to use my own version of string (or a std::vector<char>). There is a precedent in STL for the contiguity requirement: the std::vector. I wonder what kind of arguments convinced the Standard Committee to enforce vector's contiguity, and why the same arguments don't apply to std::string? [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Andrei Alexandrescu (See Guest Posted: Thu Oct 13, 2005 1:42 pm    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: This is indeed an elegant solution. The only objection is performance. You are making one unnecessary copy in ~writable_string() { s_.assign( &v_[0]); } This could be a substantial hit if the API is to return a few megabytes of text. BTW, the use of reserve and capacity is very clever. I only realized that when looking at the destructor. It won't throw an exception! I actually have an idea. How about testing whether the string is bona fide (contiguous) or smart (tricky)? std::string s(AppropriateSize()); const size_t sz = s.size(); const char * cp = s.c_str(); char * mp = &s[0]; if (!std::less<void*>()(mp, cp) && !std::less<void*>()(cp, mp)) { GetCString(mp, sz); } else { std::vector temp(sz); GetCString(&temp[0], sz); std::copy(temp.begin(), temp.end(), s.begin()); } Notice the separate calls to s.c_str() and s[0] for the sake of introducing sequence points ). Don't short-circuit that to "if (s.c_str() == &s[0])". Also, I'm calling s.size() separately because in theory it might modify the string. Also, I'm using std::less and not "==" because in the general case the pointers can be unrelated and as such uncomparable via "==". That way you are efficient on most systems, and stay correct and portable on all. Or, is there some subtler problem? Andrei [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Yuri Khan Guest Posted: Thu Oct 13, 2005 1:43 pm    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: This is indeed an elegant solution. The only objection is performance. You are making one unnecessary copy in ~writable_string() { s_.assign( &v_[0]); } This could be a substantial hit if the API is to return a few megabytes of text. If you are concerned about performance, use vectors of char instead of strings. You can do many things with std::vector using standard algorithms, that you would do with std::string's member functions. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze Guest Posted: Thu Oct 13, 2005 11:04 pm    Post subject: Re: Passing strings to APIs Bartosz Milewski wrote: Quote: int main() { std::string s; // "reserve" space in s for the return string s.reserve(16); // pass s (suitably wrapped) as a char * parameter GetCString( writable_string(s), s.capacity() ); // s contains the string that GetCString wrote to the buffer std::cout << "s contains "" s << """ << std::endl; } This is indeed an elegant solution. The only objection is performance. You are making one unnecessary copy in ~writable_string() { s_.assign( &v_[0]); } This could be a substantial hit if the API is to return a few megabytes of text. The real hit could be the excess memory allocations, especially if the interface is returning very small strings, and the string implementation uses the small string optimization. If the necessary length is not known at compile time, you probably cannot avoid the allocations, so it is no big deal. If the length is known at compile time, however, allocating a char[] directly on the stack is definitly preferrable. Quote: BTW, the use of reserve and capacity is very clever. I only realized that when looking at the destructor. It won't throw an exception! The problem is that there is no relatiionship between the capacity and the length passed to the GetCString function. This is one case where information hiding is likely to cause problems; you can be pretty sure that users are going to forget the reserve, and write something like: std::string s ; GetCString( writable_string(s), 100 ) ; (or use it like this with a function which expects a fixed sized buffer). The relationship between the two seems a bit subtle to me, especially since it is generally considered that reserve is only for optimizations. Obviously, there's no really effective solution, given the existing interface. Which suggests to me that the real solution is to wrap the interface, and not to let the users access GetCString directly. One last point: unlike std::vector, I don't think that std::string::reserve actually guarantees that the assign will not throw. Consider a reference counted implementation, for example. -- James Kanze GABI Software Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Niklas Matthies Guest Posted: Thu Oct 13, 2005 11:05 pm    Post subject: Re: Passing strings to APIs On 2005-10-13 13:42, Andrei Alexandrescu (See Website For Email) wrote: : Quote: I actually have an idea. How about testing whether the string is bona fide (contiguous) or smart (tricky)? std::string s(AppropriateSize()); const size_t sz = s.size(); const char * cp = s.c_str(); char * mp = &s[0]; if (!std::less<void*>()(mp, cp) && !std::less<void*>()(cp, mp)) { GetCString(mp, sz); } else { : That way you are efficient on most systems, and stay correct and portable on all. Or, is there some subtler problem? In theory, the string could still be a "rope" and only share some initial sequence with the c_str, where the latter continues into a read-only segment after that common initial sequence (assuming some clever MMU fiddling). -- Niklas Matthies [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Goto page 1, 2, 3  Next Page 1 of 3   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.