C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  Reading from a stream til EOF Goto page 1, 2  Next          C++Talk.NET Forum Index -> C++ language (comp.lang.c++) View previous topic :: View next topic   Author Message Hendrik Schober Guest Posted: Wed Feb 25, 2004 10:05 pm    Post subject: Reading from a stream til EOF Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? TIA; Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Rodrigo Dominguez Guest Posted: Wed Feb 25, 2004 10:31 pm    Post subject: Re: Reading from a stream til EOF Hendrik Schober wrote: Quote: Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? TIA; Schobi well, I'm not an expert on STL, but here are some examples example 1: char c; while(your_istream.get(c)) your_string.push_back(c); example 2: char c; while(your_istream >> c) your_string.push_back(c); example 3: string your_string; while(your_istream >> your_string) foo(); -- Rodrigo Dominguez <rorra (AT) rorra (DOT) com.ar> Powered Hosting (www.powered-design.com) Back to top Hendrik Schober Guest Posted: Wed Feb 25, 2004 10:37 pm    Post subject: Re: Reading from a stream til EOF Rodrigo Dominguez <rorra (AT) rorra (DOT) com.ar> wrote: Quote: Hendrik Schober wrote: Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? TIA; Schobi well, I'm not an expert on STL, but here are some examples [...] Actually I was hoping for something that would promiss more performance. Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Jonathan Turkanis Guest Posted: Wed Feb 25, 2004 11:00 pm    Post subject: Re: Reading from a stream til EOF "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote Quote: Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? I'm afraid making a copy at some point is unavoidable. I wish you could call reserve() and then write directly into the underlying storage, as with vector -- at least if the string had never been copied. Jonathan Back to top Hendrik Schober Guest Posted: Wed Feb 25, 2004 11:11 pm    Post subject: Re: Reading from a stream til EOF Jonathan Turkanis <technews (AT) kangaroologic (DOT) com> wrote: Quote: "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote in message news:c1j5uf$o2g$1 (AT) news1 (DOT) transmedia.de... Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? I'm afraid making a copy at some point is unavoidable. I wish you could call reserve() and then write directly into the underlying storage, as with vector -- at least if the string had never been copied. I suppose you mean 'resize()', where you say 'reserve()'? The problem is, I don't see how I can find out how much there is to read from the stream in advance. What I'm doing right now is this: std::string f(std::istream& is) { return std::string( std::istream_iterator<char>(is) , std::istream_iterator<char>() ); } However, I suppose this goes through all the sentries etc. for each and every char? One other thing I was thinking about is that 'operator>>' seems to be overloaded for a stream buffer on the RHS. So should this std::stringstream ss; is >> ss.rdbuf(); return ss.str(); do what I think? And if so, can I expect better performance from this compared to copying the char myself? Quote: Jonathan Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Jonathan Turkanis Guest Posted: Wed Feb 25, 2004 11:35 pm    Post subject: Re: Reading from a stream til EOF "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote Quote: Jonathan Turkanis <technews (AT) kangaroologic (DOT) com> wrote: "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote in message news:c1j5uf$o2g$1 (AT) news1 (DOT) transmedia.de... Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? I'm afraid making a copy at some point is unavoidable. I wish you could call reserve() and then write directly into the underlying storage, as with vector -- at least if the string had never been copied. I suppose you mean 'resize()', where you Yes. Quote: say 'reserve()'? The problem is, I don't see how I can find out how much there is to read from the stream in advance. Right. That's unavoidable. An exponential growth strategy is the way to go. You should get this automatically with string, or you can do it yourself. Quote: What I'm doing right now is this: std::string f(std::istream& is) { return std::string( std::istream_iterator<char>(is) , std::istream_iterator<char>() ); } You defintely don't want to do this if you're concerned with efficiency. At the very least, you should extract the underlying streambuf using is.rdbuf(), and read into a char array using sgetn. Quote: However, I suppose this goes through all the sentries etc. for each and every char? One other thing I was thinking about is that 'operator>>' seems to be overloaded for a stream buffer on the RHS. So should this std::stringstream ss; is >> ss.rdbuf(); return ss.str(); I would have guessed that a good implementation would implement this as I described above, but I checked dinkumware and it does a character-by-character extraction. So I would use a char buffer. (In my first response, I though you were mainly interested in avoiding the final copy when you call ss.str()) Jonathan Back to top Hendrik Schober Guest Posted: Thu Feb 26, 2004 10:43 am    Post subject: Re: Reading from a stream til EOF Jonathan Turkanis <technews (AT) kangaroologic (DOT) com> wrote: Quote: [...] say 'reserve()'? The problem is, I don't see how I can find out how much there is to read from the stream in advance. Right. That's unavoidable. An exponential growth strategy is the way to go. You should get this automatically with string, or you can do it yourself. I planned to let 'std::string' take care of this. :) Quote: What I'm doing right now is this: std::string f(std::istream& is) { return std::string( std::istream_iterator<char>(is) , std::istream_iterator<char>() ); } You defintely don't want to do this if you're concerned with efficiency. I see. I was expecting this. I suppose using streambuf iterators wouldn't help much with this? Quote: At the very least, you should extract the underlying streambuf using is.rdbuf(), and read into a char array using sgetn. As this avoids creating/destroying any sentries and all the formatting? Quote: However, I suppose this goes through all the sentries etc. for each and every char? One other thing I was thinking about is that 'operator>>' seems to be overloaded for a stream buffer on the RHS. So should this std::stringstream ss; is >> ss.rdbuf(); return ss.str(); I would have guessed that a good implementation would implement this as I described above, but I checked dinkumware and it does a character-by-character extraction. Thanks for checking. We are indeed using Dinkumware on two platforms. So this would not help much. I should probably ask about this MS' std lib newsgroup, as PJP and PB are reading and posting there. Quote: So I would use a char buffer. I am not sure what you mean here. Can you elaborate. Quote: (In my first response, I though you were mainly interested in avoiding the final copy when you call ss.str()) Well, actually, I would need to istream the content later anyway. However, first I need the size of it. (The real task is to parse the data, which is a rather lengthy process. OTOH the raw data itself usually is not very big. So I thought it would be better to loose some performance on copying to get the size, as this would give me a real progress bar for visual feedback to the users.) Quote: Jonathan Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Dietmar Kuehl Guest Posted: Thu Feb 26, 2004 11:57 am    Post subject: Re: Reading from a stream til EOF "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote: Quote: What I'm doing right now is this: std::string f(std::istream& is) { return std::string( std::istream_iterator<char>(is) , std::istream_iterator<char>() ); } This is not at all what you want to do, I guess: amoung others, this will strip all white spaces from the input before putting it into the string! Quote: However, I suppose this goes through all the sentries etc. for each and every char? Yes, this goes through the sentries and the preparation etc. What you probably want to do is this: std::string f(std::istream& is) { return std::string( std::istreambuf_iterator<char>(is), std::istreambuf_iterator<char>() ); } This does not go through the sentires. However, for this to be efficient, the library has either to implement the general segmented iterator optimization or it has to special case this particular use in some form. My implementation has a special case (which is pretty close to the general optimization but is not quite there) and this is the fastest method to read a string, especially for a file with the "C" facet: in this case it essentially amounts to a memcpy() from a memory mapped file to the string. Quote: One other thing I was thinking about is that 'operator>>' seems to be overloaded for a stream buffer on the RHS. So should this std::stringstream ss; is >> ss.rdbuf(); return ss.str(); I would expect this to be the fastest approach with typical implementations: this may bypass certain internal buffers, etc. For buffered input streams this should at the very least process blocks of characters from buffers directly. Quote: do what I think? And if so, can I expect better performance from this compared to copying the char myself? Go measure... I would expect the 'rdbuf()' to be significantly faster than processing individual characters. Here is something which should also be faster than processing individual characters: enum { bufsize = 8192 }; char buf[bufsize]; std::string s; for (std::streamsize size = 0; size = is.read(buf, bufsize) > 0; ) s.append(buf, size); (this code is untested and I'm somewhat humble with respect to the string interface...). -- <mailto:dietmar_kuehl (AT) yahoo (DOT) com> <http://www.dietmar-kuehl.de/> Phaidros eaSE - Easy Software Engineering: <http://www.phaidros.com/> Back to top Hendrik Schober Guest Posted: Thu Feb 26, 2004 12:57 pm    Post subject: Re: Reading from a stream til EOF Dietmar Kuehl <dietmar_kuehl (AT) yahoo (DOT) com> wrote: Quote: "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote: What I'm doing right now is this: std::string f(std::istream& is) { return std::string( std::istream_iterator<char>(is) , std::istream_iterator<char>() ); } This is not at all what you want to do, I guess: amoung others, this will strip all white spaces from the input before putting it into the string! Yes, I found this out by now. > Quote: However, I suppose this goes through all the sentries etc. for each and every char? Yes, this goes through the sentries and the preparation etc. What you probably want to do is this: std::string f(std::istream& is) { return std::string( std::istreambuf_iterator<char>(is), std::istreambuf_iterator<char>() ); } This does not go through the sentires. This was the next thing I was about to try. Quote: However, for this to be efficient, the library has either to implement the general segmented iterator optimization [...] ??? Quote: [...] std::stringstream ss; is >> ss.rdbuf(); return ss.str(); I would expect this to be the fastest approach with typical implementations: this may bypass certain internal buffers, etc. For buffered input streams this should at the very least process blocks of characters from buffers directly. Could I do this the other way around, too? std::stringstream ss; ss << is.rdbuf(); return ss.str(); And if so, is there anything different in principle or is it just down to the particular library? Quote: [...] Go measure... The problem is, I need to find a way to do this which most likely is fast on a couple of platforms without beeing able to profile it on each one. Quote: I would expect the 'rdbuf()' to be significantly faster than processing individual characters. I see. Quote: Here is something which should also be faster than processing individual characters: enum { bufsize = 8192 }; char buf[bufsize]; std::string s; for (std::streamsize size = 0; size = is.read(buf, bufsize) > 0; ) s.append(buf, size); (this code is untested and I'm somewhat humble with respect to the string interface...). The good old char buf read functions. I wonder why it is so hard to do something efficiently without having to go back to C-ish ways. Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top tom_usenet Guest Posted: Thu Feb 26, 2004 3:22 pm    Post subject: Re: Reading from a stream til EOF On Wed, 25 Feb 2004 23:05:55 +0100, "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote: Quote: Hi, I have a 'std::istream' and need to read its whole contents into a string. How can I do this? I've posted a few solutions to this in the past: http://www.google.com/groups?selm=3d047f38.4571152%40news.easynet.co.uk There are lots more ways, and the most efficient somewhat depends on the library implementation in question. Tom -- C++ FAQ: http://www.parashift.com/c++-faq-lite/ C FAQ: http://www.eskimo.com/~scs/C-faq/top.html Back to top Hendrik Schober Guest Posted: Thu Feb 26, 2004 3:33 pm    Post subject: Re: Reading from a stream til EOF tom_usenet <tom_usenet (AT) hotmail (DOT) com> wrote: Quote: [...] I've posted a few solutions to this in the past: http://www.google.com/groups?selm=3d047f38.4571152%40news.easynet.co.uk I didn't think of seeking through a stream to get its size! Of all the reasons I wanted to do this I did manage to eliminate all except that I need the size of the data to be read from the stream. Since you just showed me how to get this, I won't even need to read the whole thing into a string anymore! Quote: There are lots more ways, and the most efficient somewhat depends on the library implementation in question. Yes. What I wanted was a solution that has good performance on most platforms. However, I think I don't need it anymore. :) Quote: Tom Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top tom_usenet Guest Posted: Thu Feb 26, 2004 5:14 pm    Post subject: Re: Reading from a stream til EOF On Thu, 26 Feb 2004 16:33:55 +0100, "Hendrik Schober" <SpamTrap (AT) gmx (DOT) de> wrote: Quote: tom_usenet <tom_usenet (AT) hotmail (DOT) com> wrote: [...] I've posted a few solutions to this in the past: http://www.google.com/groups?selm=3d047f38.4571152%40news.easynet.co.uk I didn't think of seeking through a stream to get its size! Of all the reasons I wanted to do this I did manage to eliminate all except that I need the size of the data to be read from the stream. Since you just showed me how to get this, I won't even need to read the whole thing into a string anymore! There are a couple of provisos. Firstly, opening the stream in binary mode is likely to give you a better result (e.g. the number of bytes in the file) - text mode sometimes has funny ideas about where a file ends on some OSes. Secondly, it won't work for files whose length won't fit in a std::streamoff (e.g. bigger than, say, 2GB). Finally, don't forget you can just use a std::filebuf and cut out the fstream entirely. Tom -- C++ FAQ: http://www.parashift.com/c++-faq-lite/ C FAQ: http://www.eskimo.com/~scs/C-faq/top.html Back to top Hendrik Schober Guest Posted: Thu Feb 26, 2004 5:27 pm    Post subject: Re: Reading from a stream til EOF tom_usenet <tom_usenet (AT) hotmail (DOT) com> wrote: Quote: [...] Firstly, opening the stream in binary mode is likely to give you a better result (e.g. the number of bytes in the file) - text mode sometimes has funny ideas about where a file ends on some OSes. Is there anything worse to be expected than the "rn" problem? As this is just for progress indication for the users, accuracy is not as important. Quote: Secondly, it won't work for files whose length won't fit in a std::streamoff (e.g. bigger than, say, 2GB). Yes. But I woulnd't have thought of loading these into a string anyway. :) Quote: Finally, don't forget you can just use a std::filebuf and cut out the fstream entirely. How do I read a line from a streambuf? Quote: Tom Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Hendrik Schober Guest Posted: Thu Feb 26, 2004 6:05 pm    Post subject: Re: Reading from a stream til EOF Dietmar Kuehl <dietmar_kuehl (AT) yahoo (DOT) com> wrote: Quote: [...] Well, essentially, a streambuf iterator [...] Thanks for the enlightment! Quote: Could I do this the other way around, too? std::stringstream ss; std::ostringstream ss; ss << is.rdbuf(); return ss.str(); This is how I'm normally writing it. The direction should not really matter and the same function should be used underneath. I see. Quote: The problem is, I need to find a way to do this which most likely is fast on a couple of platforms without beeing able to profile it on each one. But you should get a general feeling which things work fast and which don't by trying out a couple. Actually, I'm aware of only five different libraries being in wider use: - Dinkumware (eg. shipping with MSVC++) - libstdc++ (shipping with gcc) - Metrowerk's library shipping with their compiler - RougeWave (used to ship eg. with Sun CC) - STLport (a free drop in place library) Yes, but then there is all the different versions of these libraries. And once a piece of code works, nobody will go into it and check whether with the newest version this or that could be optimized using another technique... Quote: I'm unaware of any other standard C++ library shipping with a commmercial compiler (ObjectSpace dropped their library and mine was never shipping with anything; Warum eigentlich? Quote: is there any other reasonably complete standard library implementation still in use?) The good old char buf read functions. I wonder why it is so hard to do something efficiently without having to go back to C-ish ways. Well, the segmented iterator optimization requires quite a bit of machinery to work. It gives a nice abstract interface to an efficient implementation. Just, nobody does it because the library implementers are kept busy with all kinds of other stuff and optimizations. The low-level stuff is some wiring you can apply yourself... But I wonder whether it is a flaw in the design if something like reading into a string cannot easily be done fast with the recommended approach. Schobi -- [email]SpamTrap (AT) gmx (DOT) de[/email] is never read I'm Schobi at suespammers dot org "Sometimes compilers are so much more reasonable than people." Scott Meyers Back to top Dietmar Kuehl Guest Posted: Thu Feb 26, 2004 6:11 pm    Post subject: Re: Reading from a stream til EOF Hendrik Schober wrote: Quote: Dietmar Kuehl <dietmar_kuehl (AT) yahoo (DOT) com> wrote: std::string f(std::istream& is) { return std::string( std::istreambuf_iterator<char>(is), std::istreambuf_iterator<char>() ); } However, for this to be efficient, the library has either to implement the general segmented iterator optimization [...] Well, essentially, a streambuf iterator iterates over buffers of characters. Sure, it is always the same buffer but just envision each fill of the buffer a separate one. Now, each of these buffers can be processed in a chunk making up a segment of the overall sequence. Taking advantage of this view results in faster code because rather than making two checks in each iteration, there is just one. Also, it is possible to unroll the loop even further because the sizes of the segments are known in advance, allowing to make a check only for something like every 100th character. Without this optimization, the processing of stream buffers will work more efficiently because this processing does just this, just more naturally (at least, I would expect it from most implementations). The general principle can also be applied to other kinds of sequences which are similarily segmented. 'std::deque's and hashes using lists of each bucket come to mind. Quote: Could I do this the other way around, too? std::stringstream ss; std::ostringstream ss; ss << is.rdbuf(); return ss.str(); This is how I'm normally writing it. The direction should not really matter and the same function should be used underneath. Quote: The problem is, I need to find a way to do this which most likely is fast on a couple of platforms without beeing able to profile it on each one. But you should get a general feeling which things work fast and which don't by trying out a couple. Actually, I'm aware of only five different libraries being in wider use: - Dinkumware (eg. shipping with MSVC++) - libstdc++ (shipping with gcc) - Metrowerk's library shipping with their compiler - RougeWave (used to ship eg. with Sun CC) - STLport (a free drop in place library) I'm unaware of any other standard C++ library shipping with a commmercial compiler (ObjectSpace dropped their library and mine was never shipping with anything; is there any other reasonably complete standard library implementation still in use?) Quote: The good old char buf read functions. I wonder why it is so hard to do something efficiently without having to go back to C-ish ways. Well, the segmented iterator optimization requires quite a bit of machinery to work. It gives a nice abstract interface to an efficient implementation. Just, nobody does it because the library implementers are kept busy with all kinds of other stuff and optimizations. The low-level stuff is some wiring you can apply yourself... -- Phaidros eaSE - Easy Software Engineering: <http://www.phaidros.com/> Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ language (comp.lang.c++) All times are GMT Goto page 1, 2  Next Page 1 of 2   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.