C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  STL's Stream and Buffer Object Interaction          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message Efrat Regev Guest Posted: Wed Jan 19, 2005 11:54 pm    Post subject: STL's Stream and Buffer Object Interaction Hello, I would like to ask a question on the interaction between STL stream and streambuf objects. Suppose I would like, for some reason, to store data in files, such that every 1024 bytes a CRC of the preceding 1000 bytes is stored. I would like clients of these files to see them as "normal" files, i.e., they should be unaware of CRCs. I don't want to derive a new class from a stream class, since, as Stroustrup writes in his book, streams deal with character formatting, and CRCs have nothing to do with that. I derive a new class from std::streambuf, therefore. Users of the class, however, will typically interact with it through a stream object, e.g., by selecting an object of my buffer class through rdbuf(). I don't understand well, however, how the stream object and buffer object interact: 1. Suppose my buffer object detects an incorrect CRC (=> the file was corrupted). Then naturally, it could throw an exception. However, as Stroustrup writes in his book, not all users want stream objects to throw an exception (and for this, the clear() and exceptions() method are given). Consequently, the buffer object should set the stream object's state to indicate failure. How can the buffer object know what is (if any) its stream object, though? 2. Suppose someone seeks on the stream. Then the buffer object should somehow translate the logical offset to an actual offset. What is the sequence of operations pertaining to the buffer object when a stream object is seeked? I appreciate your time in reading (and hopefully) replying to this mail. Thanks [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top msalters Guest Posted: Thu Jan 20, 2005 6:16 pm    Post subject: Re: STL's Stream and Buffer Object Interaction Efrat Regev wrote: Quote: Hello, I would like to ask a question on the interaction between STL stream and streambuf objects. Suppose I would like, for some reason, to store data in files, such that every 1024 bytes a CRC of the preceding 1000 bytes is stored. I would like clients of these files to see them as "normal" files, i.e., they should be unaware of CRCs. [ use streambuf ] 1. Suppose my buffer object detects an incorrect CRC (=> the file was corrupted). Then naturally, it could throw an exception. However, as Stroustrup writes in his book, not all users want stream objects to throw an exception (and for this, the clear() and exceptions() method are given). Consequently, the buffer object should set the stream object's state to indicate failure. How can the buffer object know what is (if any) its stream object, though? It can't. Streambufs can be used without streams. Quote: 2. Suppose someone seeks on the stream. Then the buffer object should somehow translate the logical offset to an actual offset. What is the sequence of operations pertaining to the buffer object when a stream object is seeked? I'm not sure what your question is. Streambufs have two public seek functions, pubseekoff() and pubseekpos() which are non-virtuals forwarding to the virtual protected seekoff() and seekpos(). Regards, Michiel Salters [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Thu Jan 20, 2005 6:31 pm    Post subject: Re: STL's Stream and Buffer Object Interaction Efrat Regev wrote: Quote: I would like to ask a question on the interaction between STL stream and streambuf objects. Suppose I would like, for some reason, to store data in files, such that every 1024 bytes a CRC of the preceding 1000 bytes is stored. I would like clients of these files to see them as "normal" files, i.e., they should be unaware of CRCs. A filtering streambuf should be all that is necessary. Quote: I don't want to derive a new class from a stream class, since, as Stroustrup writes in his book, streams deal with character formatting, and CRCs have nothing to do with that. I derive a new class from std::streambuf, therefore. Right. And this class uses the original filebuf or whatever as its data source. It's really fairly simple. (You might want to look at the filtering streambuf's at my site. And the articles I wrote for the late C++ Report explaining them.) Quote: Users of the class, however, will typically interact with it through a stream object, e.g., by selecting an object of my buffer class through rdbuf(). It's fairly traditional to provide a derived istream as well, which sets up the correct streambuf. Quote: I don't understand well, however, how the stream object and buffer object interact: Via the streambuf interface:-). Seriously, streambuf is just an abstract class, like any other. The istream and ostream which use it don't know anything more than the public functions. Of course, streambuf isn't so much an interface class, as it is the base class of the template pattern; istream and ostream don't (and can't) call any of the virtual functions. And in your implementation, all you have to worry about are the virtual functions. For reading, the simplest solution seems to be to provide a single character buffer, and just override underflow. Writing is even easier, just override overflow. Supporting seeking and both reading and writing in the same streambuf is a bit more complicated, but not that much. In your case, I'd probably buffer in the filtering streambuf, although this isn't usual. Basically, anytime underflow is called, you read the next 1024 characters from the source, plus the CRC, and either set up the buffers (function setg) and return the first character, if the CRC is correct, or return EOF if it isn't (or if there weren't any more characters to read). Quote: 1. Suppose my buffer object detects an incorrect CRC (= the file was corrupted). Then naturally, it could throw an exception. However, as Stroustrup writes in his book, not all users want stream objects to throw an exception (and for this, the clear() and exceptions() method are given). Consequently, the buffer object should set the stream object's state to indicate failure. How can the buffer object know what is (if any) its stream object, though? If you detect any sort of an error, you return an error status from the function that was called. All of the functions have return values with a special value which can be used: EOF for the one character functions (which return an int), and 0 for the buffer functions (which return the number of characters transfered). Quote: 2. Suppose someone seeks on the stream. Then the buffer object should somehow translate the logical offset to an actual offset. What is the sequence of operations pertaining to the buffer object when a stream object is seeked? Without buffering, it would probably be sufficient to just pass the seek command on to the source streambuf, clearing out the one character buffer. In your case, however, it is probably preferrable to keep the local buffer aligned on a 1024+CRC block boundary in the original streambuf. I think in your case, you will have to do some extra work. To begin with, you need to convert the seek position into an absolute position in the file -- in the case of seekoff, this probably means maintaining the current position in your streambuf. (To effectively implement seekoff from the end, you will also have to know the size of the file, since you'll have to backup just the right amount to read the final, partial block, in order to calculate the CRC on it.) Basically, then, you calculate the absolute seek position, break it up into block offset and block number (%1024 and /1024 respectively), then seek to the appropriate position in the source streambuf (block number * (1024+sizeof(CRC))), read and validate the block, and then set up the pointers into the buffer, adding the block offset to the start of the buffer in order to obtain the current address. -- James Kanze GABI Software http://www.gabi-soft.fr Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top dietmar_kuehl@yahoo.com Guest Posted: Fri Jan 21, 2005 10:34 am    Post subject: Re: STL's Stream and Buffer Object Interaction Efrat Regev wrote: Quote: 1. Suppose my buffer object detects an incorrect CRC (=> the file was corrupted). Then naturally, it could throw an exception. However, as Stroustrup writes in his book, not all users want stream objects to throw an exception (and for this, the clear() and exceptions() method are given). It is not the task of the stream buffer to suppress exceptions: this is what the stream methods have to do. It is perfectly reasonable to throw an exception from a stream buffer. Exceptions thrown during stream operations are handled by stream methods. That is, the stream methods encapsulate their work by a 'try' block. For example, the method for reading an 'int' looks something like this (without the template stuff as this is irrelevant here): Quote: std::ostream& std::ostream::operator>> (int& i) { try { // create 'sentry' and use 'num_get' to read the 'int' } catch (...) { try { setstate(std::ios_base::badbit); } catch (...) {} if (exceptions() & std::badbit) throw; } return *this; } If an exception is caught by a stream function, it sets 'badbit'. The actual setting is a little bit tricky as setting the flag will itself throw an exception but the user should receive the originally thrown one. Quote: 2. Suppose someone seeks on the stream. Then the buffer object should somehow translate the logical offset to an actual offset. What is the sequence of operations pertaining to the buffer object when a stream object is seeked? Are you sure you want to support seeking...? Typically, I don't support seeking with the stream buffers I write: it is rare that seeking is really required as most of the time the data is just read and then processed in-memory. For seeking support you would just overwrite 'seekoff()' and 'seekpos()': these methods are called when a user seeks in a stream. After the operation the read and/or write position should be at the indicated position. -- <mailto:dietmar_kuehl (AT) yahoo (DOT) com> <http://www.dietmar-kuehl.de/> <http://www.contendix.com> - Software Development & Consulting [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Mon Jan 24, 2005 8:15 pm    Post subject: Re: STL's Stream and Buffer Object Interaction [email]dietmar_kuehl (AT) yahoo (DOT) com[/email] wrote: Quote: Efrat Regev wrote: 1. Suppose my buffer object detects an incorrect CRC (= the file was corrupted). Then naturally, it could throw an exception. However, as Stroustrup writes in his book, not all users want stream objects to throw an exception (and for this, the clear() and exceptions() method are given). It is not the task of the stream buffer to suppress exceptions: this is what the stream methods have to do. That's what the standard says, but in practice, a lot of << and Quote: over user defined types were written in pre-standard days, and I suspect that for the most part, when they were ported to standard iostreams, the conversion was pretty much limited to changing ipfx/opfx and isfx/osfx into a use of the sentry object, and not adding a try block. At least, this is true of most of the code I've seen. So regardless of what the standard says, if you want your code to be robust, you will avoid letting an exception escape from a streambuf. Note too that g++ 2.95.2 is still the most widely used non-Windows compiler, and it doesn't support the standard iostreams, nor exceptions from streambuf's. Quote: It is perfectly reasonable to throw an exception from a stream buffer. Exceptions thrown during stream operations are handled by stream methods. s/are/should be/ My experience is that it is better not to count on it. Quote: That is, the stream methods encapsulate their work by a 'try' block. For example, the method for reading an 'int' looks something like this (without the template stuff as this is irrelevant here): | std::ostream& std::ostream::operator>> (int& i) { | try { | // create 'sentry' and use 'num_get' to read the 'int' | } | catch (...) { | try { setstate(std::ios_base::badbit); } catch (...) {} | if (exceptions() & std::badbit) | throw; | } | return *this; | } If an exception is caught by a stream function, it sets 'badbit'. The actual setting is a little bit tricky as setting the flag will itself throw an exception but the user should receive the originally thrown one. 2. Suppose someone seeks on the stream. Then the buffer object should somehow translate the logical offset to an actual offset. What is the sequence of operations pertaining to the buffer object when a stream object is seeked? Are you sure you want to support seeking...? Typically, I don't support seeking with the stream buffers I write: it is rare that seeking is really required as most of the time the data is just read and then processed in-memory. I agree, but in this case, I rather suspect a binary file. (He has to use binary mode with a filebuf, for example; otherwise, his CRC is likely to be corrupted from time to time. Unless he puts it in the file in hex.) So perhaps seeking is useful -- the file could be used to support fixed length records. Still, I certainly wouldn't implement it on the odd chance that someone might use it. Only if it is really needed. Quote: For seeking support you would just overwrite 'seekoff()' and 'seekpos()': these methods are called when a user seeks in a stream. After the operation the read and/or write position should be at the indicated position. Having adjusted for the number of bytes occupied by the CRC. This is really a bit awkward. Technically, streampos isn't a numberic type; I'm not even sure that streamoff is guaranteed to be a numeric type, and of course, numeric type or not, the standard allows any sort of mapping the implementation desires between the numeric value and the position in the file. In practice, of course, if portability is limited to Windows and Unix, streamoff probably is a numeric type representing the position in the file as a number of bytes from the beginning. Depending on what the poster is trying to do with the file, some compromise with regards to portability may be in order. (Of course, if it really is a binary file with fixed length records, one might reasonably ask if iostream were the appropriate tool for reading and writing it.) -- James Kanze GABI Software http://www.gabi-soft.fr Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top dietmar_kuehl@yahoo.com Guest Posted: Tue Jan 25, 2005 10:45 am    Post subject: Re: STL's Stream and Buffer Object Interaction [email]kanze (AT) gabi-soft (DOT) fr[/email] wrote: Quote: dietmar_kuehl (AT) yahoo (DOT) com wrote: It is not the task of the stream buffer to suppress exceptions: this is what the stream methods have to do. That's what the standard says, but in practice, a lot of << and over user defined types were written in pre-standard days, and I suspect that for the most part, when they were ported to standard iostreams, the conversion was pretty much limited to changing ipfx/opfx and isfx/osfx into a use of the sentry object, and not adding a try block. Actually, in no project I have been in was there any direct interaction between a stream buffer and a stream which was not written by me! The inserters or extractors for user defined types simply used the inserters or extractors for built-in types. I would guess that this is true for most projects since people are essentially entirely ignorant about stream buffers in the first place. It took several years even in news before you and I were not the only persons responding to the problems arising when trying to create a new stream by deriving from 'ostream' or 'istream' and "overriding" the extractors or inserters. Quote: So regardless of what the standard says, if you want your code to be robust, you will avoid letting an exception escape from a streambuf. I'd agree that you should minimize the situations where an exception is thrown from a stream buffer (or from anything else anyway) but I don't think that throwing an exception from s stream buffer should cause a real problem. ...and if it does it should arise during testing. Maybe the situation depends somewhat on the kind of project you are working on, though: if you are maintaining pretty aged code you would probably avoid anything beyond ARM's C++ but in newly created projects I would use modern C++ (not necessarily in the sense some book authors see C++ but in the sense C++ is used e.g. in this forum). Quote: Note too that g++ 2.95.2 is still the most widely used non-Windows compiler, and it doesn't support the standard iostreams, nor exceptions from streambuf's. The compiler actually supports stanard iostreams and exceptions: it is just the library shipping with this compiler which does not but there are alternatives. ... and if you are still using gcc 2.95.* it is time to switch to a newer version anyway. Of course, I know that this is not always and option (and in the project I'm currently working we are using a fairly old version of some compiler, too). -- <http://www.contendix.com> - Software Development & Consulting [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Tue Jan 25, 2005 7:28 pm    Post subject: Re: STL's Stream and Buffer Object Interaction [email]dietmar_kuehl (AT) yahoo (DOT) com[/email] wrote: Quote: kanze (AT) gabi-soft (DOT) fr wrote: [email]dietmar_kuehl (AT) yahoo (DOT) com[/email] wrote: It is not the task of the stream buffer to suppress exceptions: this is what the stream methods have to do. That's what the standard says, but in practice, a lot of and >> over user defined types were written in pre-standard days, and I suspect that for the most part, when they were ported to standard iostreams, the conversion was pretty much limited to changing ipfx/opfx and isfx/osfx into a use of the sentry object, and not adding a try block. Actually, in no project I have been in was there any direct interaction between a stream buffer and a stream which was not written by me! The inserters or extractors for user defined types simply used the inserters or extractors for built-in types. I would guess that this is true for most projects since people are essentially entirely ignorant about stream buffers in the first place. It took several years even in news before you and I were not the only persons responding to the problems arising when trying to create a new stream by deriving from 'ostream' or 'istream' and "overriding" the extractors or inserters. :-) That's actually pretty true. Even today, most user defined operator<< and >> only use other << and >>. And even today, most are pretty naïve about handling things like width() as well. The first is often pretty acceptable, or even the correct decision; if I were implementing a >> for complex, I'd certainly want to leverage off the >> for double. The second is unexcusable, but still wide spread. But where do you start? I find that there are still a lot of people writing loops with "while ( ! file.eof() )", which is even more fundamental. I think this is a serious problem, but I haven't the slightest idea how to go about solving it. I could write an article about how to do IO correctly, but people apparently aren't even reading what has already been written. Quote: So regardless of what the standard says, if you want your code to be robust, you will avoid letting an exception escape from a streambuf. I'd agree that you should minimize the situations where an exception is thrown from a stream buffer (or from anything else anyway) but I don't think that throwing an exception from s stream buffer should cause a real problem. ...and if it does it should arise during testing. Maybe the situation depends somewhat on the kind of project you are working on, though: if you are maintaining pretty aged code you would probably avoid anything beyond ARM's C++ but in newly created projects I would use modern C++ (not necessarily in the sense some book authors see C++ but in the sense C++ is used e.g. in this forum). It's actually a problem, because there isn't any other good way to indicate a hardware problem on input. If you just return EOF from underflow, the user thinks it is end of file. Given your previous remark, that almost all user defined operator >> and << are based on lower level operator<< and >> (which now that I think about it, I realize is the case), I think I'll change my mind, and agree with you. After all, it is about the only way to get badbit set on input, and if that's what you want (and it is in cases of hardware errors, etc.). There's still the problem of compatibility with classic iostream, but that can cause a number of other problems as well -- it's best to stick to the standard streams unless you're absolutely forced to do otherwise. Quote: Note too that g++ 2.95.2 is still the most widely used non-Windows compiler, and it doesn't support the standard iostreams, nor exceptions from streambuf's. The compiler actually supports stanard iostreams and exceptions: it is just the library shipping with this compiler which does not but there are alternatives. ... and if you are still using gcc 2.95.* it is time to switch to a newer version anyway. Of course, I know that this is not always and option (and in the project I'm currently working we are using a fairly old version of some compiler, too). Upgrading the compiler is obviously the preferred solution; the g++ 3.x branch is more than stable enough now to be used in production code. But I don't get to make that decision; some of the other programmers have done a bit of template code, which doesn't compile with a standard conforming compiler (no typename, quite a few non-dependant names that they need to make dependant, and at least one case of a circular reference using a non-dependant name which will involve reorganizing a header file). And management hasn't (yet) allocated a budget... The problem is, they pay me to get the code to work, not to find excuses. Even if the real problems are of their own making. -- James Kanze GABI Software http://www.gabi-soft.fr Conseils en informatique orientée objet/ Beratung in objektorientierter Datenverarbeitung 9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Page 1 of 1   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.