C++Talk.NET C++ language newsgroups   Archives   FAQ   Search   Memberlist   Usergroups   Register   Profile   Log in to check your private messages   Log in  creating a zip file stream          C++Talk.NET Forum Index -> C++ Language (Moderated) View previous topic :: View next topic   Author Message John Dill Guest Posted: Wed Jan 28, 2004 1:33 pm    Post subject: creating a zip file stream I have an implementation of a stream which wraps around some compression library; but when using file streams, there is a use case which causes problems. The problem with using my compression streambuf with a file stream is as follows (simplified): { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... file.close(); } When the file is closed previous to the zip streambuf destructor, the output is truncated because the destructor flushes the zip streambuf buffer to the destination, which no longer exists. This can be solved by explicitly calling a flush command, which is called "finish" in my library, to flush all the internal buffers of the internal zip work buffers. { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... zs.finish(); // flush internal zip buffers to destination file.close(); } The problem is that it requires an explicit call, when it should be called indirectly through the close command. The solution is to encapsulate this functionality into a zip fstream type, where close would call finish. This would transform the code to: { zip_ofstream file( "test.txt" ); zfs << "Some text..." ... file.close(); } The issue is how to implement this class(es) from my current source without a lot of code duplication (maintenance cost). The simplistic way to solve this is to copy my zip_ostreambuf to zip_ofilebuf and add the file object and interface (open/close/is_open), but I want to avoid this if possible. Any suggestions? John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Ulrich Eckhardt Guest Posted: Wed Jan 28, 2004 8:11 pm    Post subject: Re: creating a zip file stream John Dill wrote: Quote: I have an implementation of a stream which wraps around some compression library; but when using file streams, there is a use case which causes problems. The problem with using my compression streambuf with a file stream is as follows (simplified): { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... file.close(); } When the file is closed previous to the zip streambuf destructor, the output is truncated because the destructor flushes the zip streambuf buffer to the destination, which no longer exists. This can be solved by explicitly calling a flush command, which is called "finish" in my library, to flush all the internal buffers of the internal zip work buffers. { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... zs.finish(); // flush internal zip buffers to destination file.close(); } Hmmm, I would have designed this differently. The 'proper' thing to hook into is the streambuffer, not the stream. That streambuffer is responsible for two things: converting the internal characterset to external bytes and doing the raw IO. The stream itself is rather responsible for formatting the given data(that is, delegating to the locale). In order to write a compressing stream, I'd first write a compressing streambuffer. To keep the amount of code small, you can then delegate to another streambuffer for the 'real' IO (you'll just have to take care that it doesn't perform another charset conversion). To use it, I'd take this approach: ofstream out("file.name"); zip_streambuf zbuf; zbuf.set_output(out.rdbuf()); out.rdbuf(&zbuf); .... If you incorporate the compression and the file-IO into one streambuffer (derivation?) you could then do this: zip_filebuf zfbuf("file.name"); ostream out(&zbuf);// not oFstream! .... The next possible step is to aggregate such a buffer with a stream (which is pretty much all that an fstream is!) and perform the setting of the streambuffer in the constructor. Two more notes: - I seem to remember that there was a boost library[proposal] for cascading streams, which might be helpful. - "C++ IOStreams and Locales" by Langer and Kreft is sure worth reading. Uli [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top tom_usenet Guest Posted: Wed Jan 28, 2004 8:46 pm    Post subject: Re: creating a zip file stream On 28 Jan 2004 08:33:18 -0500, [email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote: Quote: I have an implementation of a stream which wraps around some compression library; but when using file streams, there is a use case which causes problems. The problem with using my compression streambuf with a file stream is as follows (simplified): { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... file.close(); } When the file is closed previous to the zip streambuf destructor, the output is truncated because the destructor flushes the zip streambuf buffer to the destination, which no longer exists. Right. The obvious solution is to not call close! Why are you calling it anyway? The destructors would have done it for you anyway, flushing the zip stream first and then flushing and closing the file. This can be solved Quote: by explicitly calling a flush command, which is called "finish" in my library, to flush all the internal buffers of the internal zip work buffers. Can't you just do: zs.flush(); ? Quote: { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... zs.finish(); // flush internal zip buffers to destination file.close(); } The problem is that it requires an explicit call, when it should be called indirectly through the close command. No, not really. The file stream is a separate object. If you start messing around with it while the zip_ostream is still using it, you'd expect things to go wrong. The solution is to Quote: encapsulate this functionality into a zip fstream type, where close would call finish. This would transform the code to: { zip_ofstream file( "test.txt" ); zfs << "Some text..." ... file.close(); } If you really need the feature... Quote: The issue is how to implement this class(es) from my current source without a lot of code duplication (maintenance cost). The simplistic way to solve this is to copy my zip_ostreambuf to zip_ofilebuf and add the file object and interface (open/close/is_open), but I want to avoid this if possible. Any suggestions? Presumably your zip_ostreambuf wraps a streambuf*? I don't see the problem - you should be able to use zip_ostreambuf without modification. e.g. class zip_ofstream: public std::ostream { zip_ostreambuf zipbuf; std::filebuf filebuf; public: zip_ofstream() :std::ostream(&zipbuf), //not yet constructed but ok zipbuf(&filebuf)//ok if zipbuf const only uses address, not object { } //other constructors to open file void open(char const* filename (and open mode)) { close(); filebuf.open(filename); zipbuf.reset(&filebuf); //or similar? } void close() { zipbuf.pubsync(); //or finish() filebuf.close(); } }; I don't understand the need for a new streambuf type, but I may be missing something. Tom C++ FAQ: http://www.parashift.com/c++-faq-lite/ C FAQ: http://www.eskimo.com/~scs/C-faq/top.html [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Sean Kelly Guest Posted: Thu Jan 29, 2004 9:04 am    Post subject: Re: creating a zip file stream Ulrich Eckhardt wrote: Quote: Hmmm, I would have designed this differently. The 'proper' thing to hook into is the streambuffer, not the stream. True enough, but using streams in this way is a bit more convenient. No need to deal with flushing and putting things back the way they should be before destruction, etc. Quote: In order to write a compressing stream, I'd first write a compressing streambuffer. To keep the amount of code small, you can then delegate to another streambuffer for the 'real' IO (you'll just have to take care that it doesn't perform another charset conversion). To use it, I'd take this approach: ofstream out("file.name"); zip_streambuf zbuf; zbuf.set_output(out.rdbuf()); out.rdbuf(&zbuf); All the logic goes in the streambuffer, but I've found it useful to provide a stream type that acts as an adaptor and works in pretty much the same way the OP described. The advantage is that streams can be chained together very conveniently: ofstream ofile( "file.name" ); ozstream ozip( ofile ); obase64 o64( ozip ); And the same for input streams. The actual stream code in each case really doesn't amount to more than a constructor and perhaps a few functions that call equivalent functions in the buffer object. The buffers likely just call read() and write() to avoid formatting issues as stuff filters through. Quote: "C++ IOStreams and Locales" by Langer and Kreft is sure worth reading. Great book. Streams have some odd quirks. Sean [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Dietmar Kuehl Guest Posted: Thu Jan 29, 2004 9:50 am    Post subject: Re: creating a zip file stream [email]john-dill (AT) uiowa (DOT) edu[/email] (John Dill) wrote: Quote: { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... file.close(); } When the file is closed previous to the zip streambuf destructor, the output is truncated because the destructor flushes the zip streambuf buffer to the destination, which no longer exists. OK, so this essentially means, your 'zip_ostream' obtains the stream buffer from the file stream and passes it on to your internal stream buffer. To avoid the problem with premature closing of the file, you simply add a constructor taking a file name to 'zip_ostream' which internally creates the needed 'std::filebuf' and opens it using the 'filebuf's 'open()' method. You also add a 'close()' function which calls the internal file buffer's 'close()' method if present. If 'zip_ostream' cannot be modified, derive from this class and construct it with a member 'std::ofstream' (unless it already has a constructor taking a stream buffer in which case only a 'std::filebuf' should constructed). Forwarding the 'close()' method should be no problem. Deriving form an 'std::ostream' or a class derived thereof for initial setup is a reasonable thing to do. However, you should never derive from such a class and attempt overloading the output operators ('<<'). -- Phaidros eaSE - Easy Software Engineering: <http://www.phaidros.com/> [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top John Dill Guest Posted: Thu Jan 29, 2004 7:03 pm    Post subject: Re: creating a zip file stream Quote: Hmmm, I would have designed this differently. The 'proper' thing to hook into is the streambuffer, not the stream. That streambuffer is responsible for two things: converting the internal characterset to external bytes and doing the raw IO. The stream itself is rather responsible for formatting the given data(that is, delegating to the locale). Actually, all the compression stuff is implemented in a streambuf, but the user interfaces through the ostream. I should have been a little more explicit. Quote: In order to write a compressing stream, I'd first write a compressing streambuffer. To keep the amount of code small, you can then delegate to another streambuffer for the 'real' IO (you'll just have to take care that it doesn't perform another charset conversion). To use it, I'd take this approach: ofstream out("file.name"); zip_streambuf zbuf; zbuf.set_output(out.rdbuf()); out.rdbuf(&zbuf); ... If you incorporate the compression and the file-IO into one streambuffer (derivation?) you could then do this: zip_filebuf zfbuf("file.name"); ostream out(&zbuf);// not oFstream! ... The next possible step is to aggregate such a buffer with a stream (which is pretty much all that an fstream is!) and perform the setting of the streambuffer in the constructor. This is actually what I am trying to figure out ;) Quote: Two more notes: - I seem to remember that there was a boost library[proposal] for cascading streams, which might be helpful. - "C++ IOStreams and Locales" by Langer and Kreft is sure worth reading. I've been wanting to read that, but haven't had the time lately :/ Best, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top John Dill Guest Posted: Thu Jan 29, 2004 7:11 pm    Post subject: Re: creating a zip file stream Quote: I don't understand the need for a new streambuf type, but I may be missing something. I just implemented the file stream similar to your design and it works great. I don't have a 'reset' functionality so I will look into the use case of opening a file which is already open. Best, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top John Dill Guest Posted: Thu Jan 29, 2004 7:18 pm    Post subject: Re: creating a zip file stream Quote: OK, so this essentially means, your 'zip_ostream' obtains the stream buffer from the file stream and passes it on to your internal stream buffer. To avoid the problem with premature closing of the file, you simply add a constructor taking a file name to 'zip_ostream' which internally creates the needed 'std::filebuf' and opens it using the 'filebuf's 'open()' method. You also add a 'close()' function which calls the internal file buffer's 'close()' method if present. My zip_ostream is currently implemented such that it derives from the zip_ostreambuf and std::ostream, much like James Kanze's filtering streambufs. I would prefer however to separate file operations from zip_ostream and place them in it's own class (zip_ofstream), to try to mimic the standard library as best as I can. If I understand you correctly, I would still derive from zip_ostreambuf and std::ostream to make zip_ofstream, but then implement my own open/close/is_open to handle the connection of the filebuf to the zip_ostreambuf? Quote: If 'zip_ostream' cannot be modified, derive from this class and construct it with a member 'std::ofstream' (unless it already has a constructor taking a stream buffer in which case only a 'std::filebuf' should constructed). Forwarding the 'close()' method should be no problem. Deriving form an 'std::ostream' or a class derived thereof for initial setup is a reasonable thing to do. However, you should never derive from such a class and attempt overloading the output operators ('<<'). Been down that path before. Definitely evil :) Thanks, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top kanze@gabi-soft.fr Guest Posted: Thu Jan 29, 2004 7:19 pm    Post subject: Re: creating a zip file stream Ulrich Eckhardt <doomster (AT) knuut (DOT) de> wrote Quote: John Dill wrote: I have an implementation of a stream which wraps around some compression library; but when using file streams, there is a use case which causes problems. The problem with using my compression streambuf with a file stream is as follows (simplified): { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... file.close(); } When the file is closed previous to the zip streambuf destructor, the output is truncated because the destructor flushes the zip streambuf buffer to the destination, which no longer exists. This can be solved by explicitly calling a flush command, which is called "finish" in my library, to flush all the internal buffers of the internal zip work buffers. { std::ofstream file( "test.txt" ); zip_ostream zs( file ); zs << "Some text..." ... zs.finish(); // flush internal zip buffers to destination file.close(); } Hmmm, I would have designed this differently. The 'proper' thing to hook into is the streambuffer, not the stream. That streambuffer is responsible for two things: converting the internal characterset to external bytes and doing the raw IO. The stream itself is rather responsible for formatting the given data(that is, delegating to the locale). In order to write a compressing stream, I'd first write a compressing streambuffer. To keep the amount of code small, you can then delegate to another streambuffer for the 'real' IO (you'll just have to take care that it doesn't perform another charset conversion). This is the classical filtering streambuf idiom. Quote: To use it, I'd take this approach: ofstream out("file.name"); zip_streambuf zbuf; zbuf.set_output(out.rdbuf()); out.rdbuf(&zbuf); ... There are simpler solutions. The usual solution involves providing a wrapper ostream for the filtering streambuf, which allows writing something like: filebuf dest( "file.name", ios::out | ios::binary ) ; // Don't forget the binary... dest.imbue( locale( "C" ) ) ; // Nor this, it's NOT the default zip_ostream out( dest ) ; // ... Take a look at some of the code at my site ([url]www.gabi-soft.fr)[/url]. Of course, this doesn't prevent someone from closing dest before the last data from out has been flushed. I'm tempted to say, that's life. One of the preconditions of a filtering streambuf is that the actual streambuf outlive it. (There are exceptions, and some of my filtering streambufs change the actual source or destination on the fly. But it is always the filtering streambuf which does it.) The real problem isn't so much that you need to avoid character translation in the filebuf (just don't do it). The real problem is that YOU have to do the character translation in your filtering streambuf, before you start compressing the data. What we really need is a translating filtering streambuf, and then chain the streambufs: filebuf file( "file.name", ios::out | ios::binary ) ; translating_streambuf xlate( file ) ; zip_ostream out( xlate ) ; If your applications need this a lot, of course, you could encapsulate it in a zip_ofstream. Quote: If you incorporate the compression and the file-IO into one streambuffer (derivation?) you could then do this: zip_filebuf zfbuf("file.name"); ostream out(&zbuf);// not oFstream! ... The next possible step is to aggregate such a buffer with a stream (which is pretty much all that an fstream is!) and perform the setting of the streambuffer in the constructor. Two more notes: - I seem to remember that there was a boost library[proposal] for cascading streams, which might be helpful. I don't know the status of the Boost proposal, but there is working code at my site. Has been whenever the site has existed. Quote: - "C++ IOStreams and Locales" by Langer and Kreft is sure worth reading. One last comment: when I had exactly the same problem (outputting a gzipped file), I found an even easier solution. I simply wrote a pipebuf (like a filebuf, but to a pipe -- fairly simple, and generally useful), and then used a pipebuf( "gzip > file.name", ios::out ). The code was a lot simpler. It was also a lot faster -- the actual compression took place while my process was blocked reading the next data. -- James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr Conseils en informatique orientée objet/ http://www.gabi-soft.fr Beratung in objektorientierter Datenverarbeitung 11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16 [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top John Dill Guest Posted: Thu Jan 29, 2004 7:20 pm    Post subject: Re: creating a zip file stream Quote: Right. The obvious solution is to not call close! Why are you calling it anyway? The destructors would have done it for you anyway, flushing the zip stream first and then flushing and closing the file. That is correct, but I want to guarantee that the closing of the file stream finishes whatever is in the internal compression buffers. I have seen plenty of file.close() calls in maintenance code I've had to work on, so this necessitates the functionality. I tend to invest my time in robustness rather than user complaints if possible :) Quote: This can be solved by explicitly calling a flush command, which is called "finish" in my library, to flush all the internal buffers of the internal zip work buffers. Can't you just do: zs.flush(); ? It depends on how the flush is implemented. Depending on the compression library (like zlib or bzip2), you may flush the input to the compression buffers, but will not generate all the compressed output at that time. The compression library has its own internal buffers which it uses for pattern matching and the like. If you force the all the output to be generated at the flush, you will get less compression due to emptying the internal compression buffers. My sync basically flushes the input to the compression structure, calls 'deflate' (a library compression call), and then takes whatever output is in the output buffer and sends it to the destination. It does not empty the zip stream internal buffers. That is what 'finish' does. It flushes any remaining chars in the streambuf to the compression buffers, and continually calls 'deflate' to empty the internal buffers. Quote: Presumably your zip_ostreambuf wraps a streambuf*? Yep. Quote: I don't see the problem - you should be able to use zip_ostreambuf without modification. e.g. class zip_ofstream: public std::ostream { zip_ostreambuf zipbuf; std::filebuf filebuf; public: zip_ofstream() :std::ostream(&zipbuf), //not yet constructed but ok zipbuf(&filebuf)//ok if zipbuf const only uses address, not object { } //other constructors to open file void open(char const* filename (and open mode)) { close(); filebuf.open(filename); zipbuf.reset(&filebuf); //or similar? } void close() { zipbuf.pubsync(); //or finish() filebuf.close(); } }; I'll give this some thought. My first tendency was to try to mimic the std::filebuf implementation, using my compression streambuf but replace sending to the destination with sending to a FILE type structure. Best, John [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] Back to top Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First         C++Talk.NET Forum Index -> C++ Language (Moderated) All times are GMT Page 1 of 1   Jump to: Select a forum C/C++----------------C LanguageC Language (Moderated)C++ Language (Moderated)C++ language (comp.lang.c++)C++ language, library and standards Non-English C/C++ Newsgroups----------------C++ (German)C++ (French)  You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum Powered by phpBB © 2001, 2006 phpBB Group SEO toolkit © 2004-2006 webmedic.