troubled by std::wifstream::open(const char*) - C++ Language (Moderated)

C++Talk.NET
C++ language newsgroups

Archives

FAQ

Memberlist

Usergroups

Profile

troubled by std::wifstream::open(const char*)
Goto page 1, 2 Next

C++Talk.NET Forum Index -> C++ Language (Moderated)

View previous topic :: View next topic

Author

Message

Seagull Manager
Guest

Posted: Tue Dec 02, 2003 5:06 pm Post subject: troubled by std::wifstream::open(const char*)

Maybe this is someone everyone knows about already, and considers completely boring, but it's been bugging me, so here goes: The standard has it that fstreams, even wide fstreams, can only be opened using 8-bit byte strings for filenames. If a filename contains, say, non-western letters in the name, it may be impossible to open it using an fstream without some kind of shim, e.g.: template<typename unicode_string> const std::string convert_wide_string_to_something_understood_by_fstreams(const unicode_string&) { // do something OS-dependent, such as encode in utf-8, or look up an alternative file name } //... std::wifstream my_file; //... my_file.open(convert_wide_string_to_something_understood_by_fstreams(gujerat i_or_hangul_or_kanji_maybe_file_name).c_str()); If I want my application to use fstreams and at the same time be useful internationally, I'll have to shim all my calls to "open" this way. This strikes me as a little clunky (even if I don't use the ridiculously long names I've used here for the discussion). Someone suggested when the C++ standard was in the draft stages, that fstream template classes be modified to have constructors and open() methods that took template<CharType> parameters, which would have put the shim onus on the library implementer, but the idea was kicked into the long grass (a.k.a. "Future"). That was about five years ago. Right now, most common operating systems have support for unicode filenames built in, and computer use outside the ISO-8859-15* region continues to boom. Could it be a sign that the standard is ripe for update now? I've heard it suggested that the next revision of the standard will happen around 2008. Will that be too late for the language? I fear it might be. Niggles like the one I've just mentioned could turn programmers away. ::Bruce_Attah; (*aside: ISO-8859-15, a.k.a. "Latin-9" is probably the character set we should all be using by default for Western languages; it is an update on ISO-8859-1 that renders the latter essentially obsolete - I wonder why so few do?) [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] [ comp.std.c++ is moderated. To submit articles, try just posting with ] [ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ] [ --- Please see the FAQ before posting. --- ] [ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Hyman Rosen
Guest

Posted: Wed Dec 03, 2003 4:05 pm Post subject: Re: troubled by std::wifstream::open(const char*)

There's nothing to stop a vendor from offering an extension which allows open on wide strings. Note that the wideness of the file contents is completely orthogonal to the wideness of the file name. [ See http://www.gotw.ca/resources/clcm.htm for info about ] [ comp.lang.c++.moderated. First time posters: Do this! ] [ comp.std.c++ is moderated. To submit articles, try just posting with ] [ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ] [ --- Please see the FAQ before posting. --- ] [ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Jonathan Turkanis
Guest

Posted: Wed Dec 03, 2003 4:05 pm Post subject: Re: troubled by std::wifstream::open(const char*)

"Seagull Manager" <seagull.manager (AT) nospamthanksbecauseisayso (DOT) demon.co.uk>
wrote in message

Quote:

The standard has it that fstreams, even wide fstreams, can only be opened
using 8-bit byte strings for filenames. If a filename contains, say,
non-western letters in the name, it may be impossible to open it using an
fstream without some kind of shim, e.g.:

First, the type of characters which can appear in a file name is separate
from the type of data the file contains. E.g., a file whose name contains
unusual characters could contain pure binary data, and a text file using a
multibyte encoding could have a pure ASCII name. So the question applies as
much to narrow-character streams as it does to wide-character streams.

For the main question, the Boost.Filesystem FAQ gives a good answer. See:
http://www.boost.org/libs/filesystem/doc/faq.htm (Why aren't wide-character
names supported?)

Jonathan

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Eugene Gershnik
Guest

Posted: Thu Dec 04, 2003 3:55 pm Post subject: Re: troubled by std::wifstream::open(const char*)

Jonathan Turkanis wrote:

Quote:

"Seagull Manager"
[email]seagull.manager (AT) nospamthanksbecauseisayso (DOT) demon.co.uk[/email]> wrote in
message
The standard has it that fstreams, even wide fstreams, can only
be opened using 8-bit byte strings for filenames. If a filename
contains, say, non-western letters in the name, it may be
impossible to open it using an fstream without some kind of
shim, e.g.:

Quote:

For the main question, the Boost.Filesystem FAQ gives a good
answer. See: http://www.boost.org/libs/filesystem/doc/faq.htm (Why
aren't wide-character names supported?)

Hmmm. On my OS the native character type is wchar_t. Conversions to char
could be ambiguous or sometimes impossible (for example when the file comes
from another system and its name contains characters from a language for
which mine dosen't have a conversion table installed). So on my system the
standard fstream simply isn't usable not to speak about portable :-(

Eugene

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Chuck McDevitt
Guest

Posted: Sun Dec 07, 2003 4:27 pm Post subject: Re: troubled by std::wifstream::open(const char*)

"Seagull Manager" <seagull.manager (AT) nospamthanksbecauseisayso (DOT) demon.co.uk>
wrote in message news:bqfmlt$64r$1$830fa79d (AT) news (DOT) demon.co.uk...

Quote:

Maybe this is someone everyone knows about already, and considers
completely
boring, but it's been bugging me, so here goes:

The standard has it that fstreams, even wide fstreams, can only be opened
using 8-bit byte strings for filenames. If a filename contains, say,
non-western letters in the name, it may be impossible to open it using an
fstream without some kind of shim, e.g.:

I agree 100% that this is wrong. The c++ language supports wchar_t.
In standard iostreams, if I write wchar_t strings to an iostream, there is a
codecvt
that can handle converting my wchar_t strings to char strings.

Given that, why can't the same apply to file names?

It would be easy enough to allow for wchar_t string name for files & paths,
with
the caveot that the conversion to the underlying file system's names and
paths is OS dependent.

This would be far more portable than how it is today.
Today, if you have a unicode wchar_t string containing a file name, you need
to code into
your application the knowledge of how to convert that to the OS's underlying
format.
For example, if I have a Japanese file name, I need to know, based on my os:

1) if japanese windows, convert to windows CP 932 (the OS will convert
back from CP 932 to Unicode).
2) if japanese Unix, convert to EUC.
3) if unicode unix/linux, convert to UTF8
4) if IBM mainframe, convert to something totally different.
5) if non-japanese windows, avoid iostreams, and open the file directly
from the unicode name,
because there is no char * string that I can convert the japanese
name to that will work,
even though the underlying filesystem can handle the name just fine.

Ugh.

My advise: If you need to write internationalized applications, avoid
iostreams.
They just cause trouble.

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

James Kanze
Guest

Posted: Sun Dec 07, 2003 4:27 pm Post subject: Re: troubled by std::wifstream::open(const char*)

Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> writes:

Quote:

Jonathan Turkanis wrote:
"Seagull Manager"
[email]seagull.manager (AT) nospamthanksbecauseisayso (DOT) demon.co.uk[/email]> wrote in
message
The standard has it that fstreams, even wide fstreams, can only
be opened using 8-bit byte strings for filenames. If a filename
contains, say, non-western letters in the name, it may be
impossible to open it using an fstream without some kind of
shim, e.g.:

For the main question, the Boost.Filesystem FAQ gives a good
answer. See: http://www.boost.org/libs/filesystem/doc/faq.htm (Why
aren't wide-character names supported?)

Hmmm. On my OS the native character type is wchar_t. Conversions to
char could be ambiguous or sometimes impossible (for example when
the file comes from another system and its name contains characters
from a language for which mine dosen't have a conversion table
installed). So on my system the standard fstream simply isn't usable
not to speak about portable Sad

This sounds like an impedence mismatch between your compiler and your
system.

C++ considers that the system has a single character type. It makes no
assumptions about this type, except that it is a single type. Any time
information is passed to or from the system, this is the type that is
used. Thus, filebuf requires that the name of the file have this type,
and that all reads and writes to and from the file are done in this
type. This type is called "char". And I repeat, not only must all
filenames be char[], all transfers to and from the system take place as
transfers of char -- a wfstream translates multi-byte sequences into
wide characters.

By intention, a wide character stream will not use a multibyte encoding.
If there is no possiblity of multibyte encodings in a narrow character
stream, there is no difference between the two, and there is no reason
to use narrow character streams. (This is not the case for any OS that
I know, however: most modern OS's use 8 bit characters, with multi-byte
encodings, usually UTF-8, being used for international characters. The
one exception is Windows, which uses 16 bit characters with the
multibyte encoding UTF-16.)

So far, so good. About the only problem I see is that C and C++ require
that the narrow character type be the smallest addressable type
supported by the implementation. Which does cause a problem on Windows;
for many reasons, it is desirable to have 8 bit addressable types in C
and C++, even though the logical implementation would be to use 16 bit
narrow characters and 32 bit wide characters.

Regretfully, the only solution I can propose is to use the 8 bit
interface to Windows (which I think uses UTF-8, which is about as
portable as you are going to get for international characters).

--
James Kanze mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France +33 1 41 89 80 93

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Seagull Manager
Guest

Posted: Mon Dec 08, 2003 10:54 pm Post subject: Re: troubled by std::wifstream::open(const char*)

"Eugene Gershnik" <gershnik (AT) nospam (DOT) hotmail.com> wrote

Quote:

Hmmm. On my OS the native character type is wchar_t.

Absolutely (although to be pendantic, it is what your compiler vendor
conveniently defines as a wchar_t).

Quote:

Conversions to char
could be ambiguous or sometimes impossible (for example when the file
comes
from another system and its name contains characters from a language for
which mine dosen't have a conversion table installed). So on my system the
standard fstream simply isn't usable not to speak about portable Sad

Well, if the file has arrived from another file system, and its name has
somehow reached you without being correctly decoded into a wide character
representation that your code understands, then you've probably got a
problem regardless of what programming language or library you're using. But
that, of course, is a problem with all networked data, not just file names.

But you are quite right, by one simple omission in the standard, fstream has
been rendered next to useless to anyone who wants to write robust code on
systems that support wide character file names.

Quote:

Eugene

void Bruce_Attah(L"$B%U%!%$%%!&%M!<%`(B");

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Eugene Gershnik
Guest

Posted: Mon Dec 08, 2003 10:54 pm Post subject: Re: troubled by std::wifstream::open(const char*)

James Kanze wrote:

Quote:

Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> writes:
Hmmm. On my OS the native character type is wchar_t. Conversions
to char could be ambiguous or sometimes impossible (for example
when the file comes from another system and its name contains
characters from a language for which mine dosen't have a
conversion table installed). So on my system the standard
fstream simply isn't usable not to speak about portable :-(

This sounds like an impedence mismatch between your compiler and
your system.

C++ considers that the system has a single character type. It
makes no assumptions about this type, except that it is a single
type. Any time information is passed to or from the system, this
is the type that is used. Thus, filebuf requires that the name of
the file have this type, and that all reads and writes to and from
the file are done in this type. This type is called "char". And I
repeat, not only must all filenames be char[], all transfers to and
from the system take place as transfers of char -- a wfstream
translates multi-byte sequences into wide characters.

I see the logic of this but unfortunately this makes abstract C++
incompatible with the existing OS interface of my system. The system is
Windows as you probably guessed. The basic OS interface on this system has 2
distinct character types mapped to char and wchar_t by all C++
implementatons. For example the API used to open a file accepts whcar_t*
while an API to obtain the host IP from its name accepts char*. There is a
compatibility layer that allows me to open a file by passing it a char* thus
allowing an application to live in a single character type world.
Unfortunately usage of this layer makes I18N quite complicated.

Quote:

By intention, a wide character stream will not use a multibyte
encoding. If there is no possiblity of multibyte encodings in a
narrow character stream, there is no difference between the two,
and there is no reason to use narrow character streams. (This is
not the case for any OS that I know, however: most modern OS's use
8 bit characters, with multi-byte encodings, usually UTF-8, being
used for international characters. The one exception is Windows,
which uses 16 bit characters with the multibyte encoding UTF-16.)

It is fixed width UCS-16 I think.

Quote:

The problem is it isn't UTF-8. When using 8-bit interface to Windows the
character strings are encoded in a language dependent manner. Each machine
contains a collection of conversion tables from each of supported languages
encodings to and from UCS-16. When you invoke an 8-bit wrapper over UCS-16
function one of these tables is used to convert the strings. Which table is
going to be used is determined by a mechanism similar to C locales. The net
result of this are huge problems with I18N portability. There are ways to
overcome the problems but they require heroic efforts from the application.
Using wide-character APIs solves most of these problems and it fits quite
easily into C++. I can write the whole application using wchar_t only if I
don't use any of the few APIs that require chars (only some socket function
do this). The perpetual problem are fstream::open, exception::what() and
ctors of <stdexcept> classes. I can see how the standard library on my
machine could provide extensions to solve most of the problems. After all we
have _wfopen(const wchar_t *filename, const wchar_t *mode) in the same
library. So from my POV this is entirely library QoI issue.
The thing I don't agree with is the argument that having only
fstream::open(char *) somehow makes the library and applications that are
using it more portable.

Eugene

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Seagull Manager
Guest

Posted: Mon Dec 08, 2003 10:54 pm Post subject: Re: troubled by std::wifstream::open(const char*)

"Hyman Rosen" <hyrosen (AT) mail (DOT) com> wrote

Quote:

There's nothing to stop a vendor from offering an extension
which allows open on wide strings.

That just means that (a) few vendors will offer such an extension
(especially as OS vendors who also sell compilers may have an incentive to
encourage their customers to write non-portable code, which is most
effectively done by steering programmers to proprietary libraries and away
from standard ones - a weakness in the standard is a boon to vendors who
choose such a strategy), so that (b) if I can get hold of an implementation
that does offer such an extension, and I use it, my code will probably no
longer be portable between compilers, and (c) the implementation will be
underused, and therefore perhaps undertested.

Yet opening files is fundamental, and all common operating systems (even OSs
for mobile devices) support Unicode (or at least the 16-bit subset of
Unicode that is UCS-2) file names. Furthermore, many users around the world
routinely save their files using non-8-bit file names. Therefore,
std::fstream is not fully functional on any common operating system. It is
possible that on some implementations of std::fstream, there is no reliable
workaround available. In that case, if one ships a product that relies on
std::fstream to open files, it is pretty much guaranteed that one will at
some stage receive a support call saying "Someone sent me this important
file, and I can't open it with your app", and fixing the problem will
potentially be hugely expensive (perhaps involving the rewriting of all or
most code that uses iostreams).

It would have been a relatively minor matter for the standard to define the
spec of a function that opens files using wchar_t, but it failed to do so -
I would argue that the myopia or parochialism must have had something to do
with this. It seems to me that there should be an update of the C++ standard
to fix issues like this as soon as possible, then at a later date more
ambitious amendments can be made. Without such measures, I suspect that C++
will very likely be marginalized over the next few years.

wchar_t Bruce_Attah(char);

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Alf P. Steinbach
Guest

Posted: Tue Dec 09, 2003 3:52 pm Post subject: Re: troubled by std::wifstream::open(const char*)

On Mon, 8 Dec 2003 22:54:10 +0000 (UTC), Seagull Manager <seagull.manager (AT) nospamthanksbecauseisayso (DOT) demon.co.uk> wrote:

Quote:

Therefore,
std::fstream is not fully functional on any common operating system.

<rant>
Well, that's not just a character representation issue, but also one of performance,
and of simplicity, and of general exception safety (most of all the safety of typical
_client code_), and much else, e.g. the related silliness where you can't write a
pure 'cat' program without using e.g. Posix functions. In short, the i/o part of the
library, including the locale support, messages, etc., sucks. It's useful for small
test programs and exercises. It's a crying shame, because originally that part wasn't
at all bad. But it's been abstracted to death, as if templatization could make it STL.
</rant>

Having concluded that, what is the remedy? Simple, one should use something else.
And perhaps someday someone will come up with the STL of file and stream handling.

I absolutely don't think it's a good idea to make the standard streams _apparently_
useful by tacking on the most desperately needed real-world functionality.

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

kanze@gabi-soft.fr
Guest

Posted: Tue Dec 09, 2003 10:47 pm Post subject: Re: troubled by std::wifstream::open(const char*)

Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> wrote

Quote:

James Kanze wrote:
Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> writes:
Hmmm. On my OS the native character type is wchar_t. Conversions
to char could be ambiguous or sometimes impossible (for example
when the file comes from another system and its name contains
characters from a language for which mine dosen't have a
conversion table installed). So on my system the standard fstream
simply isn't usable not to speak about portable :-(

This sounds like an impedence mismatch between your compiler and
your system.

C++ considers that the system has a single character type. It makes
no assumptions about this type, except that it is a single type.
Any time information is passed to or from the system, this is the
type that is used. Thus, filebuf requires that the name of the file
have this type, and that all reads and writes to and from the file
are done in this type. This type is called "char". And I repeat,
not only must all filenames be char[], all transfers to and from the
system take place as transfers of char -- a wfstream translates
multi-byte sequences into wide characters.

I see the logic of this but unfortunately this makes abstract C++
incompatible with the existing OS interface of my system. The system
is Windows as you probably guessed. The basic OS interface on this
system has 2 distinct character types mapped to char and wchar_t by
all C++ implementatons. For example the API used to open a file
accepts whcar_t* while an API to obtain the host IP from its name
accepts char*. There is a compatibility layer that allows me to open
a file by passing it a char* thus allowing an application to live in a
single character type world. Unfortunately usage of this layer makes
I18N quite complicated.

I rather suspected it was Windows:-). While the people at Microsoft
were certainly trying to do the right thing, I'm not sure that I'm happy
with the results. As you say, it is a fact of life that you need 8 bit
characters, since that is, for better or worse, what is used everywhere,
and in these days of connectivity, you have to be compatible with
everywhere. Thus, the interface to obtain the host IP uses char. The
attempt to use 16 bit characters for all internal interfaces is
laudable. But in practice, I have the feeling that it creates more
problems that it solves. To begin with, it requires that you, the
programmer, deal with two different character types. Forget the C++
problem for a moment, and consider that you have no problem using the
wide character interfaces in your application. You read a filename
from a file (as wide characters), and pass it to the API to open the
file. Excellent. You read a URL from a file (as wide characters), and
pass it to the API. Oops.

I'm not sure what the best solution should be, either from the system
point of view, or from the point of view of C++. What is happening,
however, best or not (and it doesn't seem that bad a priori) is that
everything external to the program is moving (albeit very slowly) to
UTF-8. So an 8 bit, converting interface, makes sense.

But none of this helps with your practical problem.

Quote:

By intention, a wide character stream will not use a multibyte
encoding. If there is no possiblity of multibyte encodings in a
narrow character stream, there is no difference between the two, and
there is no reason to use narrow character streams. (This is not
the case for any OS that I know, however: most modern OS's use 8 bit
characters, with multi-byte encodings, usually UTF-8, being used for
international characters. The one exception is Windows, which uses
16 bit characters with the multibyte encoding UTF-16.)

It is fixed width UCS-16 I think.

You cannot encode all of the world's characters in UCS-16. UCS-16
worked will through Unicode 3.0, but since then, you need UTF-something
is you want full Unicode support with less than 21 bits. I thought that
I had heard that Windows does support the surrogate pairs in Unicode
(which means implicitly multi-byte characters). Note that for Unicode
characters in the ranges U+0000..U+D7FF and U+E000..U+FFFF (which
includes all of the characters in Unicode 3.0), there is no difference
between the two.

In practice, Unicode was 16 bits through 3.0. Institutions which fixed
wide character width too early got caught. (Microsoft and IBM are the
big ones, and Java as a language.)

Quote:

So far, so good. About the only problem I see is that C and C++
require that the narrow character type be the smallest addressable
type supported by the implementation. Which does cause a problem on
Windows; for many reasons, it is desirable to have 8 bit addressable
types in C and C++, even though the logical implementation would be
to use 16 bit narrow characters and 32 bit wide characters.

Regretfully, the only solution I can propose is to use the 8 bit
interface to Windows (which I think uses UTF-8, which is about as
portable as you are going to get for international characters).

The problem is it isn't UTF-8. When using 8-bit interface to Windows
the character strings are encoded in a language dependent manner.
Each machine contains a collection of conversion tables from each of
supported languages encodings to and from UCS-16. When you invoke an
8-bit wrapper over UCS-16 function one of these tables is used to
convert the strings. Which table is going to be used is determined by
a mechanism similar to C locales. The net result of this are huge
problems with I18N portability.

Where are these locales decided? Can your program influence them? And
are there locales which use UTF-8?

Note that as soon as you leave the world of straight US ASCII, you have
the portability problems anyway. Locale names aren't standardized, the
presence of any useful locales isn't guaranteed, the interaction between
the C++ locales and the C locale isn't well specified and there are
still a lot of compilers out there which don't have a working version of
<locale> (nor wistream or wostream, for that matter).

Quote:

There are ways to overcome the problems but they require heroic
efforts from the application. Using wide-character APIs solves most
of these problems and it fits quite easily into C++.

One of the reasons why it isn't in C++ is that most OS's don't have a
wide character interface. And practically, most never will; as I said,
the current trend is toward using UTF-8 and sticking with 8 bits.

In the mean time, it would be perfectly legal for a library
implementation to treat the 8 bit char's as UTF-8, convert them
systematically into UTF-16 and use the wide character interface. I
don't know if any library actually does this, however, and there may be
various historical problems which mean that it isn't possible.

Quote:

In the case of the exceptions, there are very strong reasons why the
standard library cannot offer two versions. And since char is still
more widespread than wchar_t...

Quote:

Nothing using anything other than US ASCII is anywhere near portable.
Simply because different systems have different ways of handling the
differences. Offering an fstream::open(wchar_t*) doesn't help
portability if the OS's on which one is building don't support it.

--
James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Ben Hutchings
Guest

Posted: Wed Dec 10, 2003 8:31 pm Post subject: Re: troubled by std::wifstream::open(const char*)

[email]kanze (AT) gabi-soft (DOT) fr[/email] wrote:

Quote:

Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> wrote in message
news:<X8SdnegW5ekpP06iRTvUrg (AT) speakeasy (DOT) net>...
James Kanze wrote:
Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> writes:
snip
By intention, a wide character stream will not use a multibyte
encoding. If there is no possiblity of multibyte encodings in a
narrow character stream, there is no difference between the two, and
there is no reason to use narrow character streams. (This is not
the case for any OS that I know, however: most modern OS's use 8 bit
characters, with multi-byte encodings, usually UTF-8, being used for
international characters. The one exception is Windows, which uses
16 bit characters with the multibyte encoding UTF-16.)

There are other exceptions, such as SymbianOS. This can be built in a
8-bit character or Unicode (UCS-2) configuration, and in practice only
the latter is being deployed. Neither build provides wrappers that
allow use of the other character type.

Quote:

It is fixed width UCS-16 I think.

I believe Eugene means UCS-2.

Quote:

Yes, from Windows 2000 onwards the wide character encoding is UTF-16
rather than UCS-2.

<snip>

Quote:

Windows allows the locale to be changed on a global and per-thread
basis. However, it is impossible to change the encoding used for
multibyte text passed to and from system functions (the so-called
"ANSI code page") - this is a compile-time setting for the operating
system.

<snip>

Quote:

There are ways to overcome the problems but they require heroic
efforts from the application. Using wide-character APIs solves most
of these problems and it fits quite easily into C++.

One of the reasons why it isn't in C++ is that most OS's don't have a
wide character interface. And practically, most never will; as I said,
the current trend is toward using UTF-8 and sticking with 8 bits.

This is how things are going in the world of Unix, but that's not
"most OS's".

Quote:

That's probably sensible where there isn't a default multibyte
encoding imposed by the rest of the system. It's what SymbianOS's
C library does in Unicode builds.

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

Eugene Gershnik
Guest

Posted: Wed Dec 10, 2003 9:13 pm Post subject: Re: troubled by std::wifstream::open(const char*)

[email]kanze (AT) gabi-soft (DOT) fr[/email] wrote:

Quote:

Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> wrote in message
news:<X8SdnegW5ekpP06iRTvUrg (AT) speakeasy (DOT) net>...
James Kanze wrote:
C++ considers that the system has a single character type. It
makes no assumptions about this type, except that it is a single
type. Any time information is passed to or from the system, this
is the type that is used. Thus, filebuf requires that the name
of the file have this type, and that all reads and writes to and
from the file are done in this type. This type is called "char".
And I repeat, not only must all filenames be char[], all
transfers to and from the system take place as transfers of char
-- a wfstream translates multi-byte sequences into wide
characters.

I see the logic of this but unfortunately this makes abstract C++
incompatible with the existing OS interface of my system. The
system is Windows as you probably guessed. The basic OS interface
on this system has 2 distinct character types mapped to char and
wchar_t by all C++ implementatons. For example the API used to
open a file accepts whcar_t* while an API to obtain the host IP
from its name accepts char*. There is a compatibility layer that
allows me to open a file by passing it a char* thus allowing an
application to live in a single character type world.
Unfortunately usage of this layer makes I18N quite complicated.

I rather suspected it was Windows:-). While the people at Microsoft
were certainly trying to do the right thing, I'm not sure that I'm
happy with the results. As you say, it is a fact of life that you
need 8 bit characters, since that is, for better or worse, what is
used everywhere, and in these days of connectivity, you have to be
compatible with everywhere. Thus, the interface to obtain the host
IP uses char.

Actually the Winsock char * functions are a relict of backward compatibility
with Berkeley sockets. The newer non-backward compatible interfaces are
using wchar_t * and I fully expect that eventually it will be possible to
have all the Windows application code dealing with Unicode only.

Quote:

The attempt to use 16 bit characters for all
internal interfaces is laudable. But in practice, I have the
feeling that it creates more problems that it solves. To begin
with, it requires that you, the programmer, deal with two different
character types. Forget the C++ problem for a moment, and consider
that you have no problem using the wide character interfaces in
your application. You read a filename from a file (as wide
characters), and pass it to the API to open the file. Excellent.
You read a URL from a file (as wide characters), and pass it to the
API. Oops.

True. But again I beleive this is temporary. As the time goes there are less
and less places where one must use 8-bit chars.

Quote:

I'm not sure what the best solution should be, either from the
system point of view, or from the point of view of C++. What is
happening, however, best or not (and it doesn't seem that bad a
priori) is that everything external to the program is moving
(albeit very slowly) to UTF-8. So an 8 bit, converting interface,
makes sense.

For the external data I see this trend too but not for internal APIs. Win32,
Java and .NET are pretty much settled on fixed width Unicode
representations. The only major system that still uses chars is Unix. It
makes perfect sense to me. Why would an OS open_file() use UTF-8? Assuming
the internal data is already in fixed-width encoding why incur another two
translations?

Quote:

The one exception is
Windows, which uses 16 bit characters with the multibyte encoding
UTF-16.)

It is fixed width UCS-16 I think.

You cannot encode all of the world's characters in UCS-16. UCS-16
worked will through Unicode 3.0, but since then, you need
UTF-something is you want full Unicode support with less than 21
bits. I thought that I had heard that Windows does support the
surrogate pairs in Unicode (which means implicitly multi-byte
characters). Note that for Unicode characters in the ranges
U+0000..U+D7FF and U+E000..U+FFFF (which includes all of the
characters in Unicode 3.0), there is no difference between the two.

Win32 has some support for surrogates but it is quite limited and varies
with OS flavor. Though surrogates are a problem from a theoretical point of
view I beleive most programs that don't engage heavily in text manipulations
can get away with "each character is 16 bit" approach. This is of course
unfortunate but still better than the situation with JIS or EUC where you
cannot do anything without thinking about multi-byte issues.

Quote:

In practice, Unicode was 16 bits through 3.0. Institutions which
fixed wide character width too early got caught. (Microsoft and
IBM are the big ones, and Java as a language.)

I am not sure the change wasn't a big mistake on the part of Unicode group.
One of the big "selling points" of Unicode was that you don't have to deal
with lead byte/trail byte issues anymore. Do you know if anyone actually
uses the extended characters?

Quote:

Regretfully, the only solution I can propose is to use the 8 bit
interface to Windows (which I think uses UTF-8, which is about as
portable as you are going to get for international characters).

The problem is it isn't UTF-8. When using 8-bit interface to
Windows the character strings are encoded in a language dependent
manner. Each machine contains a collection of conversion tables
from each of supported languages encodings to and from UCS-16.
When you invoke an 8-bit wrapper over UCS-16 function one of these
tables is used to convert the strings. Which table is going to be
used is determined by a mechanism similar to C locales. The net
result of this are huge problems with I18N portability.

Where are these locales decided? Can your program influence them?
And are there locales which use UTF-8?

Short answers: In the context of filenames it's a system-wide setting. Not
without affecting the whole system. No
Long answers:
There is a general, application controllable, per-thread locale mechanism in
Win32 but unfortunately only a tiny part of it is used for automaatic API
parameter conversion. The locale used for this purpose is always the "system
default locale" defined in system configuration data (you can change it in
Control Panel). If the program needs to do anything smarter it must convert
parameters manually using the full locale mechanism.
A general Win32 locale is set on a per-tread basis. Initially a default
process locale is determined by user's preferences set in the Control Panel.
An application is free to modify current thread's locale at any time. There
are various facilities for enumeration of available locales, their features
etc. Each locale is associated with things like date and time formats,
calendars and other localizable stuff. The Unicode/multibyte translation
tables (Codepages) are associated with locales but are kept somewhat
separate. In fact it is possible to do a string conversion with an arbitrary
installed code page regardless of current locale. The whole system is rather
powerfull and works quite well. Interestingly on newer versions of the OS
there are locales that do not have any multi-byte support. They are entirely
Unicode only meaning that char * applications will not be able to work with
non-US strings at all.
There aren't any UTF-8 locales but there are UTF-8 and UTF-7 codepages.
Thus, it is possible to manually convert strings from and to these formats
but not let the system automatically do it.
What always irritates me is that with VC implementation of standard library
the Win32 Locales and C and C++ locales are completely ignorant of each
other. I cannot even reliably map the Win32 Locale identifier to the string
that can be passed to std::locale constructor. Yet another good C++ facility
made useless by QoI.

Quote:

Note that as soon as you leave the world of straight US ASCII, you
have the portability problems anyway. Locale names aren't
standardized, the presence of any useful locales isn't guaranteed,
the interaction between the C++ locales and the C locale isn't well
specified and there are still a lot of compilers out there which
don't have a working version of <locale> (nor wistream or wostream,
for that matter).

100% true. But then I'd expect the implementations to step in and provide
necessary bindings to exisiting facilities of their OS. For example I don't
care about standard locale names. Users of my software would expect to see
the names as they are in the rest of the system _not_ some C++ standard
names. So all I really care is to be able to construct C/C++ locale from my
system locale names.
There are also compilers out there that do not have _any_ iostreams. For
example eMbedded VC4 (roughly equivalent to desktop VC6 in the rest of its
features) doesn't have iostreams at all (which angers me a lot since I like
things like lexical_cast). I of course don't know the exact reasons for this
omission but my guess would be that implementing the full functionality on a
char * challnged system would be too expensive (both in time and library
size).

Quote:

There are ways to overcome the problems but they require heroic
efforts from the application. Using wide-character APIs solves
most of these problems and it fits quite easily into C++.

One of the reasons why it isn't in C++ is that most OS's don't have
a wide character interface. And practically, most never will; as I
said, the current trend is toward using UTF-8 and sticking with 8
bits.

A counter-example would be Windows CE (Pocket PC, Smartphone and other its
variations). This system actually almost doesn't have any char * support :-)

Quote:

It will break all applications localized the old way (using char * multibyte
strings).

Quote:

I can write the whole application using wchar_t only if I don't use
any of the few APIs that require chars (only some socket function
do this). The perpetual problem are fstream::open,
exception::what() and ctors of <stdexcept> classes.

In the case of the exceptions, there are very strong reasons why the
standard library cannot offer two versions. And since char is still
more widespread than wchar_t...

Out of curiosity what are these reasons?

Quote:

I can see how the standard library on my machine could provide
extensions to solve most of the problems. After all we have
_wfopen(const wchar_t *filename, const wchar_t *mode) in the same
library. So from my POV this is entirely library QoI issue. The
thing I don't agree with is the argument that having only
fstream::open(char*) somehow makes the library and applications
that are using it more portable.

Nothing using anything other than US ASCII is anywhere near
portable. Simply because different systems have different ways of
handling the differences. Offering an fstream::open(wchar_t*)
doesn't help portability if the OS's on which one is building don't
support it.

But the same is true for fstream::open(char *) on Windows CE for example.
All that could be said is fstream::open(char *) could be more easily
supported on more systems _today_. I wonder why not have something like

1. An implementation MUST provide at least one of the following overloads:
open(char *), open(wchar_t*)
2. An implementation MUST provide both overloads if the underlying system
allows files to be opened using both wide and narrow names.
3. An impementation MAY provide both overloads if the underlying system
allows only one form of the name. In this case the algorithm used for filena
me conversion is implementation defined. One possibility would be using
current stream locale to perform the conversion.

Eugene

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

kanze@gabi-soft.fr
Guest

Posted: Thu Dec 11, 2003 9:07 pm Post subject: Re: troubled by std::wifstream::open(const char*)

Ben Hutchings <do-not-spam-benh (AT) bwsint (DOT) com> wrote

Quote:

kanze (AT) gabi-soft (DOT) fr wrote:
Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> wrote in message
news:<X8SdnegW5ekpP06iRTvUrg (AT) speakeasy (DOT) net>...
James Kanze wrote:
Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> writes:
snip
By intention, a wide character stream will not use a multibyte
encoding. If there is no possiblity of multibyte encodings in a
narrow character stream, there is no difference between the two,
and there is no reason to use narrow character streams. (This is
not the case for any OS that I know, however: most modern OS's
use 8 bit characters, with multi-byte encodings, usually UTF-8,
being used for international characters. The one exception is
Windows, which uses 16 bit characters with the multibyte encoding
UTF-16.)

There are other exceptions, such as SymbianOS. This can be built in a
8-bit character or Unicode (UCS-2) configuration, and in practice only
the latter is being deployed. Neither build provides wrappers that
allow use of the other character type.

I was wondering about that one. I took a quick look at their site, but
was unable to find the information. (I'm not sure it is a wise decision
in their case. For what they do, I don't think UTF-8 would be much more
complex. And UTF-8 uses a lot less memory, which is important in their
case.)

I also wonder how far iostream is actually supported in their system.
(It is a freestanding system, after all, with no disks or such.)

Quote:

It is fixed width UCS-16 I think.

I believe Eugene means UCS-2.

You cannot encode all of the world's characters in UCS-16. UCS-16
worked will through Unicode 3.0, but since then, you need
UTF-something is you want full Unicode support with less than 21
bits. I thought that I had heard that Windows does support the
surrogate pairs in Unicode (which means implicitly multi-byte
characters).

Yes, from Windows 2000 onwards the wide character encoding is UTF-16
rather than UCS-2.

snip
The problem is it isn't UTF-8. When using 8-bit interface to
Windows the character strings are encoded in a language dependent
manner. Each machine contains a collection of conversion tables
from each of supported languages encodings to and from UCS-16.
When you invoke an 8-bit wrapper over UCS-16 function one of these
tables is used to convert the strings. Which table is going to be
used is determined by a mechanism similar to C locales. The net
result of this are huge problems with I18N portability.

Where are these locales decided? Can your program influence them?
And are there locales which use UTF-8?

Windows allows the locale to be changed on a global and per-thread
basis. However, it is impossible to change the encoding used for
multibyte text passed to and from system functions (the so-called
"ANSI code page") - this is a compile-time setting for the operating
system.

Now that is a bit hard:-). (On the other hand, I can understand it.
Looking at the output of ls in xterms with different locale settings
under Unix can be confusing, to say the least.)

Quote:

snip
There are ways to overcome the problems but they require heroic
efforts from the application. Using wide-character APIs solves
most of these problems and it fits quite easily into C++.

One of the reasons why it isn't in C++ is that most OS's don't have
a wide character interface. And practically, most never will; as I
said, the current trend is toward using UTF-8 and sticking with 8
bits.

This is how things are going in the world of Unix, but that's not
"most OS's".

I'm not sure how you count OS's. If it is by the number of machines
running with them, then vxWorks probably dominates. And they claim
Posix compliance. (But I suspect that most machines running vxWorks
don't have disks.) If it is by the number of incompatible variations,
then Unix surely wins:-).

The Internet is 8 bits. It is moving from US ASCII to UTF-8. Anything
connecting with the Internet has to be able to handle UTF-8.

Quote:

In the mean time, it would be perfectly legal for a library
implementation to treat the 8 bit char's as UTF-8, convert them
systematically into UTF-16 and use the wide character interface. I
don't know if any library actually does this, however, and there may
be various historical problems which mean that it isn't possible.
snip

That's probably sensible where there isn't a default multibyte
encoding imposed by the rest of the system. It's what SymbianOS's C
library does in Unicode builds.

I know. The problem is that we now have both 8 and 16 bit multibyte
encodings widespread, in addition to the 32 bit single byte encoding.

Ideally, char should be the correct size to interface with the OS --
thus, 16 bits with Windows and Symbian. Practically, however, you need
an 8 bit data type, even if the OS presents a 16 bit interface, and
C/C++ require char to be the smallest data type.

I don't have any real answers. I fear that there is enough history
behind us that there are no acceptable real answers.

--
James Kanze GABI Software mailto:kanze (AT) gabi-soft (DOT) fr
Conseils en informatique orientée objet/ http://www.gabi-soft.fr
Beratung in objektorientierter Datenverarbeitung
11 rue de Rambouillet, 78460 Chevreuse, France, +33 (0)1 30 23 45 16

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

Matthew Collett
Guest

Posted: Thu Dec 11, 2003 9:39 pm Post subject: Re: troubled by std::wifstream::open(const char*)

In article <AqOdnULSxsXVTEuiRTvUrg (AT) speakeasy (DOT) net>,
Eugene Gershnik <gershnik (AT) nospam (DOT) hotmail.com> wrote:

Quote:

James Kanze wrote:
C++ considers that the system has a single character type. It
makes no assumptions about this type, except that it is a single
type. Any time information is passed to or from the system, this
is the type that is used. Thus, filebuf requires that the name
of the file have this type, and that all reads and writes to and
from the file are done in this type. This type is called "char".
And I repeat, not only must all filenames be char[], all
transfers to and from the system take place as transfers of char
-- a wfstream translates multi-byte sequences into wide
characters.

[snip]

Quote:

I wonder why not have something like

1. An implementation MUST provide at least one of the following overloads:
open(char *), open(wchar_t*)
2. An implementation MUST provide both overloads if the underlying system
allows files to be opened using both wide and narrow names.
3. An impementation MAY provide both overloads if the underlying system
allows only one form of the name. In this case the algorithm used for filena
me conversion is implementation defined. One possibility would be using
current stream locale to perform the conversion.

Eugene

A corollary of James Kanze's observation is that the wchar_t type is in
principle an internal choice of the C++ implementation, independent of
any properties of the OS. Which in turn means that it's not clear what
it means to talk about "the underlying system allow[ing] files to be
opened using [...] wide [..] names". There is though a real
distinction between a "wide character" (fixed width) encoding, and a
"multibyte" (variable width) one - and UTF-16 is a multibyte encoding
with 16 bit bytes, not a wide character one.

I'm presently using Mac OS X. The implementation of C++ has 8 bit char,
and 32 bit wchar_t (allowing the use of a true wide character encoding
such as UCS-4), but the OS stores and manipulates filenames using
UTF-16. It is the job of the C/C++ library implementation to cope with
the mismatch of character types, which it does just fine by treating the
char* parameter to open as UTF-8. There might be a case for char being
changed to 16 bit to match the OS, but I would definitely consider it a
retrograde step if wchar_t was. (Perhaps I've overstated the case
slightly, since the OS itself does also provide a UTF-8 aware API,
easing the library's task, but this does not alter the principle.)

Best wishes,
Matthew Collett

--
Those who assert that the mathematical sciences have nothing to say
about the good or the beautiful are mistaken. -- Aristotle

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

Display posts from previous:

	C++Talk.NET Forum Index -> C++ Language (Moderated)	All times are GMT Goto page 1, 2 Next
Page 1 of 2

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum