wstring, wofstream, and encodings

[Jeffrey Walton]

Hi All,

I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

== Sample ==
wstring ws = L"wide";

wofstream ofs;
ofs.open("wide.dat", std::ios::binary | std::ios::trunc );
if( !ofs.good() ) { return; }

ofs << ws;
ofs.close();
== End Sample ==

Thanks,
Jeff
Jeffrey Walton

[1] http://www.research.att.com/~bs/3rd_loc0.html

[Ivan Vecerina]

"Jeffrey Walton" <no******@gmail.comwrote in message
news:5f**********************************@b64g2000 hsa.googlegroups.com...

>I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

From my understanding of iostream, locales will not be the answer.

Locales apply to the upper layer of the iostream, which takes
care of converting values to characters. They affect the
choice of the characters used to represent a value, but not
the encoding of these characters.

The internal filebuf or basic_filebuf is the object that will
determine how the in-memory characters are wirtten to a file.
This is the layer (the stream *buffer*) that can define whether
a file is written using UTF8 or another character encoding.

However, the C++ standard does not specify an interface allowing
to select what character encoding is to be used by (w)filebuf.

Your best bet would be to ask your question on a platform-
specific forum, related to the library implementation you use.
A specific wfilebuf (/basic_filebuf) implementation may
allow you to specify the file's enocding style.
Or maybe this is configurable at an OS or C library level.
Worst case, you will still be able to write your own streambuf
layer to write files using the specific encoding you want.
I hope this helps...
Ivan
--
http://ivan.vecerina.com/contact/?subject=NG_POST <- email contact form
Brainbench MVP for C++ <http://www.brainbench.com

[Vaclav Haisman]

Jeffrey Walton wrote, On 12.4.2008 5:27:

Hi All,

I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

I am too lazy to look into the standard right now but IIRC it says something
about it being implementation defined. In either case, I think you can imbue
the stream with your own locale that has custom codecvt facet.

--
VH
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIAG+UoUFWwtEPkHIRCORBAJ4zQNd73YPhyP8gtG+n4A 5JPlh5owCcDiR7
lSSGMtU6qa3Ndmy3YB9ftks=
=B9oY
-----END PGP SIGNATURE-----

[Jeffrey Walton]

Hi Ivan,

On Apr 12, 3:51*am, "Ivan Vecerina"
<[e-mail]:

"Jeffrey Walton" <[e-mail] in message

news:5f**********************************@b64g2000 hsa.googlegroups.com...

I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

From my understanding of iostream, locales will not be the answer.

Locales apply to the upper layer of the iostream, which takes
care of converting values to characters. *They affect the
choice of the characters used to represent a value, but not
the encoding of these characters.

The internal filebuf or basic_filebuf is the object that will
determine how the in-memory characters are wirtten to a file.
This is the layer (the stream *buffer*) that can define whether
a file is written using UTF8 or another character encoding.

However, the C++ standard does not specify an interface allowing
to select what character encoding is to be used by (w)filebuf.

Your best bet would be to ask your question on a platform-
specific forum, related to the library implementation you use.
A specific wfilebuf (/basic_filebuf) implementation may
allow you to specify the file's enocding style.
Or maybe this is configurable at an OS or C library level.
Worst case, you will still be able to write your own streambuf
layer to write files using the specific encoding you want.

I hope this helps...
Ivan
--http://ivan.vecerina.com/contact/?subject=NG_POST<- email contact form
Brainbench MVP for C++ <>http://www.brainbench.com

Your best bet would be to ask your question on a platform-
specific forum, related to the library implementation you use.

You are correct. I asked over on microsoft.public.vc.language.
Microsoft's implementation is broken [1]. I'm kind of suprised - I
though Plaugher (sp?) supplied it through Visual Studio.

Jeff
Jeffrey Walton

[1] http://groups.google.com/group/micro...8eb7489a42b568

[Jeffrey Walton]

On Apr 12, 4:15*am, Vaclav Haisman <[e-mail]:

Jeffrey Walton wrote, On 12.4.2008 5:27:Hi All,

I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

I am too lazy to look into the standard right now but IIRC it says something
about it being implementation defined. In either case, I think you can imbue
the stream with your own locale that has custom codecvt facet.

--
VH

*signature.asc
1KDownload

Thanks VH. This will help.

Jeff
Jeffrey Walton

[Jeffrey Walton]

Hi VH,

On Apr 12, 4:15*am, Vaclav Haisman <[e-mail]:

Jeffrey Walton wrote, On 12.4.2008 5:27:Hi All,

I'm attempting to write a wstring to a file by way of wofstream. I'm
getting compression on the stream (I presume it is UTF-8). How/where
do I invoke an alternate constructotor so that the stream stays wide
(UTF-16)?

I suspect that it is hidden in a locale, but I don't have much
experience with them. I also have not been able to locate it in
Stroustrup: Appendix D: Locales [1]. [1] does state the following, but
I do not have section 21.7: "Section §21.7 describes how to change
locale for a stream; this appendix describes how a locale is
constructed out of facets and explains the mechanisms through which a
locale affects its stream."

I am too lazy to look into the standard right now but IIRC it says something
about it being implementation defined. In either case, I think you can imbue
the stream with your own locale that has custom codecvt facet.

I was not able to grab a copy off the ISO web site $$$ (I wanted to
thumb through it). If you could post a link to your source, it would
be appreciated.

Jeff
Jeffrey Walton

[James Kanze]

On 12 avr, 14:42, Jeffrey Walton <[e-mail]:

On Apr 12, 3:51 am, "Ivan Vecerina"

[...]

You are correct. I asked over on microsoft.public.vc.language.
Microsoft's implementation is broken [1]. I'm kind of suprised - I
though Plaugher (sp?) supplied it through Visual Studio.

Broken, or simply that the locales which do what you want aren't
delivered with VC++? (In general, Dinkumware has been well in
advance of other implementations in its implementation of
locales, and locale support in streams. And while one can never
totally discount the possibility of an error, I'd tend to
suspect rather the absense of the necessary locales.)

--
James Kanze (GABI Software) email:ja*********@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, [phone]