BUG in StreamWriter

Hi,

When constructing StreamWriter with the following..
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f) ;

Then attempt to write out åäö letters they become garbage.

BUT

If we call StreamWriter as follows...
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f, System.Text.Enc oding.Default);

Its ok. So why is default not the actual DEFAULT as it says on the ctor?

It seems to me either the ctor is wrong or the name .Default is misleading.

Thanks.

[Jon Skeet [C# MVP]]

<di********@dis cussion.microso ft.com> wrote:

When constructing StreamWriter with the following..
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f) ;

Then attempt to write out åäö letters they become garbage.

BUT

If we call StreamWriter as follows...
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f, System.Text.Enc oding.Default);

Its ok. So why is default not the actual DEFAULT as it says on the ctor?

It seems to me either the ctor is wrong or the name .Default is misleading.

..Default is *slightly* misleading, although all the information is in
the documentation. The docs for new StreamWriter(St ream) say:

<quote>
This constructor creates a StreamWriter with UTF-8 encoding whose
GetPreamble method returns an empty byte array. The BaseStream property
is initialized using the stream parameter.
</quote>

However, the brief summary saying that it uses "the default" encoding
is misleading (I'll mail MS about it).

..Default means the default *platform* encoding - but pretty much
everything in .NET itself uses UTF-8 by default.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

So UTF8 cant handle umlaut characters it seems then
"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...
<di********@dis cussion.microso ft.com> wrote:

When constructing StreamWriter with the following..
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f) ;

Then attempt to write out åäö letters they become garbage.

BUT

If we call StreamWriter as follows...
FileStream f = new FileStream(..);
StreamWriter s = new StreamWriter(f, System.Text.Enc oding.Default);

Its ok. So why is default not the actual DEFAULT as it says on the ctor?

It seems to me either the ctor is wrong or the name .Default is

misleading.

..Default is *slightly* misleading, although all the information is in
the documentation. The docs for new StreamWriter(St ream) say:

<quote>
This constructor creates a StreamWriter with UTF-8 encoding whose
GetPreamble method returns an empty byte array. The BaseStream property
is initialized using the stream parameter.
</quote>

However, the brief summary saying that it uses "the default" encoding
is misleading (I'll mail MS about it).

..Default means the default *platform* encoding - but pretty much
everything in .NET itself uses UTF-8 by default.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

<di********@dis cussion.microso ft.com> wrote:

So UTF8 cant handle umlaut characters it seems then

Yes it can. It's just that whatever you were using to read the file
presumably wasn't aware that it was encoded in UTF-8.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

According to windows file system it says ASCII :D

I thought that was standard enough :D Because I used the same format all the
way thru the code and its umlauted ok but when its writing (using the
default ctors) its garbled. I wiped the file, changed it to construct the
SR with Encoding.Defaul t and its saving the umlat charset now, howcome the
usual ctor with FileStream doesnt save umlaut chars then as nowwhere else
did I specify any form of encoding until this change to fix it.

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

<di********@dis cussion.microso ft.com> wrote:

So UTF8 cant handle umlaut characters it seems then

Yes it can. It's just that whatever you were using to read the file
presumably wasn't aware that it was encoded in UTF-8.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

<?xml version="1.0" encoding="utf-8"?>

was even defined in the XML file that I got the string from, its even stored
in the String type correctly its just when writing to the file.

Normal calls specified WITHOUT encoding parameters did NOT save the umlaut
chars.

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

<di********@dis cussion.microso ft.com> wrote:

So UTF8 cant handle umlaut characters it seems then

Yes it can. It's just that whatever you were using to read the file
presumably wasn't aware that it was encoded in UTF-8.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Opening the text file in notepad and selecting save as shows its ANSI, not
UTF8- how come the file create when appending does not store the file as
UTF8 then as thats suppost to be the default that you state?

That would cause the mixmatch if the file create is creating as ANSI and all
methods default to UTF8.


<di********@dis cussion.microso ft.com> wrote in message
news:ev******** ******@TK2MSFTN GP10.phx.gbl...

<?xml version="1.0" encoding="utf-8"?>

was even defined in the XML file that I got the string from, its even stored in the String type correctly its just when writing to the file.

Normal calls specified WITHOUT encoding parameters did NOT save the umlaut
chars.

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

<di********@dis cussion.microso ft.com> wrote:

So UTF8 cant handle umlaut characters it seems then

Yes it can. It's just that whatever you were using to read the file
presumably wasn't aware that it was encoded in UTF-8.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

<di********@dis cussion.microso ft.com> wrote:

According to windows file system it says ASCII :D
What do you mean by "according to the Windows file system"?
I thought that was standard enough :D
ASCII doesn't have any characters with accents.
Because I used the same format all the
way thru the code and its umlauted ok but when its writing (using the
default ctors) its garbled. I wiped the file, changed it to construct the
SR with Encoding.Defaul t and its saving the umlat charset now, howcome the
usual ctor with FileStream doesnt save umlaut chars then as nowwhere else
did I specify any form of encoding until this change to fix it.
It *does* save umlaut characters, it's just that what you're using to
read the file isn't recognising that it's UTF-8. You later say:
Opening the text file in notepad and selecting save as shows its ANSI,
not UTF8

That's just notepad being confused.

UTF-8 works fine, the framework works fine - but some of your tools may
not be doing what you want them to.

See http://www.pobox.com/~skeet/csharp/unicode.html for more
information about encodings.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

You're right, because notepad isnt standard at all for reading text files.
Nobody in theyre right mind uses it or Wintail etc to view logs. No no not
at all :D

Its fine when i specify Encoding.Defaul t on StreamWriter yet its NOT when I
dont specify ANY encoding anywhere in the app.

"Jon Skeet [C# MVP]" <sk***@pobox.co m> wrote in message
news:MP******** *************** *@msnews.micros oft.com...

<di********@dis cussion.microso ft.com> wrote:

According to windows file system it says ASCII :D

What do you mean by "according to the Windows file system"?

I thought that was standard enough :D

ASCII doesn't have any characters with accents.

Because I used the same format all the
way thru the code and its umlauted ok but when its writing (using the
default ctors) its garbled. I wiped the file, changed it to construct the SR with Encoding.Defaul t and its saving the umlat charset now, howcome the usual ctor with FileStream doesnt save umlaut chars then as nowwhere else did I specify any form of encoding until this change to fix it.

It *does* save umlaut characters, it's just that what you're using to
read the file isn't recognising that it's UTF-8. You later say:

Opening the text file in notepad and selecting save as shows its ANSI,
not UTF8

That's just notepad being confused.

UTF-8 works fine, the framework works fine - but some of your tools may
not be doing what you want them to.

See http://www.pobox.com/~skeet/csharp/unicode.html for more
information about encodings.

--
Jon Skeet - <sk***@pobox.co m>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

[Frans Bouma]

Jon Skeet [C# MVP] <sk***@pobox.co m> wrote in
news:MP******** *************** *@msnews.micros oft.com:

<di********@dis cussion.microso ft.com> wrote:

Because I used the same format all the
way thru the code and its umlauted ok but when its writing (using the
default ctors) its garbled. I wiped the file, changed it to construct
the SR with Encoding.Defaul t and its saving the umlat charset now,
howcome the usual ctor with FileStream doesnt save umlaut chars then as
nowwhere else did I specify any form of encoding until this change to
fix it.

It *does* save umlaut characters, it's just that what you're using to
read the file isn't recognising that it's UTF-8. You later say:

The byte specification in the actual raw data misses UTF-8
specification when you use Default. I was bitten by the same thing. I had
to explicitly state Encoding.Unicod e. WHen I used Encoding.Defaul t, it
should work according to the docs, but it didn't. It did save stuff like
scandinavian characters away in the file, but it couldn't read it back
correctly, even if I stated UTF-8 as encoding or whatever in the xml
header. So I think he's right.

Opening the text file in notepad and selecting save as shows its ANSI,
not UTF8

That's just notepad being confused.
UTF-8 works fine, the framework works fine - but some of your tools may
not be doing what you want them to.

If you specify Encoding.Unicod e, it will work, if you specify
Encoding.Defaul t it will not in some cases. In both cases, the files do
NOT have an XML heading explaining the encoding. The actual encoding is in
the bytes in the file (and probably in a meta-data property in NTFS). That
specification is not read back / or written correctly when you use
Default. I think that's the reason for his complaint and I have to admit,
he's right, I had exactly the same thing.

Frans

--
Get LLBLGen Pro, the new O/R mapper for .NET: http://www.llblgen.com