Getting around .Net Strings being UTF-16 encoded only

[Mike Murray]

Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray

[Cor Ligthert [MVP]]

Mike,

Your name sounds that you are in the uni code 1252 area, wich is huge.

http://www.geocities.com/Athens/Acad.../fontset.htm#b

You should only be busy with encoding when you know that you get characters
from outside that area.

I hope this helps,

Cor
"Mike Murray" <ou*********@gmail.com> schreef in bericht
news:11*********************@f14g2000cwb.googlegro ups.com...
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray

[Jon Skeet [C# MVP]]

Mike Murray <ou*********@gmail.com> wrote:

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Just pass the XmlTextWriter constructor a stream and Encoding.UTF8.

I suspect if that doesn't work, you aren't getting the data back
properly in the first place.

See http://www.pobox.com/~skeet/csharp/unicode.html and
http://www.pobox.com/~skeet/csharp/d...ngunicode.html

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Patrice]

It should be easier by using the constructor that allows to specify the
stream encoding...

I would first try with a simple text file in particular :
- paste this char into notepad and save the file as UTF-8
- create this file programmaticaly

You can then use "debug" to compare those files bytes per bytes (should be
easy with 4 bytes !). For now it could range from the character not being
stored correctly in the db to the tool used to read back the file to see if
it worked that wouldn't take into account that this is an UTF-8 encoded file
making appear it bad when it is correct etc...

--
Patrice

"Mike Murray" <ou*********@gmail.com> a écrit dans le message de
news:11*********************@f14g2000cwb.googlegro ups.com...
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray

[Mike Murray]

Hi folks,

Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)

What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.

Thanks,

Mike

[Jon Skeet [C# MVP]]

Mike Murray <ou*********@gmail.com> wrote:

Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)
By the time you get the string, it should be correct Unicode.
What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.

It sounds okay to me. You shouldn't need to do anything to get
XmlTextWriter to create files correctly in UTF-8.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too