473,320 Members | 1,839 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,320 software developers and data experts.

Getting around .Net Strings being UTF-16 encoded only

Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray

Nov 1 '05 #1
5 3246
Mike,

Your name sounds that you are in the uni code 1252 area, wich is huge.

http://www.geocities.com/Athens/Acad.../fontset.htm#b

You should only be busy with encoding when you know that you get characters
from outside that area.

I hope this helps,

Cor
"Mike Murray" <ou*********@gmail.com> schreef in bericht
news:11*********************@f14g2000cwb.googlegro ups.com...
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray
Nov 2 '05 #2
Mike Murray <ou*********@gmail.com> wrote:
I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?


Just pass the XmlTextWriter constructor a stream and Encoding.UTF8.

I suspect if that doesn't work, you aren't getting the data back
properly in the first place.

See http://www.pobox.com/~skeet/csharp/unicode.html and
http://www.pobox.com/~skeet/csharp/d...ngunicode.html

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 2 '05 #3
It should be easier by using the constructor that allows to specify the
stream encoding...

I would first try with a simple text file in particular :
- paste this char into notepad and save the file as UTF-8
- create this file programmaticaly

You can then use "debug" to compare those files bytes per bytes (should be
easy with 4 bytes !). For now it could range from the character not being
stored correctly in the db to the tool used to read back the file to see if
it worked that wouldn't take into account that this is an UTF-8 encoded file
making appear it bad when it is correct etc...

--
Patrice

"Mike Murray" <ou*********@gmail.com> a écrit dans le message de
news:11*********************@f14g2000cwb.googlegro ups.com...
Hi I have the following issue.

I have a character that is return by a SQL Server database "É" to be
precise, the issue is that when I store character in a .net string
variable my understanding is that it converts it to UTF-16
automatically. I've seen a lot of quotes saying "All String datatype
variables in .NET are UTF-16 encoded". The thing is that I want to
write this character to a XML file in UTF-8 encoded format instead. How
do I go about doing this?

Right now I've tried the following scenerio's (all in VB.Net) when
writing to my XmlTextWriter object:

1) Encoding the string into UTF-8 bytes, then placing the string into
a Stream Reader and then from there reading the stream into the
XmlTextWriter

Dim utf8 As New UTF8Encoding(True)
Dim fNameBytes As Byte() = utf8.GetBytes(strFName)

Dim stmFName As StreamReader = _
New StreamReader( New MemoryStream( fNameBytes ), utf8)

....

EFSXml_UserInfo.WriteAttributeString("firstname",
stmFName.ReadToEnd() )

2) I've also tried the code below:

Dim utf8 As New UTF8Encoding(True)

Dim xtrFName As XmlTextReader = _
New XmlTextReader(New StringReader(strFName))
Dim msFName As MemoryStream = New MemoryStream()

Dim xtwFName As XmlTextWriter = New XmlTextWriter(msFName, utf8)

xtwFName.WriteNode(xtrFName, false)
xtwFName.Flush()

msFName.Position = 0

Dim srFName As StreamReader = New StreamReader(msFName)

....

EFSXml_UserInfo.WriteAttributeString("firstname", srFName
..ReadToEnd() )

Neither of the 2 result in encoding the string correctly, can someone
please help me out, I've spend close to 2 days trying to figure this
stupid thing out.

Thanks,

Mike Murray
Nov 2 '05 #4
Hi folks,

Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)

What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.

Thanks,

Mike

Nov 3 '05 #5
Mike Murray <ou*********@gmail.com> wrote:
Here is some more information.

The application that is expecting the UTF-8 Encoding creates a file
using the the default that comes in from the web browser and creates
the file via vbscript, windows scripting code. Right now there is no
conversion to UTF-8 happening in the vbscript, the programmer is just
relying on what comes in through the browser to the vbscript (is there
anything funny with variables storing the incoming text as UTF-16 just
like .NET does?)
By the time you get the string, it should be correct Unicode.
What I want to do is create the same file but from vb.NET, therefore
that is why i'm using the XmlTextWriter code. Which makes sense to me,
I just want to clarify that my code is correct, and i'm going about
this the right way.


It sounds okay to me. You shouldn't need to do anything to get
XmlTextWriter to create files correctly in UTF-8.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Nov 4 '05 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

11
by: Laurent Therond | last post by:
Maybe you have a minute to clarify the following matter... Consider: --- from cStringIO import StringIO def bencode_rec(x, b): t = type(x)
4
by: Joerg Lehmann | last post by:
I am using Python 2.2.3 (Fedora Core 1). The problem is, that strings containing umlauts do not work as I would expect. Here is my example: >>> a = 'äöü' >>> b = '123' >>> print "%-5s...
97
by: Kjetil Torgrim Homme | last post by:
often when re-factoring code, I need to change the indent level of some chunk of code. due to the lack of an end marker, my Emacs has to use heuristics when re-indenting, and this will...
9
by: Ksenia Marasanova | last post by:
Hi, I have a little problem with encoding. Was hoping maybe anyone can help me to solve it. There is some amount of data in a database (PG) that must be inserted into Excel sheet and emailed....
1
by: Ziemowit Skowronski | last post by:
Hi, I have a problem with displaying unicode strings in ASP.NET (C#). I recieve data from MS SQL Server database and use them to construct navigation on the page. Because I'm from Poland, I have...
29
by: Ron Garret | last post by:
>>> u'\xbd' u'\xbd' >>> print _ Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in position 0: ordinal not in...
5
by: robson.cozendey.rj | last post by:
Hi, I´m trying desperately to tell the interpreter to put an 'á' in my string, so here is the code snippet: # -*- coding: utf-8 -*- filename = u"Ataris Aquáticos #2.txt" f = open(filename,...
3
by: 7stud | last post by:
Can anyone tell me why I can print out the individual variables in the following code, but when I print them out combined into a single string, I get an error? symbol = u'ibm' price = u'4 \xbd'...
6
by: | last post by:
Hi, I used extensively python and now I find this mess with strings, I can't even reproduce tutorial examples: File "<stdin>", line 0 ^ SyntaxError: 'ascii' codec can't decode byte 0xc4 in...
5
by: Asterix | last post by:
how could I test that those 2 strings are the same: 'séd' (repr is 's\\xc3\\xa9d') u'séd' (repr is u's\\xe9d')
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
0
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.