incorrect encoding after serialisation to XML

[Stephen]

Using the code below I am trying, in VB .Net 2003, to serialise classes
defined in a couple of XSD documents. The encoding for both is
Unicode(UTF-8). However the resulting XML is encoded as UTF-16. This is
causing me problems when I try to load it into an XPath document. I would
imagine I should be able to use System.Text.Enc oding to define the encoding
as UTF-8 but I haven't been able to figure out how so far.

Dim ser As New XmlSerializer(G etType(GTPAPP))
Dim sw As New StringWriter
ser.Serialize(s w, domainObj.getGT PApp)
Return sw.ToString()

any help would be appreciated.

[Stephen]

one additional point on this is that when I try to load the XML when encoded
as UTF-16 I get the error
"There is no Unicode byte order mark. Cannot switch to Unicode."

"Stephen" wrote:

Using the code below I am trying, in VB .Net 2003, to serialise classes
defined in a couple of XSD documents. The encoding for both is
Unicode(UTF-8). However the resulting XML is encoded as UTF-16. This is
causing me problems when I try to load it into an XPath document. I would
imagine I should be able to use System.Text.Enc oding to define the encoding
as UTF-8 but I haven't been able to figure out how so far.

Dim ser As New XmlSerializer(G etType(GTPAPP))
Dim sw As New StringWriter
ser.Serialize(s w, domainObj.getGT PApp)
Return sw.ToString()

any help would be appreciated.

[Derek Harmon]

"Stephen" <St*****@discus sions.microsoft .com> wrote in message news:8C******** *************** ***********@mic rosoft.com...

"Stephen" wrote:

Using the code below I am trying, in VB .Net 2003, to serialise classes
defined in a couple of XSD documents. The encoding for both is
Unicode(UTF-8).
The encoding of the schema documents is irrelevant, and the
encoding of the classes -- huh?

Once the Framework loads a piece of XML, that XML becomes
a node set. Metaphysically, think of it as astral projection (an
out-of-body experience) for the XML ... it goes to a higher plane
of existence where encodings no longer matter (whether attributes
are delimited by single- or double- quotes no longer matters,
whether content came from a CDATA section no longer matters,
etc.)

The challenges you're facing seem to center on entering and
leaving 'the Body' (think of the XML as being corporeal
whenever you see angle brackets).
However the resulting XML is encoded as UTF-16. : : Dim sw As New StringWriter
ser.Serialize(s w, domainObj.getGT PApp)
Return sw.ToString()

A String is always UTF-16 encoded. There is no such thing as a UTF-8
string. It's a myth. It's fiction. UTF-8 strings went out with the dragon,
leprechauns, three-headed dogs guarding the Underworld, and Java.
one additional point on this is that when I try to load the XML when encoded
as UTF-16 I get the error
"There is no Unicode byte order mark. Cannot switch to Unicode."

This depends on how the XML was saved. Was it saved with a Stream
that used System.Text.Uni codeEncoding?

You'll get this XmlException when the XML isn't encoded in the encoding
it says it is.

My advice is not to write the XML to a String. If you want to put it into
a file (and have control over it's encoding), use an XmlTextWriter and
wrap it around a Stream. Then read it back in with a TextWriter that
is wrapped around a Stream, with a matching encoding (and any
encoding declaration that appears in the XML declaration should
match both the encoding you used to serialize out and deserialize
in.)

- - - utf8xml.vb (excerpt)
' . . .
Dim tw As New System.Xml.XmlT extWriter( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Create), _
New System.Text.UTF 8Encoding( True) )

' XML will be serialized to file.xml, in UTF-8, with BOM.
ser.Serialize( tw, domainObj.getGT PApp)

' Finish writing the file and close it.
tw.Flush( )
tw.Close( )

Dim tr As New System.IO.TextR eader( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Open), _
New System.Text.UTF 8Encoding( True) )

' XML will be deserialized from file.xml, in UTF-8, with BOM.
domainObj.setGT PApp( CType( _
ser.Deserialize ( tr), GTPApp) )

tr.Close( )
' . . .
- - -
Derek Harmon

[Stephen]

Derek
I have a further question on this query. What I am actually trying to do is
generate XML that can be used as the input XML for a call to a second
application. So rather then generating a file I need to generate the XML to
be used in code. Would you know how to do this?

"Derek Harmon" wrote:

"Stephen" <St*****@discus sions.microsoft .com> wrote in message news:8C******** *************** ***********@mic rosoft.com...

"Stephen" wrote:

Using the code below I am trying, in VB .Net 2003, to serialise classes
defined in a couple of XSD documents. The encoding for both is
Unicode(UTF-8).
The encoding of the schema documents is irrelevant, and the
encoding of the classes -- huh?

Once the Framework loads a piece of XML, that XML becomes
a node set. Metaphysically, think of it as astral projection (an
out-of-body experience) for the XML ... it goes to a higher plane
of existence where encodings no longer matter (whether attributes
are delimited by single- or double- quotes no longer matters,
whether content came from a CDATA section no longer matters,
etc.)

The challenges you're facing seem to center on entering and
leaving 'the Body' (think of the XML as being corporeal
whenever you see angle brackets).
However the resulting XML is encoded as UTF-16. : : Dim sw As New StringWriter
ser.Serialize(s w, domainObj.getGT PApp)
Return sw.ToString()

A String is always UTF-16 encoded. There is no such thing as a UTF-8
string. It's a myth. It's fiction. UTF-8 strings went out with the dragon,
leprechauns, three-headed dogs guarding the Underworld, and Java.

one additional point on this is that when I try to load the XML when encoded
as UTF-16 I get the error
"There is no Unicode byte order mark. Cannot switch to Unicode."

This depends on how the XML was saved. Was it saved with a Stream
that used System.Text.Uni codeEncoding?

You'll get this XmlException when the XML isn't encoded in the encoding
it says it is.

My advice is not to write the XML to a String. If you want to put it into
a file (and have control over it's encoding), use an XmlTextWriter and
wrap it around a Stream. Then read it back in with a TextWriter that
is wrapped around a Stream, with a matching encoding (and any
encoding declaration that appears in the XML declaration should
match both the encoding you used to serialize out and deserialize
in.)

- - - utf8xml.vb (excerpt)
' . . .
Dim tw As New System.Xml.XmlT extWriter( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Create), _
New System.Text.UTF 8Encoding( True) )

' XML will be serialized to file.xml, in UTF-8, with BOM.
ser.Serialize( tw, domainObj.getGT PApp)

' Finish writing the file and close it.
tw.Flush( )
tw.Close( )

Dim tr As New System.IO.TextR eader( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Open), _
New System.Text.UTF 8Encoding( True) )

' XML will be deserialized from file.xml, in UTF-8, with BOM.
domainObj.setGT PApp( CType( _
ser.Deserialize ( tr), GTPApp) )

tr.Close( )
' . . .
- - -
Derek Harmon

[Derek Harmon]

Stephen,

Certainly, no problem. In the code below, simply replace references to
FileStream with references to MemoryStream. This will yield a Byte( )
array that you can pass along in binary form to this other application.
They're Bytes and not Chars, so the Byte( ) can be UTF-8 encoded.

As far as

The XmlTextWriter declarations below become.

Dim tw As New System.Xml.XmlT extWriter( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Create), _
New System.Text.UTF 8Encoding( True) )
Dim stream As New System.IO.Memor yStream( )
Dim tw As New System.Xml.XmlT extWriter( _
stream, New System.Text.UTF 8Encoding( True) )

Again, this produces the Byte Order Mark because I'm creating the
UTF8Encoding with True. When you're sending a binary stream, you
might not want a BOM (or at least the consumer might not want the
BOM). You may want to try it both ways, with- and w/o BOM.

After you are done Serializing to the tw, just like in the code example
below, you would extract the Byte( ) from the MemoryStream.

Dim buffer As Byte( ) = stream.GetBuffe r( )

and you can pass this forward to another application (i.e., through
a Socket) or call a method,

Me.OtherMethod( buffer)

' . . .

Public Sub OtherMethod( ByVal incomingXml As Byte( ) )
' OtherMethod proceeds to wrap incomingXml in a
' MemoryStream and deserialize it using an XmlReader
' . . .
End Sub
The deserialization works much the same as the translation of the serialization
I've shown above. Polymorphism is what makes this possible. To the Xml-
Reader and Writer classes it doesn't matter what variety of System.IO.Strea m
you use (MemoryStream, FileStream, NetworkStream), only that it is a Stream.
Derek Harmon
"Stephen" <St*****@discus sions.microsoft .com> wrote in message news:0A******** *************** ***********@mic rosoft.com...

Derek
I have a further question on this query. What I am actually trying to do is
generate XML that can be used as the input XML for a call to a second
application. So rather then generating a file I need to generate the XML to
be used in code. Would you know how to do this?

"Derek Harmon" wrote:

: : - - - utf8xml.vb (excerpt)
' . . .
Dim tw As New System.Xml.XmlT extWriter( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Create), _
New System.Text.UTF 8Encoding( True) )

' XML will be serialized to file.xml, in UTF-8, with BOM.
ser.Serialize( tw, domainObj.getGT PApp)

' Finish writing the file and close it.
tw.Flush( )
tw.Close( )

Dim tr As New System.IO.TextR eader( _
New System.IO.FileS tream( "file.xml", System.IO.FileM ode.Open), _
New System.Text.UTF 8Encoding( True) )

' XML will be deserialized from file.xml, in UTF-8, with BOM.
domainObj.setGT PApp( CType( _
ser.Deserialize ( tr), GTPApp) )

tr.Close( )
' . . .
- - -