Difference in encoding? - Confused

[Ian Harding]

I'm working on a .NET application that requests an XML document, in
string form, from a legacy COM component, then deserializes it. In
order to deserialize the document, the string needs to be placed into a
stream. AFAIK, .NET strings are UTF-16 encoded, but the COM component
returns a UTF-8 encoded document, so my first attempt at creating and
filling a stream used this code:

string response = <some UTF-8 encoded xml from a COM component>;
MemoryStream result = new MemoryStream(response.Length);
UTF8Encoding utf8Encoding = new UTF8Encoding();
result.Write(utf8Encoding.GetBytes(response), 0, response.Length);

The deserialization then worked fine until one of the XML documents
contained a UK pound character - £ - at which point an exception was
thrown indicating an invalid document.

After searching Google, I came across the following alternative code to
create and fill the stream for deserialization. After testing with
pound and a few other problem characters, this seems to work properly.

string response = <some UTF-8 encoded xml from a COM component>;
MemoryStream result = new MemoryStream(response.Length);
StreamWriter writer = new StreamWriter( result, new UTF8Encoding());
writer.Write( response );
writer.Flush();

I have a couple of questions: First, is my understanding of the
encoding issues correct? If I have a UTF-8 encoded document, is it up
to me to decode it into the stream explicitly? Secondly, my reading of
the two code snippets is that they should produce an identical result,
but in reality the first one doesn't seem to be decoding the document
correctly for all characters - can anyone explain what is causing the
different behaviour?

Thanks
Ian

[Oleg Tkachenko [MVP]]

Ian Harding wrote:

I'm working on a .NET application that requests an XML document, in
string form, from a legacy COM component, then deserializes it. In
order to deserialize the document, the string needs to be placed into a
stream. AFAIK, .NET strings are UTF-16 encoded, but the COM component
returns a UTF-8 encoded document, so my first attempt at creating and
filling a stream used this code:

string response = <some UTF-8 encoded xml from a COM component>;

I think you should do nothing here. Just parse the string. Why do you
need a stream? This should work:
XmlDocument doc = new XmlDocument();
doc.LoadXml(response);

--
Oleg Tkachenko [XML MVP, MCP]
http://blog.tkachenko.com

[Ian Harding]

Oleg Tkachenko [MVP] wrote:

Ian Harding wrote:

I'm working on a .NET application that requests an XML document, in
string form, from a legacy COM component, then deserializes it. In
order to deserialize the document, the string needs to be placed into
a stream. AFAIK, .NET strings are UTF-16 encoded, but the COM
component returns a UTF-8 encoded document, so my first attempt at
creating and filling a stream used this code:

string response = <some UTF-8 encoded xml from a COM component>;

I think you should do nothing here. Just parse the string. Why do you
need a stream? This should work:
XmlDocument doc = new XmlDocument();
doc.LoadXml(response);

I probably should have explained what I was doing with the data more
clearly.

We have a class library, containing serializable classes that represent
each type of document that can be returned by the COM component. For a
given request, we always know the type of the returned document, so we
just use XmlSerializer to populate a class instance from the XML. Saves
messing about with DOM and XPATH on the client-side. As I understand
it, it isn't possibly to pass a string directly to the serializer for
de-serialization. A MemoryStream seemed like the lowest-overhead way of
getting it into a stream.

Thanks
Ian

[Oleg Tkachenko [MVP]]

Ian Harding wrote:

We have a class library, containing serializable classes that represent
each type of document that can be returned by the COM component. For a
given request, we always know the type of the returned document, so we
just use XmlSerializer to populate a class instance from the XML. Saves
messing about with DOM and XPATH on the client-side. As I understand
it, it isn't possibly to pass a string directly to the serializer for
de-serialization. A MemoryStream seemed like the lowest-overhead way of
getting it into a stream.

XmlSerializer accepts TextReader, that means you can pass it new
StringReader(response). Fiddling with encoding with MemoryStream is
usually very error-prone. Basically if your XML is in .NET string that
means it's already UTF-16 encoded, but its XML declaration says UTF-8.
..NET supports such case just fine by switching to UTF-16.

--
Oleg Tkachenko [XML MVP, MCP]
http://blog.tkachenko.com

[Ian Harding]

Oleg Tkachenko [MVP] wrote:

Ian Harding wrote:

We have a class library, containing serializable classes that
represent each type of document that can be returned by the COM
component. For a given request, we always know the type of the
returned document, so we just use XmlSerializer to populate a class
instance from the XML. Saves messing about with DOM and XPATH on the
client-side. As I understand it, it isn't possibly to pass a string
directly to the serializer for de-serialization. A MemoryStream
seemed like the lowest-overhead way of getting it into a stream.

XmlSerializer accepts TextReader, that means you can pass it new
StringReader(response). Fiddling with encoding with MemoryStream is
usually very error-prone. Basically if your XML is in .NET string that
means it's already UTF-16 encoded, but its XML declaration says UTF-8.
.NET supports such case just fine by switching to UTF-16.

Thank you Oleg. I hoped there was an easier way then getting involved
in encoding issues but just couldn't see it.