XmlSerializer Question

[preport]

I have a webservice that returns data from a database.

Our services (the clients) have been blowing up because of illegal
character problems. Is there anything I can do on the server side to
work around this problem. I'm OK with deleting all the "bad"
characters.

I have a couple DTOs that are marked with [Serializable()] that I
return from the web service after they're populated with data from the
database. Is there some kind of attribute or "easy" way to stip out
illegal characters?

What is weird to me, is that the XmlSerializer serializes it just fine,
the clients are the ones blowing up. Here is the error I'm getting:

---------------------------------

System.InvalidOperationException: There is an error in XML document
(230, 966838). ---System.Xml.XmlException: '', hexadecimal value
0x04, is an invalid character. Line 230, position 966862.
at System.Xml.XmlScanner.ScanHexEntity()
at System.Xml.XmlTextReader.ParseBeginTagExpandCharEn tities()
at System.Xml.XmlTextReader.Read()
at System.Xml.XmlReader.ReadElementString()
at
Microsoft.Xml.Serialization.GeneratedAssembly.XmlS erializationReader1.Read11
_EVariable(Boolean isNullable, Boolean checkType) at
Microsoft.Xml.Serialization.GeneratedAssembly.XmlS erializationReader1.Read9_
EmailSpec(Boolean isNullable, Boolean checkType) at
Microsoft.Xml.Serialization.GeneratedAssembly.XmlS erializationReader1.Read8_
EmailInfo(Boolean isNullable, Boolean checkType) at
Microsoft.Xml.Serialization.GeneratedAssembly.XmlS erializationReader1.Read2_
CRobe(Boolean isNullable, Boolean checkType) at
Microsoft.Xml.Serialization.GeneratedAssembly.XmlS erializationReader1.Read28
_CRobeGetForGuidResponse()

--- End of inner exception stack trace ---

I do populate the object properties by looping through each DataRow in
a DataSet and accessing the dr[column]. Is there a function that will
stip out "illegal" characters?

Any help would be greatly appreciated.

[preport]

I found this article:

http://msdn.microsoft.com/library/de...trblshtxsd.asp

It says:
You can avoid this problem if you deserialize with an XmlTextReader that has
its Normalization property set to true. Unfortunately, the XmlTextReader
used under the covers by ASP.NET Web services has its Normalization property
set to false; i.e., it will not deserialize SOAP messages containing these
invalid characters.

Does this mean that I'm SOL?

[Steven Cheng[MSFT]]

Hello preport,

From your description, I understand you have an ASP.NET webservice which
transfer some data(retrieved from database) from server to client. However,
it currently report exception when the response data contains some invalid
characters, correct?

Based on my experience, the problem you suffer here is likely due to you
directly expose large blob data(binary or text) through webservice's
paramter or return value. Since webservice request/response data is
encapsulated in XML based SOAP message, if there contains some invalid
byte/chars (such as '<', '>', '&' ...) in the message data, it will cause
error.
For your scenario, I'd like to confirm the following things first:

** how did you define your custom class which contains the database data

** have you already manually encode the data being transfered in
webservice's code?

If convenient you can post some code snippet here so that I can get a clear
view. Also, marking the class as "Serializable" does not help here because
webservice will serialize the class objects through XMLSerialization while
the "Serializable" attribute is used for binary seriailzation.

For this scenario, generally there are two approaches here:

1. In our webservice's code, we can manually use some string or binary
encoding methods to encode the data we would transfer in our custom class's
property, and at client-side, we can decode it. For example, we can use
Convert.ToBase64String to convert binary data into base64 string which is
safe to transfer in webservice. We can also use HttpUtility.HtmlEncode to
make the string/text data safe to be transfer(those particular chars will
be escaped...).

Also, if you do want to send large binary data in webservice, you can
consider using the WSE component (latest version 3.0 for .net framework
2.0).
#Web Services Enhancements (WSE) 3.0 for Microsoft .NET
http://www.microsoft.com/downloads/d...9fd-3a74-43c5-
8ec1-8d789091255d&displaylang=en
2. In .net framework 2.0, it expose the "IXmlSerializable" interface to let
us customize our custom class's XML Serialization process. thus, for your
scenario, you can also consider let your custom class which contains the
database data(contains the invalid chars) and then add the
encoding/escaping code in the IXmlSerialable interface's implementation
methods.

#Customizing XML Serialization in .NET 2.0 Using the IXmlSerializable
Interface
http://www.devx.com/dotnet/Article/29720

#How to: Send and Receive Large Amounts of Data to and from a Web Service
http://msdn.microsoft.com/library/en...4e4-4056-906d-
72504ed8c0df.asp?frame=true

Please feel free to let me know if there is anything I missed or if you
have any further questions.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead

==================================================

Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscripti...ult.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscripti...t/default.aspx.

==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

[preport]

From your description, I understand you have an ASP.NET webservice which

transfer some data(retrieved from database) from server to client.
However,
it currently report exception when the response data contains some invalid
characters, correct?

Exactly Correct.

** how did you define your custom class which contains the database data

The particular object that blows up has a collection of these:

[Serializable]
public class EVariable
{
private string _name;
private string _value;

public EVariable() { }

public EVariable(string name, string value)
{
this._name = name;
this._value = value;
}

public string Name
{
get { return this._name; }
set { this._name = value; }
}

public string Value
{
get { return this._value; }
set { this._value = value; }
}

}

** have you already manually encode the data being transfered in
webservice's code?

No. I've found that everything gets encoded for me.....most of our objects
contain some kind of HTML that is sent over and when I look at the raw XML
passed to the clients, everything is already encoded.

Infact...the character that is causing the problem gets encoded as: "&#x4;",
In one particular case this is causing the clients to blow up.

While these objects tend to be large, they consist of collections of smaller
objects. What I mean is that I'm not returning an object property populated
by a 'text' or 'blob' DB field. These are small addresses 'varchar(200)'
fields in the DB.

---------

I believe the above answered most of your questions...now....I've been
posting in the csharp newsgroup about using this function to get rid of the
characters in question:

private static string formatXMLString(string n)
{
if (string.IsNullOrEmpty(n)) return n;
System.Text.StringBuilder sb = new System.Text.StringBuilder();
char[] chrs = n.ToCharArray();
char c;
int x, j = chrs.Length;
for (x = 0; x < j; x++)
{
c = chrs[x];
if (c == 0x9 || c == 0xA || c == 0xD ||
(c 0x20 && c < 0xd7ff) ||
(c 0xe000 && c < 0xffd))
{
sb.Append(c);
}
}
return sb.ToString();
}
While it solved the "Clients blowing up" problem, I've lost all the spaces
in these fields.

Any more suggestions?

[Steven Cheng[MSFT]]

Thanks for your reply preport,

After some further research, I have found some similar issues in the former
cases and for the invalid character deserialization issue in webservice, it
is occuring when we contains some invalid control character whch is stored
in a .net string variable (class field/property).

So far since we can not customize webservice's internal xmlserializaer(as
the msdn article has mentioned), what we can get are the following
workaround options:
1. Add additional custom code logic to prevent our custom class
properties(string type) from containing those invalid characters. e.g.
manually escapgting those special chars/

2. Use other type other than string to hold such text data. For example,
you can define a propety/field of "XmlElement" type and then store the
original text as inner content of an XmlElment. Or you can use System.Text
namespace's encoding class to encode such text data into byte[] array, and
pass byte array in your webservice.

You can try the above ones to see whether it works for your scenario.
Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead
This posting is provided "AS IS" with no warranties, and confers no rights.

[preport]

Thanks...I ended up running this on every property:

private static string formatXMLString(string n)
{
if (string.IsNullOrEmpty(n)) return n;
System.Text.StringBuilder sb = new System.Text.StringBuilder();
char[] chrs = n.ToCharArray();
char c;
int x, j = chrs.Length;
for (x = 0; x < j; x++)
{
c = chrs[x];
if (c == 0x9 || c == 0xA || c == 0xD ||
(c >= 0x20 && c <= 0xd7ff) ||
(c >= 0xe000 && c <= 0xffd))
{
sb.Append(c);
}
}
return sb.ToString();
}

It just strips out that "bad" characters (which is perfectly fine in my
situation). I hope this helps someone else.

[Steven Cheng[MSFT]]

Thanks for your followup preport,

Glad that you have worked out a solution.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead
This posting is provided "AS IS" with no warranties, and confers no rights.