SP1 Problem SOAPException doesn't return quote and Umlaute correcty

Dany

Our web service was working fine until we installed .net Framework 1.1 service pack 1. Uninstalling SP1 is not an option because our largest customer says service packs marked as "critical" by Microsoft must be installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct. This worked fine in .net 1.1 without the SP1. The correct <faultstring> should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception message of "Fahrvergnügen". It is converted incorrectly to <faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error messages will have special characters like ö,ä, é, à in them and these are not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #1

Subscribe Post Reply

4853

Dan Rogers

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #2

Dany

Hi Dan

Yes, I think you're right. I went through the XML 1.0 Specs and it is correct XML as such.

The problem seems to be on the client side. We use MSXML-DOM to parse the SOAPFault, but when getting nodeValue it does not correctly convert these symbols to their wide string representation. Will have to look into that further and probably post it in the MSXML newsgroup if this is really the problem.

Do you think using a SOAPExtension to replace all the special chars with the "real" representation like they were before .net Framework Service Pack 1 would work? Or is the conversion done afterwards? This would give us the possibility to update only the server. Updating all clients is a huge enterprise with our biggest customer.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #3

Dany

Found out .net web service clients are just as helpless with handling these character references.

I've built a simple web service consumer in VC#. On a button click the following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{
MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method "HelloWorld" is under construction.' as the exception message. The user will be shown 'Error has occured: The method "HelloWorld" is under construction. In a real world example the user would be sure to complain about this display.

Before SP1 everything was fine, but now handling of exception messages coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special characters are correctly sent. So if 'The method "HelloWorld" is under construction.' is returned the client will also receive it unchanged and without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany

Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #4

Dan Rogers

Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany

Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Nov 23 '05 #5

Dany

Hi Dan

The more I look into it the more I get the impression there is really something very wrong here in the .net Framework SP1. I've been talking to an XML specialist and he noticed that it is not correct XML generated when there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this. Now when the XML Fault is read through an XML parser the & is correctly converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net Framework before landing on the wire. The first time everything would be Ok and the ü character is converted to a valid character reference "ü". But then this is replaced by character references once again which results in "&#252;" which is of course not a valid XML representation of the "ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the .net Framework. I'd be really thankful if you could check with the serialization team. Reproducing the error should be quite simple. Hope they have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #6

Dan Rogers

Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
.net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Nov 23 '05 #7

Dany

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """, "<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is created here. So I don't understand what we could do differently. Please elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP response and falsly converts the characters in the fault string to XML twice. The fact that a pure .net consumer can't even handle converting the fault string back to a correct string seems to prove, that it can't be correct.

We have done a workaround (=hack) now, assuming that MS won't be able to fix the bug quick enough. We've created a SOAP Extension that takes the SOAP response and undoes the wrongly converted characters. But this is by no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{
MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message box with the incorrectly converted error message. If you use a proxy to analyse the SOAP request and response you can see that the fault string in the SOAPFault is: This is a quote &quot; or this is a lesser than symbol &lt; or these are german Umlaute &#246;, &#228;, &#252;
This is of course the wrong XML representation and should actually be: This is a quote " or this is a lesser than symbol < or these are german Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
..net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #8

Dany

Hi Dan

An error has crept into my last post. The HelloWorld web service implementation should be:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a quote \" or this is a lesser than symbol < or these are german Umlaute ö, ä, ü";
throw new SoapException( msgStr,new XmlQualifiedName("Server.Problem"));
}

Regards
Dany

Dany<Da**@newsgroups.com> 14.12.2004 13:53:37 >>> Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """, "<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is created here. So I don't understand what we could do differently. Please elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP response and falsly converts the characters in the fault string to XML twice. The fact that a pure .net consumer can't even handle converting the fault string back to a correct string seems to prove, that it can't be correct.

We have done a workaround (=hack) now, assuming that MS won't be able to fix the bug quick enough. We've created a SOAP Extension that takes the SOAP response and undoes the wrongly converted characters. But this is by no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{
MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message box with the incorrectly converted error message. If you use a proxy to analyse the SOAP request and response you can see that the fault string in the SOAPFault is: This is a quote &quot; or this is a lesser than symbol &lt; or these are german Umlaute &#246;, &#228;, &#252;
This is of course the wrong XML representation and should actually be: This is a quote " or this is a lesser than symbol < or these are german Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
..net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #9

Dan Rogers

Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
.net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Nov 23 '05 #10

Dany

Hi Dan

I can only supply the XML for the Fault/detail element, not for the Fault/faultstring element. The Fault/faultstring is by definition the human-readable explanation of why the Fault occurred. That is the string we need. But this is the string, that is converted incorrectly from .net Framework SP1. So I don't see where I can provide an XML Node for the faultstring (not the fault detail). Or have I not understood your explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be fixed in the .net framework. And believe me I've studied this problem quite intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany

Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
..net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #11

Dan Rogers

Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the
unicode string is being URLEncoded as part of converting it to XML. To
display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the
human-readable explanation of why the Fault occurred. That is the string we
need. But this is the string, that is converted incorrectly from .net
Framework SP1. So I don't see where I can provide an XML Node for the
faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be
fixed in the .net framework. And believe me I've studied this problem quite
intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany

Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #12

Dany

Hi Dan

The problem is, that the web service provides for different clients, and I can't change them all. Also it is very difficult telling the implementers of the clients to make a hack (decoding twice instead of once) to correct a bug in the webservice on their side dependent of the server having .net framework SP1 installed or not. At the moment we've resorted to implementing a SOAP Extension to correct the error on the server side. But this is just a workaround for the moment and performance-wise not ideal. Also we have to to check if .net Framework SP1 is installed on installation of a server and only then install the SOAP Extension. As you can imagine this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a chance of MS fixing the bug at least for a future service pack or version?

Regards
Dany

Dan Rogers<da***@microsoft.com> 15.12.2004 18:58:39 >>> Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the
unicode string is being URLEncoded as part of converting it to XML. To
display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the
human-readable explanation of why the Fault occurred. That is the string we
need. But this is the string, that is converted incorrectly from .net
Framework SP1. So I don't see where I can provide an XML Node for the
faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be
fixed in the .net framework. And believe me I've studied this problem quite
intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
..net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #13

Dan Rogers

Hi Dany,

I do understand your desire to see the behavior change. You've explained
your perspective well. I think you'll find though, that the behavior you
are seeing _is_ the bug fix. The point of view that makes that the current
state stems from the belief that it is NOT unreasonable to expect a web
client to UrlDecode data that was passed over the wire before displaying it
to the end user.

At the core of it is : Should SOAP fault description strings be intended
to be end use ready?

I understand your argument for the answer being "YES".

I'm glad you've found a work around,

Dan

--------------------
Date: Thu, 16 Dec 2004 11:28:08 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#H**************@TK2MSFTNGP12.phx.gbl>
<Uu**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <Oa**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8127
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The problem is, that the web service provides for different clients, and I
can't change them all. Also it is very difficult telling the implementers
of the clients to make a hack (decoding twice instead of once) to correct a
bug in the webservice on their side dependent of the server having .net
framework SP1 installed or not. At the moment we've resorted to
implementing a SOAP Extension to correct the error on the server side. But
this is just a workaround for the moment and performance-wise not ideal.
Also we have to to check if .net Framework SP1 is installed on installation
of a server and only then install the SOAP Extension. As you can imagine
this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a
chance of MS fixing the bug at least for a future service pack or version?

Regards
Dany

Dan Rogers<da***@microsoft.com> 15.12.2004 18:58:39 >>> Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the
unicode string is being URLEncoded as part of converting it to XML. To
display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the
human-readable explanation of why the Fault occurred. That is the string we
need. But this is the string, that is converted incorrectly from .net
Framework SP1. So I don't see where I can provide an XML Node for the
faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be
fixed in the .net framework. And believe me I've studied this problem quite
intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
.net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
.phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Nov 23 '05 #14

Dany

Hi Dan

Sorry, for being so insistant, but I think I might not have made the problem clear enough.

You are absolutely correct, that the SOAP client needs to URL decode the fault description. This is what for example a .net webservice consumer will also do and this works correctly if the web service is not running on a server with .net Framework SP1.
But the way it is now in SP1 it would have to be decoded twice, which of course a web service caller will not do normally and is not according to W3C SOAP specs.

The SOAP fault string returned in the SOAP fault contains f.e. a quote (") as:

&quot;

Now the normal decoding which is done will generate:

"

But this is still not enough the client would have to decode once again to receive a ".

The need to decode a message 2 times is just not correct. Correct would be the SOAP response containing:

"

This is then decoded on the client side correctly to a " before displaying to the user.

I've tested this with different implementations of web service clients and non of them does the conversion twice. I have also checked the SOAP and XML specs and they all state that the correct representation of a doublequote is " and not &quot;.

So there is no way to see the new behavior as a bug fix, it is a bug. The client is not required to decode the message TWICE before being able to display it to the end user.

Hope the problem is more clear now. Or is there an error in the example or my reasoning?

Regards
Dany

Dan Rogers<da***@microsoft.com> 16.12.2004 22:26:11 >>> Hi Dany,

I do understand your desire to see the behavior change. You've explained
your perspective well. I think you'll find though, that the behavior you
are seeing _is_ the bug fix. The point of view that makes that the current
state stems from the belief that it is NOT unreasonable to expect a web
client to UrlDecode data that was passed over the wire before displaying it
to the end user.

At the core of it is : Should SOAP fault description strings be intended
to be end use ready?

I understand your argument for the answer being "YES".

I'm glad you've found a work around,

Dan

--------------------
Date: Thu, 16 Dec 2004 11:28:08 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#H**************@TK2MSFTNGP12.phx.gbl>
<Uu**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <Oa**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8127
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The problem is, that the web service provides for different clients, and I
can't change them all. Also it is very difficult telling the implementers
of the clients to make a hack (decoding twice instead of once) to correct a
bug in the webservice on their side dependent of the server having .net
framework SP1 installed or not. At the moment we've resorted to
implementing a SOAP Extension to correct the error on the server side. But
this is just a workaround for the moment and performance-wise not ideal.
Also we have to to check if .net Framework SP1 is installed on installation
of a server and only then install the SOAP Extension. As you can imagine
this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a
chance of MS fixing the bug at least for a future service pack or version?

Regards
Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 18:58:39 >>> Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the
unicode string is being URLEncoded as part of converting it to XML. To
display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the
human-readable explanation of why the Fault occurred. That is the string we
need. But this is the string, that is converted incorrectly from .net
Framework SP1. So I don't see where I can provide an XML Node for the
faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be
fixed in the .net framework. And believe me I've studied this problem quite
intensly. Just try the example I posted, maybe then you'll see what I mean.

Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and
then let the default system convert it to XML, the first thing it's going
to do is URLEncode the string, and then include THAT in the fault node.
Try creating the XML yourself (Xml.Document.CreateNode .. or something like
that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.

It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

where errMsg is a normal string, which can contain characters like """,
"<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is
created here. So I don't understand what we could do differently. Please
elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP
response and falsly converts the characters in the fault string to XML
twice. The fact that a pure .net consumer can't even handle converting the
fault string back to a correct string seems to prove, that it can't be
correct.

We have done a workaround (=hack) now, assuming that MS won't be able to
fix the bug quick enough. We've created a SOAP Extension that takes the
SOAP response and undoes the wrongly converted characters. But this is by
no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS
who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a
quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new
XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message
box with the incorrectly converted error message. If you use a proxy to
analyse the SOAP request and response you can see that the fault string in
the SOAPFault is: This is a quote &quot; or this is a lesser than
symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This
is a quote " or this is a lesser than symbol < or these are german
Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the
fault string on the wire. What it will do is take an XML node and insert
it into the proper place in a fault. It sounds like there may be the case
where the code you are using to set up the XML is double converting. This
could happen if you try and treat a string with embedded angle brackets in
it like XML. Try creating the fault node data by using XmlElement's if
this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really
something very wrong here in the .net Framework SP1. I've been talking to
an XML specialist and he noticed that it is not correct XML generated when
there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie
ein Abreisedatum für das Dokument." is represented in the SOAP fault as

<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this.
Now when the XML Fault is read through an XML parser the & is correctly
converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum
für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net
Framework before landing on the wire. The first time everything would be Ok
and the ü character is converted to a valid character reference "ü".
But then this is replaced by character references once again which results
in "&#252;" which is of course not a valid XML representation of the
"ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the
..net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they
have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the
XML. Is there an inconsistency between the XML encoding mark up and the
actual encoding on the wire? UrlEncoded XML is supported by the DOM, but
if someone somewhere along the way is loading what looks like the XML into
a unicode string without doing an encoding conversion first, issues can
happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1
fixed bugs. It is possible that it introduced some as well. I'll see what
I can dig up from the serialization team - can't promise they have a fix,
but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute
correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these
character references.

I've built a simple web service consumer in VC#. On a button click the
following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method
"HelloWorld" is under construction.' as the exception message. The user
will be shown 'Error has occured: The method "HelloWorld" is
under construction. In a real world example the user would be sure to
complain about this display.

Before SP1 everything was fine, but now handling of exception messages
coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special
characters are correctly sent. So if 'The method "HelloWorld" is under
construction.' is returned the client will also receive it unchanged and
without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.

Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>

Hi,

What you are describing is the use of encoded strings. These are, to my
knowlege, valid in XML and the responsibility of the parser to convert
properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these
properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14
..phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1
service pack 1. Uninstalling SP1 is not an option because our largest
customer says service packs marked as "critical" by Microsoft must be
installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP
fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));

The following SOAP response is created when calling this SOAP-Exception:

HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct.
This worked fine in .net 1.1 without the SP1. The correct <faultstring>
should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception
message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error
messages will have special characters like ö,ä, é, à in them and these are
not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #15

Dave

Dany,

I too am experiencing this problem with .Net 1.1 SP1. There definitely
is some double encoding of special characters going on.

Our app returns XML in the fault string. Our client uses the
webservice behavior. Before SP1, we could have a webservice that
returned <whatever> in the fault string of a soap exception. This
would get properly encoded as <whatever>. You can verify that by
looking at the webservice behavior's errorDetail.raw.xml - you can see
<whatever> in the faultstring element. errorDetail.string would
properly return <whatever>.

Now, with SP1, errorDetail.raw.xml shows &lt;whatever&gt;.
This very clearly indicates that SP1 causes special characters to be
encoded twice. This bug broke some code that parses the fault string
on the client.

Dave

Dany wrote:

Hi Dan

Sorry, for being so insistant, but I think I might not have made the problem clear enough.
You are absolutely correct, that the SOAP client needs to URL decode the fault description. This is what for example a .net webservice
consumer will also do and this works correctly if the web service is
not running on a server with .net Framework SP1. But the way it is now in SP1 it would have to be decoded twice, which of course a web service caller will not do normally and is not
according to W3C SOAP specs.
The SOAP fault string returned in the SOAP fault contains f.e. a quote (") as:
&quot;

Now the normal decoding which is done will generate:

"

But this is still not enough the client would have to decode once again to receive a ".
The need to decode a message 2 times is just not correct. Correct would be the SOAP response containing:
"

This is then decoded on the client side correctly to a " before displaying to the user.
I've tested this with different implementations of web service clients and non of them does the conversion twice. I have also checked
the SOAP and XML specs and they all state that the correct
representation of a doublequote is " and not &quot;.
So there is no way to see the new behavior as a bug fix, it is a bug. The client is not required to decode the message TWICE before being
able to display it to the end user.
Hope the problem is more clear now. Or is there an error in the example or my reasoning?
Regards
Dany
Dan Rogers<da***@microsoft.com> 16.12.2004 22:26:11 >>> Hi Dany,

I do understand your desire to see the behavior change. You've explained your perspective well. I think you'll find though, that the behavior you are seeing _is_ the bug fix. The point of view that makes that the current state stems from the belief that it is NOT unreasonable to expect a web client to UrlDecode data that was passed over the wire before displaying it to the end user.

At the core of it is : Should SOAP fault description strings be intended to be end use ready?

I understand your argument for the answer being "YES".

I'm glad you've found a work around,

Dan

--------------------
Date: Thu, 16 Dec 2004 11:28:08 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#H**************@TK2MSFTNGP12.phx.gbl>
<Uu**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <Oa**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8127
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The problem is, that the web service provides for different clients, and I can't change them all. Also it is very difficult telling the implementers of the clients to make a hack (decoding twice instead of once) to correct a bug in the webservice on their side dependent of the server having ..net framework SP1 installed or not. At the moment we've resorted to
implementing a SOAP Extension to correct the error on the server side. But this is just a workaround for the moment and performance-wise not ideal. Also we have to to check if .net Framework SP1 is installed on installation of a server and only then install the SOAP Extension. As you can imagine this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a chance of MS fixing the bug at least for a future service pack or version?
Regards
Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 18:58:39 >>> Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the unicode string is being URLEncoded as part of converting it to XML. To display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the human-readable explanation of why the Fault occurred. That is the string we need. But this is the string, that is converted incorrectly from .net Framework SP1. So I don't see where I can provide an XML Node for the faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be fixed in the .net framework. And believe me I've studied this problem quite intensly. Just try the example I posted, maybe then you'll see what I mean.
Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and then let the default system convert it to XML, the first thing it's going to do is URLEncode the string, and then include THAT in the fault node. Try creating the XML yourself (Xml.Document.CreateNode .. or something like that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.
It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));
where errMsg is a normal string, which can contain characters like """, "<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is created here. So I don't understand what we could do differently. Please elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP response and falsly converts the characters in the fault string to XML twice. The fact that a pure .net consumer can't even handle converting the fault string back to a correct string seems to prove, that it can't be correct.

We have done a workaround (=hack) now, assuming that MS won't be able to fix the bug quick enough. We've created a SOAP Extension that takes the SOAP response and undoes the wrongly converted characters. But this is by no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message box with the incorrectly converted error message. If you use a proxy to analyse the SOAP request and response you can see that the fault string in the SOAPFault is: This is a quote &quot; or this is a lesser than symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This is a quote " or this is a lesser than symbol < or these are german Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the fault string on the wire. What it will do is take an XML node and insert it into the proper place in a fault. It sounds like there may be the case where the code you are using to set up the XML is double converting. This could happen if you try and treat a string with embedded angle brackets in it like XML. Try creating the fault node data by using XmlElement's if this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really something very wrong here in the .net Framework SP1. I've been talking to an XML specialist and he noticed that it is not correct XML generated when there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie ein Abreisedatum für das Dokument." is represented in the SOAP fault as
<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this. Now when the XML Fault is read through an XML parser the & is correctly converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net Framework before landing on the wire. The first time everything would be Ok and the ü character is converted to a valid character reference "ü". But then this is replaced by character references once again which results in "&#252;" which is of course not a valid XML representation of the "ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the .net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the XML. Is there an inconsistency between the XML encoding mark up and the actual encoding on the wire? UrlEncoded XML is supported by the DOM, but if someone somewhere along the way is loading what looks like the XML into a unicode string without doing an encoding conversion first, issues can happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1 fixed bugs. It is possible that it introduced some as well. I'll see what I can dig up from the serialization team - can't promise they have a fix, but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these character references.

I've built a simple web service consumer in VC#. On a button click the following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method "HelloWorld" is under construction.' as the exception message. The user will be shown 'Error has occured: The method "HelloWorld" is under construction. In a real world example the user would be sure to complain about this display.

Before SP1 everything was fine, but now handling of exception messages coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special characters are correctly sent. So if 'The method "HelloWorld" is under construction.' is returned the client will also receive it unchanged and without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.
Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>
Hi,

What you are describing is the use of encoded strings. These are, to

my knowlege, valid in XML and the responsibility of the parser to convert properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1 service pack 1. Uninstalling SP1 is not an option because our largest customer says service packs marked as "critical" by Microsoft must be installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));
The following SOAP response is created when calling this SOAP-Exception:
HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct. This worked fine in .net 1.1 without the SP1. The correct <faultstring> should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error messages will have special characters like ö,ä, é, à in them and these are not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #16

Dany

Hi Dave

I'm glad at least someone else is experiencing the same problems. I almost started to believe either we are the only ones using the SOAPException fault string with special characters or almost no one uses SOAP Webservices in .net productivly today.

I'm still waiting for MS to acknowledge this bug. Don't know how to get through to them. It's quite frustrating.

Regards
Dany

Dave<dh***@bakerhill.com> 17.12.2004 20:56:07 >>>
Dany,

I too am experiencing this problem with .Net 1.1 SP1. There definitely
is some double encoding of special characters going on.

Our app returns XML in the fault string. Our client uses the
webservice behavior. Before SP1, we could have a webservice that
returned <whatever> in the fault string of a soap exception. This
would get properly encoded as <whatever>. You can verify that by
looking at the webservice behavior's errorDetail.raw.xml - you can see
<whatever> in the faultstring element. errorDetail.string would
properly return <whatever>.

Now, with SP1, errorDetail.raw.xml shows &lt;whatever&gt;.
This very clearly indicates that SP1 causes special characters to be
encoded twice. This bug broke some code that parses the fault string
on the client.

Dave

Dany wrote:
Hi Dan

Sorry, for being so insistant, but I think I might not have made the problem clear enough.
You are absolutely correct, that the SOAP client needs to URL decode the fault description. This is what for example a .net webservice
consumer will also do and this works correctly if the web service is
not running on a server with .net Framework SP1. But the way it is now in SP1 it would have to be decoded twice, which of course a web service caller will not do normally and is not
according to W3C SOAP specs.
The SOAP fault string returned in the SOAP fault contains f.e. a quote (") as:
&quot;

Now the normal decoding which is done will generate:

"

But this is still not enough the client would have to decode once again to receive a ".
The need to decode a message 2 times is just not correct. Correct would be the SOAP response containing:
"

This is then decoded on the client side correctly to a " before displaying to the user.
I've tested this with different implementations of web service clients and non of them does the conversion twice. I have also checked
the SOAP and XML specs and they all state that the correct
representation of a doublequote is " and not &quot;.
So there is no way to see the new behavior as a bug fix, it is a bug. The client is not required to decode the message TWICE before being
able to display it to the end user.
Hope the problem is more clear now. Or is there an error in the example or my reasoning?
Regards
Dany
Dan Rogers<da***@microsoft.com> 16.12.2004 22:26:11 >>> Hi Dany,

I do understand your desire to see the behavior change. You've explained your perspective well. I think you'll find though, that the behavior you are seeing _is_ the bug fix. The point of view that makes that the current state stems from the belief that it is NOT unreasonable to expect a web client to UrlDecode data that was passed over the wire before displaying it to the end user.

At the core of it is : Should SOAP fault description strings be intended to be end use ready?

I understand your argument for the answer being "YES".

I'm glad you've found a work around,

Dan

--------------------
Date: Thu, 16 Dec 2004 11:28:08 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#H**************@TK2MSFTNGP12.phx.gbl>
<Uu**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <Oa**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8127
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The problem is, that the web service provides for different clients, and I can't change them all. Also it is very difficult telling the implementers of the clients to make a hack (decoding twice instead of once) to correct a bug in the webservice on their side dependent of the server having ..net framework SP1 installed or not. At the moment we've resorted to
implementing a SOAP Extension to correct the error on the server side. But this is just a workaround for the moment and performance-wise not ideal. Also we have to to check if .net Framework SP1 is installed on installation of a server and only then install the SOAP Extension. As you can imagine this is not the way I expect bugs to be handled.

What is the best approach (I'm an msdn universal subscriber) to have a chance of MS fixing the bug at least for a future service pack or version?
Regards
Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 18:58:39 >>> Hi Dany,

Given your example, I expect the behavior you cite. The issue is that the unicode string is being URLEncoded as part of converting it to XML. To display it, try URLDecoding it.

Dan
--------------------
Date: Wed, 15 Dec 2004 15:20:23 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<#f**************@tk2msftngp13.phx.gbl>
<iw**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <#H**************@TK2MSFTNGP12.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP12 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8106
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

I can only supply the XML for the Fault/detail element, not for the
Fault/faultstring element. The Fault/faultstring is by definition the human-readable explanation of why the Fault occurred. That is the string we need. But this is the string, that is converted incorrectly from .net Framework SP1. So I don't see where I can provide an XML Node for the faultstring (not the fault detail). Or have I not understood your
explanation correctly?

I'm still absolutely sure there is a bug here in SP1 which needs to be fixed in the .net framework. And believe me I've studied this problem quite intensly. Just try the example I posted, maybe then you'll see what I mean.
Dany
Dan Rogers<da***@microsoft.com> 15.12.2004 01:56:27 >>> Hi Dany,

You can manuallyl supply the XML for the fault node. The issue is
converting a unicode string to XML. When you have unicode in strings, and then let the default system convert it to XML, the first thing it's going to do is URLEncode the string, and then include THAT in the fault node. Try creating the XML yourself (Xml.Document.CreateNode .. or something like that). And populating the fault yourself. I think you'll find that
SoapException has some overrides in the constructor that let you do this.
It's been a while... but I remember doing this some time back myself.

Dan
--------------------
Date: Tue, 14 Dec 2004 13:53:37 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<et**************@TK2MSFTNGP09.phx.gbl>
<03*************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <#f**************@tk2msftngp13.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!tk2msftngp13 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:8068
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

We are not creating any fault node data. All we do is:

throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));
where errMsg is a normal string, which can contain characters like """, "<", ">", "ä","ü", etc.

The way I understand, we have no influence as to how the SOAP response is created here. So I don't understand what we could do differently. Please elaborate if you have an idea.

The way we see it the SOAP subsystem serializes the SOAPException to a SOAP response and falsly converts the characters in the fault string to XML twice. The fact that a pure .net consumer can't even handle converting the fault string back to a correct string seems to prove, that it can't be correct.

We have done a workaround (=hack) now, assuming that MS won't be able to fix the bug quick enough. We've created a SOAP Extension that takes the SOAP response and undoes the wrongly converted characters. But this is by no meens something I want to have to keep in a productive system.

I really would appreciate if you could pass this bug on to someone in MS who could take care of it.
Reproducing is very easy:
1. In VS create a new web service project.
2. Add a HelloWorld webservice method as so:
[WebMethod]
public string HelloWorld()
{
string msgStr = "This is a quote: ""
throw new SoapException( "Now I've got a problem \"here\". ><",new XmlQualifiedName("Server.Problem"));
}
3. Create a new WindowsApplication
4. Add a web reference to the web service created above (call it
TestWebService)
5. Add a button and add the following code for it's click handler
private void button1_Click(object sender, System.EventArgs e)
{
try
{
TestWebService.Service1 appserv = new TestWebService.Service1();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( exc.Message );
}
}
6. Now run the client and after pressing the button1 you will see a message box with the incorrectly converted error message. If you use a proxy to analyse the SOAP request and response you can see that the fault string in the SOAPFault is: This is a quote &quot; or this is a lesser than symbol &lt; or these are german Umlaute &#246;, &#228;,
&#252;
This is of course the wrong XML representation and should actually be: This is a quote " or this is a lesser than symbol < or these are german Umlaute ö, ä, ü

Regards
Dany

Dan Rogers<da***@microsoft.com> 13.12.2004 21:12:11 >>> Hi Dany,

The SOAP subsystem doesn't actually do much when it comes to getting the fault string on the wire. What it will do is take an XML node and insert it into the proper place in a fault. It sounds like there may be the case where the code you are using to set up the XML is double converting. This could happen if you try and treat a string with embedded angle brackets in it like XML. Try creating the fault node data by using XmlElement's if this is the case.

I hope this helps

Dan
--------------------
Date: Wed, 8 Dec 2004 09:11:03 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<eD**************@TK2MSFTNGP15.phx.gbl>
<#1**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <et**************@TK2MSFTNGP09.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTNGXA01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP09 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7978
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Hi Dan

The more I look into it the more I get the impression there is really something very wrong here in the .net Framework SP1. I've been talking to an XML specialist and he noticed that it is not correct XML generated when there is an Umlaut (ä, ö, ü) f.e. the error message for "Bitte erfassen sie ein Abreisedatum für das Dokument." is represented in the SOAP fault as
<faultstring>Bitte erfassen sie ein Abreisedatum f&#252;r das
Dokument.</faultstring>

Only when using a proxy to log the messages on the wire did I notice this. Now when the XML Fault is read through an XML parser the & is correctly converted to a "&" and the result is "Bitte erfassen sie ein Abreisedatum für das Dokument." which is of course wrong after a conversion.

So it looks like the SOAP fault string is converted twice by the .net Framework before landing on the wire. The first time everything would be Ok and the ü character is converted to a valid character reference "ü". But then this is replaced by character references once again which results in "&#252;" which is of course not a valid XML representation of the "ü" character.

In my view this is a serious bug in the handling of SOAPExceptions by the .net Framework. I'd be really thankful if you could check with the
serialization team. Reproducing the error should be quite simple. Hope they have a solution then.

Thanks & Regards
Dany

Dan Rogers<da***@microsoft.com> 07.12.2004 21:36:25 >>> Hi Dany,

I suspect that there is an encoding issue in the way you are loading the XML. Is there an inconsistency between the XML encoding mark up and the actual encoding on the wire? UrlEncoded XML is supported by the DOM, but if someone somewhere along the way is loading what looks like the XML into a unicode string without doing an encoding conversion first, issues can happen.

As for how/why this happened - I'm not that guy. For the most part, 1.1 fixed bugs. It is possible that it introduced some as well. I'll see what I can dig up from the serialization team - can't promise they have a fix, but I'll see.

Dan
--------------------
Date: Tue, 7 Dec 2004 14:00:36 +0100
References: <uC**************@TK2MSFTNGP14.phx.gbl>
<hz**************@cpmsftngxa10.phx.gbl>
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: RE: SP1 Problem SOAPException doesn't return quote and Umlaute correcty
Message-ID: <eD**************@TK2MSFTNGP15.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: gonzo.opus.ch 213.200.237.2
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED01.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP15 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7944
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Found out .net web service clients are just as helpless with handling these character references.

I've built a simple web service consumer in VC#. On a button click the following code is executed:
try
{
localhost.UmbrellaNTAppServer appserv = new
localhost.UmbrellaNTAppServer();
string result = appserv.HelloWorld();
MessageBox.Show( result );
}
catch (Exception exc)
{

MessageBox.Show( "Error has occured: " + exc.Message );
}

If an exception is raised in the web service method saying i.e. 'The method "HelloWorld" is under construction.' as the exception message. The user will be shown 'Error has occured: The method "HelloWorld" is under construction. In a real world example the user would be sure to complain about this display.

Before SP1 everything was fine, but now handling of exception messages coming from web service calls must be handled specifically.
In normal SOAP responses (not SOAP faults) strings containing special characters are correctly sent. So if 'The method "HelloWorld" is under construction.' is returned the client will also receive it unchanged and without any character references.

So I wonder why this inconsistent behaviour was built into the SP1 by MS.
Regards Dany
Dan Rogers<da***@microsoft.com> 06.12.2004 21:42:15 >>>
Hi,

What you are describing is the use of encoded strings. These are, to

my knowlege, valid in XML and the responsibility of the parser to convert properly.

What is the source of the problem, given that these are valid
representations of the data? Is the client not able to process these properly as it should?

Regards

Dan Rogers
Microsoft Corporation

--------------------
Date: Sat, 4 Dec 2004 13:43:30 +0100
X-Newsreader: Groupwise 6.5
From: Dany <Da**@newsgroups.com>
Subject: SP1 Problem SOAPException doesn't return quote and Umlaute correcty Message-ID: <uC**************@TK2MSFTNGP14.phx.gbl>
Newsgroups: microsoft.public.dotnet.framework.webservices
NNTP-Posting-Host: adsl-62-167-138-146.adslplus.ch 62.167.138.146
Lines: 1
Path:
cpmsftngxa10.phx.gbl!TK2MSFTFEED02.phx.gbl!TK2MSFT NGP08.phx.gbl!TK2MSFTNGP14 .phx.gbl
Xref: cpmsftngxa10.phx.gbl
microsoft.public.dotnet.framework.webservices:7851
X-Tomcat-NG: microsoft.public.dotnet.framework.webservices

Our web service was working fine until we installed .net Framework 1.1 service pack 1. Uninstalling SP1 is not an option because our largest customer says service packs marked as "critical" by Microsoft must be installed on their servers.

Now german Umlaute (ä, ü, ö) and quotes are returned incorrectly in SOAP fault responses.
This can be easily verified:

Implement the following in a web service method (just raises a
SOAPException with a message text containing quote characters ("):

string errMsg = "Here is a \"quote\".";
throw new SoapException( errMsg,new XmlQualifiedName("Server.Problem"));
The following SOAP response is created when calling this SOAP-Exception:
HTTP/1.1 500 Internal Server Error.
Server: Microsoft-IIS/5.1
Date: Fri, 03 Dec 2004 13:02:23 GMT
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Content-Length: 431

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>Server.Problem</faultcode>
<faultstring>Here is a "quote".</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

All though the encoding should be utf-8 and "& q u o t ;" is not correct. This worked fine in .net 1.1 without the SP1. The correct <faultstring> should be:
<faultstring>Here is a "quote".</faultstring>

This problem also occurs when using german Umlaute: f.e. with an exception message of "Fahrvergnügen". It is converted incorrectly to
<faultstring>Fahrvergn&#252;gen</faultstring>.

This is a serious problem for us, because in german and french many error messages will have special characters like ö,ä, é, à in them and these are not correctly processed.

Thanks for any help!
Dany

Nov 23 '05 #17

SP1 Problem SOAPException doesn't return quote and Umlaute correcty

Similar topics