German "Umlaute" in QueryString

[Uwe Braunholz]

Hello,

working on a asp.net Website brought me to a strange problem.

I want to enable my users to pass a search string via the query string
of an url.

It works if the user calls the URL like "default.aspx?search=mystring"
But as soon as a German umlaut is in the query string it does not
handle the char correctly (like "default.aspx?search=müstring" will be
displayed as "m?string" on the page).

I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
the display as label is:
m�string

If I call the page from another asp.net page using
Response.Redirect("Default.aspx?search=" +
Server.UrlEncode("müstring"));
the result is corretly, but the URL is changed to:
?search=m%c3%bcstring
debugging results in
Request.QueryString = {search=m%u00fcstring}

The problem is, that the users will call the URL from different
applications (not necessarily .NET) so Server.URLEncode will never be
called so the first case will occur.

How can I solve this?

Thank you!
Regards,
Uwe

[Sriram Srivatsan]

Needs to be treated as unicode. Look at requestEncoding, responseEncoding
and the related configuration entries in MSDN under globalization node in
the web.config file. May throw some light.
"Uwe Braunholz" <uw***********@hotmail.dewrote in message
news:d9**********************************@79g2000h sk.googlegroups.com...
Hello,

working on a asp.net Website brought me to a strange problem.

I want to enable my users to pass a search string via the query string
of an url.

It works if the user calls the URL like "default.aspx?search=mystring"
But as soon as a German umlaut is in the query string it does not
handle the char correctly (like "default.aspx?search=müstring" will be
displayed as "m?string" on the page).

I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
the display as label is:
m?string

If I call the page from another asp.net page using
Response.Redirect("Default.aspx?search=" +
Server.UrlEncode("müstring"));
the result is corretly, but the URL is changed to:
?search=m%c3%bcstring
debugging results in
Request.QueryString = {search=m%u00fcstring}

The problem is, that the users will call the URL from different
applications (not necessarily .NET) so Server.URLEncode will never be
called so the first case will occur.

How can I solve this?

Thank you!
Regards,
Uwe

[Uwe Braunholz]

Hello Sriram,
thank you for your answer. I thougth so too, but the default seems to
be utf-8. If I set
<globalization requestEncoding="UTF-8" responseEncoding="UTF-8" />
in web.config, nothing changes. Only if I set a different encoding
like "ISO-8859-1". But what if another user enters some greek, russian
or chinese characters?

I think fixing my application to a specific encoding is not a good
idea. I thought utf8 would be the choice for that?

Regards,
Uwe

On 11 Aug., 12:00, "Sriram Srivatsan" <[e-mail]>
wrote:

Needs to be treated as unicode. *Look at requestEncoding, responseEncoding
and the related configuration entries in MSDN under globalization node in
the web.config file. May throw some light.

"Uwe Braunholz" <[e-mail] in message

news:d9**********************************@79g2000h sk.googlegroups.com...
Hello,

working on a asp.net Website brought me to a strange problem.

I want to enable my users to pass a search string via the query string
of an url.

It works if the user calls the URL like "default.aspx?search=mystring"
But as soon as a German umlaut is in the query string it does not
handle the char correctly (like "default.aspx?search=müstring" will be
displayed as "m?string" on the page).

I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
the display as label is:
m?string

If I call the page from another asp.net page using
Response.Redirect("Default.aspx?search=" +
Server.UrlEncode("müstring"));
the result is corretly, but the URL is changed to:
?search=m%c3%bcstring
debugging results in
Request.QueryString = {search=m%u00fcstring}

The problem is, that the users will call the URL from different
applications (not necessarily .NET) so Server.URLEncode will never be
called so the first case will occur.

How can I solve this?

Thank you!
Regards,
Uwe

[Alexey Smirnov]

On Aug 11, 11:44*am, Uwe Braunholz <[e-mail]:

Hello,

working on a asp.net Website brought me to a strange problem.

I want to enable my users to pass a search string via the query string
of an url.

It works if the user calls the URL like "default.aspx?search=mystring"
But as soon as a German umlaut is in the query string it does not
handle the char correctly (like "default.aspx?search=müstring" will be
displayed as "m?string" on the page).

I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
the display as label is:
m string

If I call the page from another asp.net page using
Response.Redirect("Default.aspx?search=" +
Server.UrlEncode("müstring"));
the result is corretly, but the URL is changed to:
?search=m%c3%bcstring
debugging results in
Request.QueryString = {search=m%u00fcstring}

The problem is, that the users will call the URL from different
applications (not necessarily .NET) so Server.URLEncode will never be
called so the first case will occur.

How can I solve this?

Thank you!
Regards,
Uwe

Hallo Uwe,

generally all umlauts must be encoded (as UTF-8 in your case).

So, if your search string value is from the Search textbox, you have
to encode it as

string search = HttpUtility.UrlEncode(SearchTextBox.Text,
Encoding.UTF8));
Response.Redirect("default.aspx?search=" + search);

this would redirect you to default.aspx?search=m%c3%bcstring

In this case you would not have any problem to get the value using
HttpUtility.UrlDecode() function...

If you want to get umlauts out of the "default.aspx?search=müstring"
use RawUrl:

string search = Request.RawUrl.Split('=')[1];

and then search would return you "müstring"...

To handle both requests

default.aspx?search=m%c3%bcstring
default.aspx?search=müstring

use

string search = Request.RawUrl.Split('=')[1];
string newsearch = HttpUtility.UrlDecode(search);

[Uwe Braunholz]

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m�string"

The problem is that I do not have any control over the search-source.
This could be any application where the url is put together
automatically.

Regards,
Uwe

On 11 Aug., 13:37, Alexey Smirnov <[e-mail]:

On Aug 11, 11:44Â*am, Uwe Braunholz <[e-mail]:

Hello,

working on a asp.net Website brought me to a strange problem.

I want to enable my users to pass a search string via the query string
of an url.

It works if the user calls the URL like "default.aspx?search=mystring"
But as soon as a German umlaut is in the query string it does not
handle the char correctly (like "default.aspx?search=müstring" will be
displayed as "m?string" on the page).

I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
the display as label is:
m string

If I call the page from another asp.net page using
Response.Redirect("Default.aspx?search=" +
Server.UrlEncode("müstring"));
the result is corretly, but the URL is changed to:
?search=m%c3%bcstring
debugging results in
Request.QueryString = {search=m%u00fcstring}

The problem is, that the users will call the URL from different
applications (not necessarily .NET) so Server.URLEncode will never be
called so the first case will occur.

How can I solve this?

Thank you!
Regards,
Uwe

Hallo Uwe,

generally all umlauts must be encoded (as UTF-8 in your case).

So, if your search string value is from the Search textbox, you have
to encode it as

string search = HttpUtility.UrlEncode(SearchTextBox.Text,
Encoding.UTF8));
Response.Redirect("default.aspx?search=" + search);

this would redirect you to default.aspx?search=m%c3%bcstring

In this case you would not have any problem to get the value using
HttpUtility.UrlDecode() function...

If you want to get umlauts out of the "default.aspx?search=müstring"
use RawUrl:

string search = Request.RawUrl.Split('=')[1];

and then search would return you "müstring"...

To handle both requests

default.aspx?search=m%c3%bcstring
default.aspx?search=müstring

use

string search = Request.RawUrl.Split('=')[1];
string newsearch = HttpUtility.UrlDecode(search);

[Alexey Smirnov]

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

http://www.google.com/search?hl=en&q=m%25ufffdstring

[Uwe Braunholz]

Hello Alexey,

the "m%ufffdstring" appears, if I hover the variable in my Visual
Studio 2005 with my mouse.

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

http://www.google.com/search?hl=en&q=m%25ufffdstring

[Uwe Braunholz]

Ok, I think the problem is about that:
http://www.captain.at/howto-php-urle...IComponent.php

Assuming, I will never be able to pass a regular ü to the server by
simply put it into the url:
The Server.UrlEncode converts ü to %C3%BC
But my browsers convert ü to %FC
Why is this? How do I know what I will get on the serverside?

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

http://www.google.com/search?hl=en&q=m%25ufffdstring

[Alexey Smirnov]

On Aug 12, 8:03*am, Uwe Braunholz <[e-mail]:

Ok, I think the problem is about that:http://www.captain.at/howto-php-urle...codeURICompone...

Assuming, I will never be able to pass a regular ü to the server by
simply put it into the url:
The Server.UrlEncode converts ü to %C3%BC
But my browsers convert ü to %FC
Why is this? How do I know what I will get on the serverside?

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

http://www.google.com/search?hl=en&q=m%25ufffdstring- Hide quoted text -

- Show quoted text -

Uwe,

you used Firefox 2 to get the "ü" converted into "%FC", right? While
FF2 sends it by default in ISO-8859-1, FF3 and IE send it in UTF-8 and
that means that you cannot get "ü" working in FF2 only. But in FF2 you
can change this behavior too. Go to about:config, and set
"network.standard-url.encode-utf8" parameter to true.

If you want to make umlauts working in all browsers I guess you would
need to check if letter has 1 or 2 bytes encoded. But the easiest way
is probably to use a function as the one below:

string convertLetters(string s)
{
string[] oldchars = new string[] {"%FC","%C3%BC", ...};
string[] newchars = new string[] {"ü","ü", "ä", "ö"...};

for (i==0;i<oldchars.length();i++)
{
if (s.indexOf(oldchars[i])>-1)
{
s = s.replace(oldchars[i],newchars[i]);
}
}

return s;
}

[Uwe Braunholz]

Hello Alexey,

thanks for keeping up on this!
I am using FF3, but the settings was false, so I assume the behavior
did not change. As I switched to true, it worked.
On my testmachine I use IE6 as well, and this one does not seem to use
UTF8 also, because the result is the same.

How could I check if the string is encoded with one or two bytes?

Regards,
uwe

On 12 Aug., 09:34, Alexey Smirnov <[e-mail]:

On Aug 12, 8:03*am, Uwe Braunholz <[e-mail]:

Ok, I think the problem is about that:http://www.captain.at/howto-php-urle...codeURICompone...

Assuming, I will never be able to pass a regular ü to the server by
simply put it into the url:
The Server.UrlEncode converts ü to %C3%BC
But my browsers convert ü to %FC
Why is this? How do I know what I will get on the serverside?

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

>http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

>http://www.google.com/search?hl=en&q...ffdstring-Hide quoted text -

- Show quoted text -

Uwe,

you used Firefox 2 to get the "ü" converted into "%FC", right? While
FF2 sends it by default in ISO-8859-1, FF3 and IE send it in UTF-8 and
that means that you cannot get "ü" working in FF2 only. But in FF2 you
can change this behavior too. Go to about:config, and set
"network.standard-url.encode-utf8" parameter to true.

If you want to make umlauts working in all browsers I guess you would
need to check if letter has 1 or 2 bytes encoded. But the easiest way
is probably to use a function as the one below:

string convertLetters(string s)
{
string[] oldchars = new string[] {"%FC","%C3%BC", ...};
string[] newchars = new string[] {"ü","ü", "ä", "ö"...};

for (i==0;i<oldchars.length();i++)
{
if (s.indexOf(oldchars[i])>-1)
{
s = s.replace(oldchars[i],newchars[i]);

}
}

return s;

}

[Alexey Smirnov]

On Aug 12, 10:41*am, Uwe Braunholz <[e-mail]:

Hello Alexey,

thanks for keeping up on this!
I am using FF3, but the settings was false, so I assume the behavior
did not change. As I switched to true, it worked.
On my testmachine I use IE6 as well, and this one does not seem to use
UTF8 also, because the result is the same.

How could I check if the string is encoded with one or two bytes?

Regards,
uwe

On 12 Aug., 09:34, Alexey Smirnov <[e-mail]:

On Aug 12, 8:03*am, Uwe Braunholz <[e-mail]:

Ok, I think the problem is about that:http://www.captain.at/howto-php-urle...codeURICompone...

Assuming, I will never be able to pass a regular ü to the server by
simply put it into the url:
The Server.UrlEncode converts ü to %C3%BC
But my browsers convert ü to %FC
Why is this? How do I know what I will get on the serverside?

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

http://www.google.com/search?hl=en&q...ing-Hidequoted text -

- Show quoted text -

Uwe,

you used Firefox 2 to get the "ü" converted into "%FC", right? While
FF2 sends it by default in ISO-8859-1, FF3 and IE send it in UTF-8 and
that means that you cannot get "ü" working in FF2 only. But in FF2 you
can change this behavior too. Go to about:config, and set
"network.standard-url.encode-utf8" parameter to true.

If you want to make umlauts working in all browsers I guess you would
need to check if letter has 1 or 2 bytes encoded. But the easiest way
is probably to use a function as the one below:

string convertLetters(string s)
{
string[] oldchars = new string[] {"%FC","%C3%BC", ...};
string[] newchars = new string[] {"ü","ü", "ä", "ö"...};

for (i==0;i<oldchars.length();i++)
{
if (s.indexOf(oldchars[i])>-1)
{
s = s.replace(oldchars[i],newchars[i]);

}
}

return s;

}- Hide quoted text -

- Show quoted text -

Well, I think you would need to check if querystring has %C3%XX (UTF
encoded), or just %XX was presented.

Sample code:

string s;

s = "m%C3%BCstring";
s = "m%FCstring";

if (s.IndexOf("%C3", StringComparison.InvariantCultureIgnoreCase) >
-1)
s = HttpUtility.UrlDecode(s, Encoding.UTF8);
else
s = HttpUtility.UrlDecode(s, Encoding.GetEncoding("ISO-8859-1"));

Hope this helps,

Tch%C3%BCss :-)

[Uwe Braunholz]

Hello Alexey,

thank you once more!
Unfortunately, the UrlDecode with ISO-8859-1 does not give any good
sign, only the small "char not displayable box".
I wonder if this is related to the debugoutput:
Request.QueryString = {search=m%ufffdstring} instead of something like
m%FCstring shown in the browser addressbar.

As it seems like I am the only one on the web experiencing this, I
leave it there. I have no idea why such a problem can exist at all.

Regards,
uwe
On 13 Aug., 21:50, Alexey Smirnov <[e-mail]:

On Aug 12, 10:41*am, Uwe Braunholz <[e-mail]:

Hello Alexey,

thanks for keeping up on this!
I am using FF3, but the settings was false, so I assume the behavior
did not change. As I switched to true, it worked.
On my testmachine I use IE6 as well, and this one does not seem to use
UTF8 also, because the result is the same.

How could I check if the string is encoded with one or two bytes?

Regards,
uwe

On 12 Aug., 09:34, Alexey Smirnov <[e-mail]:

On Aug 12, 8:03*am, Uwe Braunholz <[e-mail]:

Ok, I think the problem is about that:http://www.captain.at/howto-php-urle...codeURICompone...

Assuming, I will never be able to pass a regular ü to the server by
simply put it into the url:
The Server.UrlEncode converts ü to %C3%BC
But my browsers convert ü to %FC
Why is this? How do I know what I will get on the serverside?

Regards,
Uwe

On 11 Aug., 17:05, Alexey Smirnov <[e-mail]:

On Aug 11, 3:10*pm, Uwe Braunholz <[e-mail]:

Hello Alexey,

thank you for your reply. Unfortunately even the RawUrl does not give
me the real (intended) URL-string. There still is a ? instead of the
"ü".
Debugging says
Request.QueryString = {search=m%ufffdstring}
Request.RawUrl = "/WebSite2/Default.aspx?search=m string"

Well, but where did you get the "m%ufffdstring"?

%ufff is wrong there... you can simply check it by typing "müstring"
in google. When UTF8 is used you will see that url is encoded as "m
%C3%BCstring"

>http://www.google.com/search?hl=en&q=m%C3%BCstring

and not

>http://www.google.com/search?hl=en&q...Hidequotedtext -

- Show quoted text -

Uwe,

you used Firefox 2 to get the "ü" converted into "%FC", right? While
FF2 sends it by default in ISO-8859-1, FF3 and IE send it in UTF-8 and
that means that you cannot get "ü" working in FF2 only. But in FF2 you
can change this behavior too. Go to about:config, and set
"network.standard-url.encode-utf8" parameter to true.

If you want to make umlauts working in all browsers I guess you would
need to check if letter has 1 or 2 bytes encoded. But the easiest way
is probably to use a function as the one below:

string convertLetters(string s)
{
string[] oldchars = new string[] {"%FC","%C3%BC", ...};
string[] newchars = new string[] {"ü","ü", "ä", "ö"...};

for (i==0;i<oldchars.length();i++)
{
if (s.indexOf(oldchars[i])>-1)
{
s = s.replace(oldchars[i],newchars[i]);

}
}

return s;

}- Hide quoted text -

- Show quoted text -

Well, I think you would need to check if querystring has %C3%XX (UTF
encoded), or just %XX was presented.

Sample code:

string s;

s = "m%C3%BCstring";
s = "m%FCstring";

if (s.IndexOf("%C3", StringComparison.InvariantCultureIgnoreCase) >
-1)
s = HttpUtility.UrlDecode(s, Encoding.UTF8);
else
s = HttpUtility.UrlDecode(s, Encoding.GetEncoding("ISO-8859-1"));

Hope this helps,

Tch%C3%BCss :-)

[Alexey Smirnov]

On Aug 15, 4:24*pm, Uwe Braunholz <[e-mail]:

Unfortunately, the UrlDecode with ISO-8859-1 does not give any good
sign, only the small "char not displayable box".

because [ü] (in lower case) in ISO-8859-1 equals to [%fc] and not
[%ufffd]
http://en.wikipedia.org/wiki/ISO-8859-1

Actually, I still don't get where [m%ufffdstring] comes from

You said:

------------------------------
I notice that firefox converts the URL automatically to:
?search=m%FCstring
using the debugger results in:
Request.QueryString = {search=m%ufffdstring}
------------------------------

Why do you get Request.QueryString very different from the url?

Request.QueryString["search"] should be equal to "m%FCstring"