Is this HttpWebRequest correct?

[Nightcrawler]

I am currently using the HttpWebRequest and HttpWebResponse to pull
webpages down from a few urls.

string url = "some url";
HttpWebRequest httpWebRequest =
(HttpWebRequest )WebRequest.Cre ate(url);

using (HttpWebRespons e httpWebResponse =
(HttpWebRespons e)httpWebReques t.GetResponse() )
{
string html = string.Empty;

StreamReader responseReader = new
StreamReader(ht tpWebResponse.G etResponseStrea m(), Encoding.UTF7);
html = responseReader. ReadToEnd();
}

My code works but my question is, am I doing it the right way
(especially the encoding part)? Some of the websites I pull content
from have charachters in them that do not exist in the english
alphabet and currently the only way for these to be read correctly by
my streamreader is if I am using UTF7 encoding. Is this really the
only way?

Before I move forward in the project I would like to understand if
this indeed is the way to do this or if I am missing anything?

Any help is appreciated.

Thanks

[Martin Honnen]

Nightcrawler wrote:

I am currently using the HttpWebRequest and HttpWebResponse to pull
webpages down from a few urls.

string url = "some url";
HttpWebRequest httpWebRequest =
(HttpWebRequest )WebRequest.Cre ate(url);

using (HttpWebRespons e httpWebResponse =
(HttpWebRespons e)httpWebReques t.GetResponse() )
{
string html = string.Empty;

StreamReader responseReader = new
StreamReader(ht tpWebResponse.G etResponseStrea m(), Encoding.UTF7);
html = responseReader. ReadToEnd();
}

My code works but my question is, am I doing it the right way
(especially the encoding part)? Some of the websites I pull content
from have charachters in them that do not exist in the english
alphabet and currently the only way for these to be read correctly by
my streamreader is if I am using UTF7 encoding. Is this really the
only way?

You should check the HTTP response header Content-Type for a charset
parameter and use that to create the stream reader. So for instance if
the server sends a header
Content-Type: text/html; charset=Windows-1252
then you would use
new StreamReader(ht tpWebResponse.G etResponseStrea m(),
Encoding.GetEnc oding("Windows-1252"))

On the other hand on the wild wild web the server often does not send a
charset parameter and the author of the HTML document only includes the
charset in a meta element e.g.
<meta http-equiv="Content-Type" content="text/html;
charset=Windows-1252">
Therefore user agents like browsers put in a lot of effort to try to
read enough of the document to find and parse that meta element to then
be able to decode the rest of the document.
--

Martin Honnen --- MVP XML
http://JavaScript.FAQTs.com/

[Nightcrawler]

So what you basically are saying is that my best bet is to look for
the meta tags in the page to determine the encoding to use and don't
rely on the HTTP response header.

Most of the sites I read using the streamreader say: <meta http-
equiv="Content-type" content="text/html; charset=UTF-8" /but there
are a few that do not have that meta tag included in their code. How
should I approach those? Is there a way for the streamreader to detect
what encoding the page is using?

Thanks for you help!

[Nightcrawler]

What is even more annoying is that one of the websites I read is
stating it's using UTF-8 and my streamreader still does not translate
the charachters correctly. I get little square boxes instead of the
charachters.

[Peter Duniho]

On Fri, 03 Oct 2008 10:28:21 -0700, Nightcrawler
<th************ @gmail.comwrote :

What is even more annoying is that one of the websites I read is
stating it's using UTF-8 and my streamreader still does not translate
the charachters correctly. I get little square boxes instead of the
charachters.

"little square boxes" might, but does not necessarily, mean that the
characters are being decoded incorrectly. It may simply be that the
characters are not displaying with whatever font you're using to show them.

How are you determining that the StreamReader doesn't correctly decode the
characters? How are you specifying, if at all, that the encoding used by
the StreamReader is UTF-8?

Pete

[Nightcrawler]

If I view the very same page in my browser it shows up correctly.

The meta tag states it's using UTF-8 but when I use:

StreamReader responseReader = new
StreamReader(ht tpWebResponse.G etResponseStrea m(), Encoding.UTF8);

The charachters are still unreadable. However, if I use UTF7 instead
the charachters show up correctly BUT, when I try to convert the page
to XML I get an error saying "hexadecima l value 0xD85E, is an invalid
character". I am very confused with all this. Seems a little like the
wild wild west.

Any further help is highly appreciated.

Thanks

[Nightcrawler]

I guess another interesting point is that when I change the code to
use: "ISO-8859-1" instead of UTF-8 like the website claims it uses, it
seems that it actuallly is reading the charachters correctly AND the
string translates into XML without any issues. Why? I have no idea and
I wish I understood it better. Again, any insight to this problem is
appreciated.

Thanks

[Peter Duniho]

On Fri, 03 Oct 2008 10:43:19 -0700, Nightcrawler
<th************ @gmail.comwrote :

If I view the very same page in my browser it shows up correctly.

Unless your own code is using the same fonts to display the text that the
browser uses, that's not a relevant test.

As for the other behaviors you've noticed, it does sound to me as though
it's possible that the page is not encoded in UTF-8, but rather
ISO-8859-1. But it's hard to know for sure, since we don't have the
actual data to look at.

Pete

[Nightcrawler]

Pete,

You can see the page if you go to the link below. It's iTunes
linkmaker page:

http://ax.phobos.apple.com.edgesuite...ss&media=music

As you can see they claim they use utf-8 but when you read it using a
streamreader with that encoding, it does not read "foreign"
charachters correctly. However, when I tried the ISO-8859-1 encoding
it seemed to work.

Thanks

[Peter Duniho]

On Fri, 03 Oct 2008 13:47:43 -0700, Nightcrawler
<th************ @gmail.comwrote :

Pete,

You can see the page if you go to the link below. It's iTunes
linkmaker page:

http://ax.phobos.apple.com.edgesuite...ss&media=music

As you can see they claim they use utf-8 but when you read it using a
streamreader with that encoding, it does not read "foreign"
charachters correctly. However, when I tried the ISO-8859-1 encoding
it seemed to work.

What data in the page are you having trouble with? Can you be more
specific about what's not being shown correctly?

I haven't spend a huge amount of time with the file. But a cursory look
at it shows that it appears, at least to me, to have ISO-8859-1 data
embedded within the page itself, in certain URLs.

It seems possible to me that the page encoding is technically UTF-8, but
using only the subset of UTF-8 that is the same as ISO-8859-1, and that
the page also has data that's not supposed to be interpreted as text
within the HTML, but rather should be decoded as ISO-8859-1.

That would explain why the page claims to be encoded as UTF-8 but there
are still characters that don't display correctly unless you read the HTML
as ISO-8859-1.

Or maybe the meta tag really is wrong. I'm not completely sure. :)

Pete