Get document as normal text and not as binary data

[Markus Franz]

Hi.

I used urllib2 to load a html-document through http. But my problem
is:
The loaded contents are returned as binary data, that means that every
character is displayed like lÀÃt, for example. How can I get the
contents as normal text?

My script was:

import urllib2
req = urllib2.Request (url)
f = urllib2.urlopen (req)
contents = f.read()
print contents
f.close()

Thanks!

Markus

[Diez B. Roggisch]

Markus Franz wrote:

Hi.

I used urllib2 to load a html-document through http. But my problem
is:
The loaded contents are returned as binary data, that means that every
character is displayed like lÀÃt, for example. How can I get the
contents as normal text?

You get what the server sends. That is always binary - either it _is_ a
binary file, or maybe in an unknown encoding.

--
Regards,

Diez B. Roggisch

[Fredrik Lundh]

Markus Franz wrote:

I used urllib2 to load a html-document through http. But my problem
is: The loaded contents are returned as binary data, that means that every
character is displayed like lÃ?Ãt, for example. How can I get the
contents as normal text?

My script was:

import urllib2
req = urllib2.Request (url)
f = urllib2.urlopen (req)
adding

print f.headers

and checking the header fields (especially the content-type) may help you
figure out what's going on...
contents = f.read()
print contents
f.close()

</F>

[Markus Franz]

Diez B. Roggisch wrote:

You get what the server sends. That is always binary - either it _is_ a
binary file, or maybe in an unknown encoding.

And how can I convert those binary data to a "normal" string with
"normal" characters?

Best regards

Markus

[Diez B. Roggisch]

Markus Franz wrote:

Diez B. Roggisch wrote:

You get what the server sends. That is always binary - either it _is_ a
binary file, or maybe in an unknown encoding.

And how can I convert those binary data to a "normal" string with
"normal" characters?

There is no "normal" - it's just bytes, and a string is just bytes. No
difference, no translation necessary.

As others have said: look into the http header what the server is trying to
transmit - maybe an image. The mimetype header is telling you that.

Or use wget to fetch the url and look what you get - it shouldn't look
different.
--
Regards,

Diez B. Roggisch

[Diez B. Roggisch]

Addendum: If you give us the url you're fetching data from, we might be able
to look at the delivered data ourselves.
--
Regards,

Diez B. Roggisch

[Kent Johnson]

Markus Franz wrote:

Hi.

I used urllib2 to load a html-document through http. But my problem
is:
The loaded contents are returned as binary data, that means that every
character is displayed like lÀÃt, for example. How can I get the
contents as normal text?
My guess is the html is utf-8 encoded - your sample looks like utf-8-interpreted-as-latin-1. Try
contents = f.read().decode ('utf-8')

Kent

My script was:

import urllib2
req = urllib2.Request (url)
f = urllib2.urlopen (req)
contents = f.read()
print contents
f.close()

Thanks!

Markus

[Markus Franz]

Kent Johnson wrote:

My guess is the html is utf-8 encoded - your sample looks like
utf-8-interpreted-as-latin-1. Try
contents = f.read().decode ('utf-8')

YES! That helped!

I used the following:

....
contents = f.read().decode ('utf-8')
contents = contents.encode ('iso-8859-15')
....

That was the perfect solution for my problem! Thanks a lot!

Best regards

Markus

[Markus Franz]

Diez B. Roggisch wrote:

Addendum: If you give us the url you're fetching data from, we might be able
to look at the delivered data ourselves.

To guess my problem please have a look at the document title of
<http://portal.suse.de/sdb/de/1997/01/xntp.html>

Markus