from WWW through urllib2 to MySQL --> encoding fuss

[przemeq]

Hi all,
I've spent last few hours trying to figure out the following:

1. fetch a website using urllib2
2. filter out some of its content using SGMLParser
3. decode result from p. 2. using charset as set in <META http-equiv="Content-Type" ...> of the website
4. encode the whole thing using UTF-8
5. stuff it into MySQL

Here it dies with the following:

 
 File "c:\python24\lib\site-packages\MySQLdb\cursors.py", line 147, in execute
 query = query.encode(charset)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position 8584: ordinal not in range (128)
 

First of all, I have no idea why it's trying to use 'ascii' codec in the first place, as charset in MySQL is set to 'latin1': it's fetched properly by MySQLdb/cursors.py - that's checked.
Secondly, there seems to be little change no matter if I do any decode/encode or not in points 3 and 4 above. It just won't go.
Right now I'm on the verge of claiming that Python's decode() doesn't do its job. As an example:

websiteContent.decode('iso-8859-1').encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character u'\x97' in position 8033: ordinal not in range(128)
 

won't go, but when I'm encode()ing using 'utf8' it works, failing only as described above, when I put the result into SQL query.

Any hints on how to do the 1-2-3-4-5 sequence above will be appreciated.
PTB

[zbabinga]

Not sure what the etiquette is on replying to a month-old topic...

1. If you do a s.encode('latin-1') and s is not a unicode object, python will first try to convert the string to unicode using the default encoding, which is ascii. That's why you got the first error. To resolve this, first decode the query you have using the unicode() or decode() functions.

2. When you try to encode into a specific charset, but the original string contains characters which aren't supported by this charset, you're going to get an exception. You can instruct python to ignore errors by using u.encode('ascii', 'ignore')

>>> s = "hello\x97"
 
>>> s.encode('latin-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position 5: ordinal not in range(128)
 
>>> s.decode('iso-8859-1')
u'hello\x97'
 
>>> s.decode('iso-8859-1').encode('latin-1')
'hello\x97'
 
>>> s.decode('iso-8859-1').encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\x97' in position 5: ordinal not in range(128)
 
>>> s.decode('iso-8859-1').encode('ascii', 'ignore')
'hello'
 

[bartonc]

Not sure what the etiquette is on replying to a month-old topic...

Welcome to TheScripts. Thanks for posting. I say "if it's not a closed thread, post away!". This is very useful info for all to see and we appreciate your input.
Drop in anytime,
Barton (python forum moderator)