467,915 Members | 1,146 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 467,915 developers. It's quick & easy.

Unicode Question

Hi. I am working through some tutorials on unicode and am hoping that
someone can help explain this for me. I am on mac platform using python
2.4.1 at the moment. I am experimenting with unicode with the 3/4 symbol.

I want to prepare strings for db storage that come from normal Windows
machine (cp1252) so my understanding is to unicode and encode to utf-8
and to store properly. Since data will be used on the web I would not
have to change my encoding when extracting from the database. This first
example I believe simulates this with the 3/4 symbol. Here I want to
store '\xc2\xbe' in my database.
tq = u'\xbe'
tq_utf = tq.encode('utf8')
tq, tq_utf (u'\xbe', '\xc2\xbe')

To unicode withat a valiable, my understanding is that I can unicode and
encode at the same time
tq = '\xbe'
tq_utf = unicode(tq, 'utf-8')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 0:
unexpected code byte

This is not working for me. Can someone explain why. Many thanks.

Regards,
David
Jan 10 '06 #1
  • viewed: 2135
Share:
4 Replies
David Pratt wrote:
This is not working for me. Can someone explain why. Many thanks.


Because '\xbe' isn't UTF-8 for the character you want, '\xc2\xbe' is, as
you just showed yourself in the code snippet.

--
Erik Max Francis && ma*@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Where are they?
-- Enrico Fermi, 1901-1954
Jan 10 '06 #2
David Pratt wrote:
I want to prepare strings for db storage that come from normal Windows
machine (cp1252) so my understanding is to unicode and encode to utf-8
and to store properly.
That also depends on the database. The database must accept
UTF-8-encoded strings, and must not modify them in any form or way.
Some databases fail here, and work better if you pass Unicode objects
to them directly.
Since data will be used on the web I would not
have to change my encoding when extracting from the database. This first
example I believe simulates this with the 3/4 symbol. Here I want tox
store '\xc2\xbe' in my database.
tq = u'\xbe'
You can verify that this is really 3/4:

py> import unicodedata
py> unicodedata.name(u"\xbe")
'VULGAR FRACTION THREE QUARTERS'
tq_utf = tq.encode('utf8')
tq, tq_utf (u'\xbe', '\xc2\xbe')
So it should be clear now that '\xc2\xbe' is the UTF-8 encoding
of that character.
To unicode withat a valiable, my understanding is that I can unicode and
encode at the same time


Not sure what you mean by "same time" (I'm not even sure what
"I can unicode" means - unicode is not a verb, it's a noun).
tq = '\xbe'
tq_utf = unicode(tq, 'utf-8')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 0:
unexpected code byte

This is not working for me. Can someone explain why. Many thanks.


Of course not. The UTF-8 encoding of the character, as we have seen
earlier, is '\xc2\xbe'. So you should write

py> unicode('\xc2\xbe', 'utf-8')
u'\xbe'

You mentioned windows-1252 at some point. If you are given windows-1252
bytes, you can do

py> unicode('\xbe', 'windows-1252')
u'\xbe'

If you are looking for "at the same time", perhaps this is also
interesting:

py> unicode('\xbe', 'windows-1252').encode('utf-8')
'\xc2\xbe'

Regards,
Martin
Jan 10 '06 #3
Hi Martin. Many thanks for your reply. What I am reall after, the
following accomplishes.

If you are looking for "at the same time", perhaps this is also
interesting:

py> unicode('\xbe', 'windows-1252').encode('utf-8')
'\xc2\xbe'


Your answer really helped quite a bit to clarify this for me. I am using
sqlite3 so it is very happy to have utf-8 encoded unicode.

The examples you provided were the additional help I needed. Thank you.

Regards,
David
Jan 10 '06 #4
Hi Erik. Thank you for your reply. The advice I has helped clarify this
for me.

Regards,
David

Erik Max Francis wrote:
David Pratt wrote:

This is not working for me. Can someone explain why. Many thanks.

Because '\xbe' isn't UTF-8 for the character you want, '\xc2\xbe' is, as
you just showed yourself in the code snippet.

Jan 10 '06 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

8 posts views Thread by sebastien.hugues | last post: by
9 posts views Thread by FranÁois Pinard | last post: by
27 posts views Thread by EU citizen | last post: by
3 posts views Thread by dalei | last post: by
12 posts views Thread by damjan | last post: by
14 posts views Thread by abhi147 | last post: by
2 posts views Thread by willie | last post: by
5 posts views Thread by =?Utf-8?B?S2V2aW4gVGFuZw==?= | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.