472,779 Members | 1,987 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,779 software developers and data experts.

inserting Unicode character in dictionary - Python

Hello All,

The following code does not work for unicode characters:

keyword = dict()
kw = ''
keyword.setdefault(key, []).append (kw)

It works fine for inserting ASCII character. Any suggestion?

Thanks,

Gita
Oct 17 '08 #1
6 3811
On Fri, 17 Oct 2008 13:07:38 -0400, gita ziabari wrote:
The following code does not work for unicode characters:

keyword = dict()
kw = 'генских'
keyword.setdefault(key, []).append (kw)

It works fine for inserting ASCII character. Any suggestion?
What do you mean by "does not work"? And you are aware that the above
snipped doesn't involve any unicode characters!? You have a byte string
there -- type `str` not `unicode`.

Ciao,
Marc 'BlackJack' Rintsch
Oct 17 '08 #2
On Oct 17, 2008, at 11:24 AM, Marc 'BlackJack' Rintsch wrote:
>kw = 'генских'
What do you mean by "does not work"? And you are aware that the above
snipped doesn't involve any unicode characters!? You have a byte
string
there -- type `str` not `unicode`.
Just checking my understanding here -- are the following all true:

1. If you had prefixed that literal with a "u", then you'd have Unicode.

2. Exactly what Unicode you get would be dependent on Python properly
interpreting the bytes in the source file -- which you can make it do
by adding something like "-*- coding: utf-8 -*-" in a comment at the
top of the file.

3. Without the "u" prefix, you'll have some 8-bit string, whose
interpretation is... er... here's where I get a bit fuzzy. What if
your source file is set to utf-8? Do you then have a proper UTF-8
string, but the problem is that none of the standard Python library
methods know how to properly interpret UTF-8?

4. In Python 3.0, this silliness goes away, because all strings are
Unicode by default.

Thanks for any answers/corrections,
- Joe


Oct 17 '08 #3
On Fri, 17 Oct 2008 11:32:36 -0600, Joe Strout wrote:
On Oct 17, 2008, at 11:24 AM, Marc 'BlackJack' Rintsch wrote:
>>kw = 'генских'
What do you mean by "does not work"? And you are aware that the above
snipped doesn't involve any unicode characters!? You have a byte
string there -- type `str` not `unicode`.

Just checking my understanding here -- are the following all true:

1. If you had prefixed that literal with a "u", then you'd have Unicode.
Yes.
2. Exactly what Unicode you get would be dependent on Python properly
interpreting the bytes in the source file -- which you can make it do by
adding something like "-*- coding: utf-8 -*-" in a comment at the top of
the file.
Yes, assuming the encoding on that comment matches the actual encoding of
the file.
3. Without the "u" prefix, you'll have some 8-bit string, whose
interpretation is... er... here's where I get a bit fuzzy.
No interpretation at all, just the bunch of bytes that happen to be in
the source file.
What if your source file is set to utf-8? Do you then have a proper
UTF-8 string, but the problem is that none of the standard Python
library methods know how to properly interpret UTF-8?
Well, the decode method knows how to decode that bytes into a `unicode`
object if you call it with 'utf-8' as argument.
4. In Python 3.0, this silliness goes away, because all strings are
Unicode by default.
Yes and no. The problem just shifts because at some point you get into
similar troubles, just in the other direction. Data enters the program
as bytes and must leave it as bytes again, so you have to deal with
encodings at those points.

Ciao,
Marc 'BlackJack' Rintsch
Oct 17 '08 #4
Thanks for the answers. That clears things up quite a bit.
>What if your source file is set to utf-8? Do you then have a proper
UTF-8 string, but the problem is that none of the standard Python
library methods know how to properly interpret UTF-8?

Well, the decode method knows how to decode that bytes into a
`unicode`
object if you call it with 'utf-8' as argument.
OK, good to know.
>4. In Python 3.0, this silliness goes away, because all strings are
Unicode by default.

Yes and no. The problem just shifts because at some point you get
into
similar troubles, just in the other direction. Data enters the
program
as bytes and must leave it as bytes again, so you have to deal with
encodings at those points.
Yes, but that's still much better than having to litter your code with
'u' prefixes and .decode calls and so on. If I'm using a UTF-8-savvy
text editor (as we all should be doing in the 21st century!), and type
"foo = '2π'", I should get a string containing a '2' and a pi
character, and all the text operations (like counting characters,
etc.) should Just Work.

When I read and write files or sockets or whatever, of course I'll
have to think about what encoding the text should be... but internal
to my own source code, I shouldn't have to.

I understand the need for a transition strategy, which is what we have
in 2.x, and that's working well enough. But I'll be glad when it's
over. :)

Cheers,
- Joe
Oct 17 '08 #5
2. Exactly what Unicode you get would be dependent on Python properly
interpreting the bytes in the source file -- which you can make it do by
adding something like "-*- coding: utf-8 -*-" in a comment at the top of
the file.
That depends on the Python version. Up to (and including) 2.4, the bytes
on the disk where interpreted as Latin-1 in absence of an encoding
declaration. In 2.5, not having an encoding declaration is an error. In
3.x, in absence of an encoding declaration, the bytes are interpreted as
UTF-8 (giving an error when ill-formed UTF-8 sequences are encountered).
3. Without the "u" prefix, you'll have some 8-bit string, whose
interpretation is... er... here's where I get a bit fuzzy. What if your
source file is set to utf-8?
You need to distinguish between the declared encoding, and the intended
(editor) encoding also. Some editors (like Emacs or IDLE) interpret the
declaration, others may not. What you see on the display is the editor's
interpretation; what Python uses is the declared encoding.

However, Python uses the declared encoding just for Unicode strings.
Do you then have a proper UTF-8 string,
but the problem is that none of the standard Python library methods know
how to properly interpret UTF-8?
There is (probably) no such thing as a "proper UTF-8 string" (in the
sense in which you probably mean it). Python doesn't have a data type
for "UTF-8 string". It only has a data type "byte string". It's up to
the application whether it gets interpreted in a consistent manner.
Libraries are (typically) encoding-agnostic, i.e. they work for UTF-8
encoded strings the same way as for, say, Big-5 encoded strings.
4. In Python 3.0, this silliness goes away, because all strings are
Unicode by default.
You still need to make sure that the editor's encoding and the declared
encoding match.

Regards,
Martin
Oct 18 '08 #6
On Oct 17, 2:38 pm, Joe Strout <j...@strout.netwrote:
Thanks for the answers. That clears things up quite a bit.
What if your source file is set to utf-8? Do you then have a proper
UTF-8 string, but the problem is that none of the standard Python
library methods know how to properly interpret UTF-8?
Well, the decode method knows how to decode that bytes into a
`unicode`
object if you call it with 'utf-8' as argument.

OK, good to know.
4. In Python 3.0, this silliness goes away, because all strings are
Unicode by default.
Yes and no. The problem just shifts because at some point you get
into
similar troubles, just in the other direction. Data enters the
program
as bytes and must leave it as bytes again, so you have to deal with
encodings at those points.

Yes, but that's still much better than having to litter your code with
'u' prefixes and .decode calls and so on. If I'm using a UTF-8-savvy
text editor (as we all should be doing in the 21st century!), and type
"foo = '2'", I should get a string containing a '2' and a pi
character, and all the text operations (like counting characters,
etc.) should Just Work.

When I read and write files or sockets or whatever, of course I'll
have to think about what encoding the text should be... but internal
to my own source code, I shouldn't have to.

I understand the need for a transition strategy, which is what we have
in 2.x, and that's working well enough. But I'll be glad when it's
over. :)

Cheers,
- Joe
Thanks for the answers. The following factors should be considerd when
dealing with unicode characters in python:
1. Declaring # -*- coding: utf-8 -*- at the top of script
2. Opening files with appropriate encoding:
txt = codecs.open (filename, 'w+', encoding='utf-8')

My program works fine now. There is no specific way of adding unicode
characters in list or dictionaies. The character itself has to be in
unicode.

Cheers,

Gita
Oct 19 '08 #7

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: sebastien.hugues | last post by:
Hi I would like to retrieve the application data directory path of the logged user on windows XP. To achieve this goal i use the environment variable APPDATA. The logged user has this name:...
23
by: Hallvard B Furuseth | last post by:
Has someone got a Python routine or module which converts Unicode strings to lowercase (or uppercase)? What I actually need to do is to compare a number of strings in a case-insensitive manner,...
3
by: Antioch | last post by:
Ok, so Im a newb python programmer and I'm trying to create a simple python web-application. The program is simply going to read in pairs of words, parse them into a dictionary file, then randomly...
4
by: Tony Nelson | last post by:
Is there a faster way to decode from charmaps to utf-8 than unicode()? I'm writing a small card-file program. As a test, I use a 53 MB MBox file, in mac-roman encoding. My program reads and...
12
by: manstey | last post by:
I am writing a program to translate a list of ascii letters into a different language that requires unicode encoding. This is what I have done so far: 1. I have # -*- coding: UTF-8 -*- as my...
2
by: Frantic | last post by:
I'm working on a list of japaneese entities that contain the entity, the unicode hexadecimal code and the xml/sgml entity used for that entity. A unicode document is read into the program, then the...
5
by: Nicolas Pontoizeau | last post by:
Hi, I am handling a mixed languages text file encoded in UTF-8. Theres is mainly French, English and Asian languages. I need to detect every asian characters in order to enclose it by a special...
7
by: 7stud | last post by:
Based on this example and the error: ----- u_str = u"abc\u9999" print u_str UnicodeEncodeError: 'ascii' codec can't encode character u'\u9999' in position 3: ordinal not in range(128) ------
0
by: Terry Reedy | last post by:
Peter Bulychev wrote: I believe you will have to make up your own translation dictionary for the translations *you* want. You should then be able to use that with the .translate() method. tjr
0
by: Rina0 | last post by:
Cybersecurity engineering is a specialized field that focuses on the design, development, and implementation of systems, processes, and technologies that protect against cyber threats and...
3
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 2 August 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
0
by: erikbower65 | last post by:
Using CodiumAI's pr-agent is simple and powerful. Follow these steps: 1. Install CodiumAI CLI: Ensure Node.js is installed, then run 'npm install -g codiumai' in the terminal. 2. Connect to...
0
by: erikbower65 | last post by:
Here's a concise step-by-step guide for manually installing IntelliJ IDEA: 1. Download: Visit the official JetBrains website and download the IntelliJ IDEA Community or Ultimate edition based on...
0
by: Taofi | last post by:
I try to insert a new record but the error message says the number of query names and destination fields are not the same This are my field names ID, Budgeted, Actual, Status and Differences ...
14
DJRhino1175
by: DJRhino1175 | last post by:
When I run this code I get an error, its Run-time error# 424 Object required...This is my first attempt at doing something like this. I test the entire code and it worked until I added this - If...
0
by: Rina0 | last post by:
I am looking for a Python code to find the longest common subsequence of two strings. I found this blog post that describes the length of longest common subsequence problem and provides a solution in...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
2
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.