Novice: replacing strings with unicode variables in a list

[aine_canby]

Hi,

Im totally new to Python so please bare with me.

Data is entered into my program using the folling code -

str = raw_input(command)
words = str.split()

for word in words:
word = unicode(word,'latin-1')
word.encode('utf8')

This gives an error:

File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x94' in
position 0
: character maps to <undefined>

but the following works.

str = raw_input(command)
words = str.split()

for word in words:
uni = u""
uni = unicode(word,'latin-1')
uni.encode('utf8')

so the problem is that I want replace my list with unicode variables.
Or maybe I should create a new list.

I also tried this:

for word in words[:]:
word = u""
word = unicode(word,'latin-1')
word.encode('utf8')
print word

but got TypeError: decoding Unicode is not supported.

What should I be doing?

Thanks for your help,

Aine.

[Diez B. Roggisch]

ai********@yahoo.com wrote:

Hi,

Im totally new to Python so please bare with me.

Data is entered into my program using the folling code -

str = raw_input(command)
words = str.split()

for word in words:
word = unicode(word,'latin-1')
word.encode('utf8')

This gives an error:

File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x94' in
position 0
: character maps to <undefined>

but the following works.

str = raw_input(command)
words = str.split()

for word in words:
uni = u""
uni = unicode(word,'latin-1')
uni.encode('utf8')

This is the exact same code as the one above - there is no type declaration
in python, so your

uni = u''

statement is bollocks.

And I seriously doubt, that the above code and the above error message are
related - as you can see, the error is from cp850, a windows codepage. But
that isn't in the code above - so there must be something else happening.

Please provide the full script, and the desired input - then we might be
able to help you.

Diez

[Fredrik Lundh]

Diez B. Roggisch wrote:

Please provide the full script, and the desired input - then we might be
able to help you.

the full *traceback* would pretty useful, too:

http://effbot.org/pyfaq/tutor-i-need...hould-i-do.htm

my guess is that the OP left out the print statement that causing the error, and the
traceback lines that pointed to that print statement:

more test.py

uni = u"p\x95l"
print uni

python test.py

Traceback (most recent call last):
File "test.py", line 2, in <module>
print uni
File "C:\python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x95' in position 1
: character maps to <undefined>

</F>

[Peter Otten]

ai********@yahoo.com wrote:

Im totally new to Python so please bare with me.

That's no problem, really. I don't use a spellchecker, either, and it
wouldn't have protected you from that particular typo...

Data is entered into my program using the folling code -

str = raw_input(command)
words = str.split()

for word in words:
word = unicode(word,'latin-1')
word.encode('utf8')

This gives an error:

File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x94' in
position 0
: character maps to <undefined>

but the following works.

str = raw_input(command)
words = str.split()

for word in words:
uni = u""
uni = unicode(word,'latin-1')
uni.encode('utf8')

Here you show us the same code twice, as the

uni = u""

assignment has no effect, and a traceback that is probably generated when
you try to

print uni

Here's my guess: The encoding you actually need is cp850, the same that your
Python interpreter is trying to use, but in which unichr(0x94) is
undefined. In general, you are not free to use a random encoding; rather,
you have to use what your console expects.

import sys

s = raw_input(command)
s = unicode(s, sys.stdin.encoding) # trust python to find out the proper
# encoding. If that fails use a constant,
# probably "cp850"
words = s.split():
for word in words:
print word # trust python, but if it doesn't work out:
# word = word.encode("cp850")
# print word

By the way, strings are immutable (cannot be altered once created), so the
following

word.encode('utf8')
print word

is actually spelt

word = word.encode("utf8")
print word

If your data is not read from the console and it contains characters that
cannot be printed, unicode.encode() accepts a second parameter to deal with
it, see

>>help(u"".encode)

Peter

[Peter Otten]

Peter Otten wrote:

words = s.split():

Oops, remove bogus colon here.

[John Machin]

ai********@yahoo.com wrote:

Hi,

Im totally new to Python so please bare with me.

Data is entered into my program using the folling code -

str = raw_input(command)
words = str.split()

for word in words:
word = unicode(word,'latin-1')
word.encode('utf8')

The above statement produces a string in utf8 and then throws it away.
It does not update "word". To retain the utf8 string, you would have to
do word = word.encode('utf8') and in any case that won't update the
original list.

*** missing source code line(s) here ***

>
This gives an error:

*** missing traceback lines here ***

File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x94' in
position 0
: character maps to <undefined>

No, it doesn't. You must have put "print word" to get the error that
you did.
*Please* when you are asking a question, copy/paste (1) the exact
source code that you ran (2) the exact traceback that you got.

>
but the following works.

What do you mean by "works"? It may not have triggered an error, but on
the other hand it doesn't do anything useful.

>
str = raw_input(command)
words = str.split()

for word in words:
uni = u""

Above line is pointless. Removing it will have no effect

uni = unicode(word,'latin-1')
uni.encode('utf8')

Same problem as above -- utf8 string is produced and then thrown away.

>
so the problem is that I want replace my list with unicode variables.
Or maybe I should create a new list.

I also tried this:

for word in words[:]:
word = u""
word = unicode(word,'latin-1')

You got the error on the above statement because you are trying
(pointlessly) to decode the value u"". Decoding means to convert from
some encoding to unicode.

word.encode('utf8')

Again, utf8 straight down the gurgler.

print word

This (if executed) will try to print the UNICODE version, and die [as
in the 1st example] encoding the unicode in cp950, which is the
encoding for your Windows command console.

>
but got TypeError: decoding Unicode is not supported.

What should I be doing?

(1) Reading the Unicode howto: http://www.amk.ca/python/howto/

(2) Writing some code like this:

| >>strg = "\x94 foo bar zot"
| >>words = strg.split()
| >>words
| ['\x94', 'foo', 'bar', 'zot']
| >>utf8words = [unicode(word, 'latin1').encode('utf8') for word in
words]
| >>utf8words
| ['\xc2\x94', 'foo', 'bar', 'zot']
| >>>

HTH,
John

[aine_canby]

Fredrik Lundh skrev:

Diez B. Roggisch wrote:

Please provide the full script, and the desired input - then we might be
able to help you.

the full *traceback* would pretty useful, too:

http://effbot.org/pyfaq/tutor-i-need...hould-i-do.htm

my guess is that the OP left out the print statement that causing the error, and the
traceback lines that pointed to that print statement:

more test.py

uni = u"p\x95l"
print uni

python test.py

Traceback (most recent call last):
File "test.py", line 2, in <module>
print uni
File "C:\python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x95' in position 1
: character maps to <undefined>

</F>

Thanks for the replies. OK heres the full code I'm using now -

compare = u"Äis"
print compare

str = raw_input("Enter music:")
words = str.split()

uniList=[]
for word in words:
uni=unicode(word,'latin-1')
uniList.append(uni)

print uniList[0]

if(compare!=uniList[0]):
print "Not the same: " + compare + " " + uniList[0]

This gives the following error -

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "test.py", line 14, in <module>
print uniList[0]
File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x8e' in
position 0
: character maps to <undefined>

How come I can print compare but not uniList[0]? What encoding is being
used to store compare? How can I print out the raw bytes stored in
compare and uniList[0]?

Thanks again for your help,

Aine.

[aine_canby]

[e-mail] skrev:

Fredrik Lundh skrev:

Diez B. Roggisch wrote:

Please provide the full script, and the desired input - then we mightbe
able to help you.

the full *traceback* would pretty useful, too:

http://effbot.org/pyfaq/tutor-i-need...hould-i-do.htm

my guess is that the OP left out the print statement that causing the error, and the
traceback lines that pointed to that print statement:

more test.py

uni = u"p\x95l"
print uni

python test.py

Traceback (most recent call last):
File "test.py", line 2, in <module>
print uni
File "C:\python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x95' in position 1
: character maps to <undefined>

</F>

Thanks for the replies. OK heres the full code I'm using now -

compare = u"Äis"
print compare

str = raw_input("Enter music:")
words = str.split()

uniList=[]
for word in words:
uni=unicode(word,'latin-1')
uniList.append(uni)

print uniList[0]

if(compare!=uniList[0]):
print "Not the same: " + compare + " " + uniList[0]

This gives the following error -

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "test.py", line 14, in <module>
print uniList[0]
File "C:\Python25\lib\encodings\cp850.py", line 12, in encode
return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\x8e' in
position 0
: character maps to <undefined>

How come I can print compare but not uniList[0]? What encoding is being
used to store compare? How can I print out the raw bytes stored in
compare and uniList[0]?

Thanks again for your help,

Aine.

Cool. It works for me now.

import sys

compare = u"Äis"
print compare

str = raw_input("Enter music:")
words = str.split()

uniList=[]
for word in words:
uni=unicode(word,sys.stdin.encoding)
uniList.append(uni)

print uniList[0]

if(compare!=uniList[0]):
print "Not the same: " + compare + " " + uniList[0]