WTF? Printing unicode strings

[Ron Garret]

>>> u'\xbd'
u'\xbd'

print _ Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

[John Salerno]

Ron Garret wrote:

u'\xbd' u'\xbd' print _ Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

Not sure if this really helps you, but:

u'\xbd' u'\xbd' print _ ½

[Fredrik Lundh]

Ron Garret wrote:

u'\xbd' u'\xbd' print _

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?

</F>

[Ron Garret]

In article <ma***************************************@python. org>,
Fredrik Lundh <fr*****@pythonware.com> wrote:

Ron Garret wrote:

> u'\xbd'

u'\xbd'

> print _

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?

</F>

I forgot to mention:

sys.getdefaultencoding() 'utf-8' print u'\xbd' Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

[Robert Kern]

Ron Garret wrote:

I forgot to mention:

sys.getdefaultencoding()
'utf-8'

A) You shouldn't be able to do that.
B) Don't do that.
C) It's not relevant to the encoding of stdout which determines how unicode
strings get converted to bytes when printing them:

import sys
sys.stdout.encoding 'UTF-8' sys.getdefaultencoding() 'ascii' print u'\xbd'

½

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Garret]

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

I forgot to mention:

>sys.getdefaultencoding()
'utf-8'

A) You shouldn't be able to do that.

What can I say? I can.
B) Don't do that.
OK. What should I do instead?
C) It's not relevant to the encoding of stdout which determines how unicode
strings get converted to bytes when printing them:

import sys
sys.stdout.encoding 'UTF-8' sys.getdefaultencoding() 'ascii' print u'\xbd' 1�2

OK, so how am I supposed to change the encoding of sys.stdout? It comes
up as US-ASCII on my system. Simply setting it doesn't work:

import sys
sys.stdout.encoding='utf-8' Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: readonly attribute

rg

[Robert Kern]

Ron Garret wrote:

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

I forgot to mention:
>>sys.getdefaultencoding()

'utf-8'

A) You shouldn't be able to do that.

What can I say? I can.

See B).

B) Don't do that.

OK. What should I do instead?

See below.

C) It's not relevant to the encoding of stdout which determines how unicode
strings get converted to bytes when printing them:

>import sys
>sys.stdout.encoding

'UTF-8'

>sys.getdefaultencoding()

'ascii'

>print u'\xbd'

1�2

OK, so how am I supposed to change the encoding of sys.stdout? It comes
up as US-ASCII on my system. Simply setting it doesn't work:

You will have to use a terminal that accepts UTF-8.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Serge Orlov]

Ron Garret wrote:

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

I forgot to mention:

>>>sys.getdefaultencoding()

'utf-8'

A) You shouldn't be able to do that.

What can I say? I can.

B) Don't do that.

OK. What should I do instead?

Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.

[Ron Garret]

In article <11**********************@j73g2000cwa.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

> I forgot to mention:
>
>>>>sys.getdefaultencoding()
>
> 'utf-8'

A) You shouldn't be able to do that.

What can I say? I can.

B) Don't do that.

OK. What should I do instead?

Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.

I'm using an OS X terminal to ssh to a Linux machine.

But what about this:

f2=open('foo','w')
f2.write(u'\xFF') Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)

That should have nothing to do with my terminal, right?

I just found http://www.amk.ca/python/howto/unicode, which seems to be
enlightening. The answer seems to be something like:

import codecs
f = codecs.open('foo','w','utf-8')

but that seems pretty awkward.

rg

[Robert Kern]

Ron Garret wrote:

I'm using an OS X terminal to ssh to a Linux machine.
Click on the "Terminal" menu, then "Window Settings...". Choose "Display" from
the combobox. At the bottom you will see a combobox title "Character Set
Encoding". Choose "Unicode (UTF-8)".
But what about this:

f2=open('foo','w')
f2.write(u'\xFF')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)

That should have nothing to do with my terminal, right?

Correct, that is a different problem. f.write() expects a string of bytes, not a
unicode string. In order to convert unicode strings to byte strings without an
explicit .encode() method call, Python uses the default encoding which is
'ascii'. It's not easily changeable for a good reason. Your modules won't work
on anyone else's machine if you hack that setting.
I just found http://www.amk.ca/python/howto/unicode, which seems to be
enlightening. The answer seems to be something like:

import codecs
f = codecs.open('foo','w','utf-8')

but that seems pretty awkward.

<shrug> About as clean as it gets when dealing with text encodings.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Paul Boddie]

Ron Garret wrote:

But what about this:

f2=open('foo','w')
f2.write(u'\xFF') Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)

That should have nothing to do with my terminal, right?

Correct. But first try to answer this: given that you want to write the
Unicode character value 255 to a file, how is that character to be
represented in the file?

For example, one might think that one could just get a byte whose value
is 255 and write that to a file, but what happens if one chooses a
Unicode character whose value is greater than 255? One could use two
bytes or three bytes or as many as one needs, but what if the lowest 8
bits of that value are all set? How would one know, if one reads a file
back and gets a byte whose value is 255 whether it represents a
character all by itself or is part of another character's
representation? It gets complicated!

The solution is that you choose an encoding which allows you to store
the characters in the file, thus answering indirectly the question
above: encodings determine how the characters are represented in the
file and allow you to read the file and get back the characters you put
into it. One of the most common encodings suitable for the storage of
Unicode character values is UTF-8, which has been designed with the
above complications in mind, but as long as you remember to choose an
encoding, you don't have to think about it: Python takes care of the
difficult stuff on your behalf. In the above code you haven't made that
choice.

So, to answer the above question, you can either...

* Use the encode method on Unicode objects to turn them into plain
strings, then write them to a file - at that point, you are
writing specific byte values.
* Use the codecs.open function and other codecs module features to
write Unicode objects directly to files and streams - here, the
module's infrastructure deals with byte-level issues.
* If you're using something like an XML library, you can often pass a
normal file or stream object to some function or method whilst
stating the output encoding.

There is no universally correct answer to which encoding should be used
when writing Unicode character values to files, contrary to some
beliefs and opinions which, for example, lead to people pretending that
everything is in UTF-8 in order to appease legacy applications with the
minimum of tweaks necessary to stop them from breaking completely.
Thus, Python doesn't make a decision for you here.

Paul

[Ron Garret]

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

I'm using an OS X terminal to ssh to a Linux machine.

Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
from
the combobox. At the bottom you will see a combobox title "Character Set
Encoding". Choose "Unicode (UTF-8)".

It was already set to UTF-8.

But what about this:

>f2=open('foo','w')
>f2.write(u'\xFF')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)

That should have nothing to do with my terminal, right?

Correct, that is a different problem. f.write() expects a string of bytes,
not a
unicode string. In order to convert unicode strings to byte strings without
an
explicit .encode() method call, Python uses the default encoding which is
'ascii'. It's not easily changeable for a good reason. Your modules won't
work
on anyone else's machine if you hack that setting.

OK.

I just found http://www.amk.ca/python/howto/unicode, which seems to be
enlightening. The answer seems to be something like:

import codecs
f = codecs.open('foo','w','utf-8')

but that seems pretty awkward.

<shrug> About as clean as it gets when dealing with text encodings.

OK. Thanks.

rg

[Serge Orlov]

Ron Garret wrote:

In article <11**********************@j73g2000cwa.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

> Ron Garret wrote:
>
> > I forgot to mention:
> >
> >>>>sys.getdefaultencoding()
> >
> > 'utf-8'
>
> A) You shouldn't be able to do that.

What can I say? I can.

> B) Don't do that.

OK. What should I do instead?

Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.

I'm using an OS X terminal to ssh to a Linux machine.

In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:

LANG=en_US.utf-8 python -c "print unichr(0xbd)"

Or put the following line in ~/.bashrc and logout/login

export LANG=en_US.utf-8

[Robert Kern]

Ron Garret wrote:

In article <ma***************************************@python. org>,
Robert Kern <ro*********@gmail.com> wrote:

Ron Garret wrote:

I'm using an OS X terminal to ssh to a Linux machine.

Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
from
the combobox. At the bottom you will see a combobox title "Character Set
Encoding". Choose "Unicode (UTF-8)".

It was already set to UTF-8.

Then take a look at your LANG environment variable on your Linux machine. For
example, I have LANG=en_US.UTF-8 on my Linux machine, and I can ssh into it from
a UTF-8-configured Terminal.app and print unicode strings just fine.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Garret]

In article <11**********************@y43g2000cwc.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

In article <11**********************@j73g2000cwa.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:
> In article <ma***************************************@python. org>,
> Robert Kern <ro*********@gmail.com> wrote:
>
> > Ron Garret wrote:
> >
> > > I forgot to mention:
> > >
> > >>>>sys.getdefaultencoding()
> > >
> > > 'utf-8'
> >
> > A) You shouldn't be able to do that.
>
> What can I say? I can.
>
> > B) Don't do that.
>
> OK. What should I do instead?

Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.

I'm using an OS X terminal to ssh to a Linux machine.

In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:

LANG=en_US.utf-8 python -c "print unichr(0xbd)"

Or put the following line in ~/.bashrc and logout/login

export LANG=en_US.utf-8

No joy.

ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$

rg

[Serge Orlov]

Ron Garret wrote:

I'm using an OS X terminal to ssh to a Linux machine.

In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:

LANG=en_US.utf-8 python -c "print unichr(0xbd)"

Or put the following line in ~/.bashrc and logout/login

export LANG=en_US.utf-8

No joy.

ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version

[Ron Garret]

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

> I'm using an OS X terminal to ssh to a Linux machine.

In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:

LANG=en_US.utf-8 python -c "print unichr(0xbd)"

Or put the following line in ~/.bashrc and logout/login

export LANG=en_US.utf-8

No joy.

ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version

ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

[Serge Orlov]

Ron Garret wrote:

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

> > I'm using an OS X terminal to ssh to a Linux machine.
>
> In theory it should work out of the box. OS X terminal should set
> enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> variable to Linux and python will know that your terminal is utf-8.
> Unfortunately AFAIK OS X terminal doesn't set that variable and most
> (all?) ssh clients don't transfer it between machines. As a workaround
> you can set that variable on linux yourself . This should work in the
> command line right away:
>
> LANG=en_US.utf-8 python -c "print unichr(0xbd)"
>
> Or put the following line in ~/.bashrc and logout/login
>
> export LANG=en_US.utf-8

No joy.

ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version

ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason. Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:

LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"

Should be working now :)

[Serge Orlov]

Serge Orlov wrote:

Ron Garret wrote:

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:
> > > I'm using an OS X terminal to ssh to a Linux machine.
> >
> > In theory it should work out of the box. OS X terminal should set
> > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > variable to Linux and python will know that your terminal is utf-8.
> > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > (all?) ssh clients don't transfer it between machines. As a workaround
> > you can set that variable on linux yourself . This should work in the
> > command line right away:
> >
> > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> >
> > Or put the following line in ~/.bashrc and logout/login
> >
> > export LANG=en_US.utf-8
>
> No joy.
>
> ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> Traceback (most recent call last):
> File "<string>", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)
> ron@www01:~$

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version

ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason. Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:

LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"

Should be working now :)

I've pulled myself together and installed linux in vwware player.
Apparently there is another way linux distributors can screw up. I
chose debian 3.1 minimal network install and after answering all
installation questions I found that only ascii and latin-1 english
locales were installed:
$ locale -a
C
en_US
en_US.iso88591
POSIX

In 2006, I would expect utf-8 english locale to be present even in
minimal install. I had to edit /etc/locale.gen and run locale-gen as
root. After that python started to print unicode characters.

[Laurent Pointal]

Ron Garret a écrit :

In article <ma***************************************@python. org>,
Fredrik Lundh <fr*****@pythonware.com> wrote:

Ron Garret wrote:

>> u'\xbd'
u'\xbd'
>> print _
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?

</F>

I forgot to mention:

sys.getdefaultencoding() 'utf-8' print u'\xbd' Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

This is default encoding for evaluation of expressions in u"..."
strings, this has nothing to do with printing.

For the output encoding, see sys.stdout.encoding.

import sys
sys.stdout.encoding 'cp850'

A+

Laurent.

[John Salerno]

Fredrik Lundh wrote:

Ron Garret wrote:

> u'\xbd'

u'\xbd'

> print _

Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?

</F>

AFAIK, I'm all ASCII (at least, I never made explicit changes to the
default Python install), so how am I able to print out the character?

[Robert Kern]

John Salerno wrote:

AFAIK, I'm all ASCII (at least, I never made explicit changes to the
default Python install), so how am I able to print out the character?

Because sys.stdout.encoding isn't determined by your Python configuration, but
your terminal's.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[skip]

Robert> Because sys.stdout.encoding isn't determined by your Python
Robert> configuration, but your terminal's.

Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?

% python
Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
[GCC 3.4.1] on sunos5
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.stdout.encoding '646' import codecs
codecs.lookup("646")

(<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii.StreamReader at 0x819aa4c>, <class encodings.ascii.StreamWriter at 0x819aa1c>)

Skip

[John Salerno]

sk**@pobox.com wrote:

Robert> Because sys.stdout.encoding isn't determined by your Python
Robert> configuration, but your terminal's.

Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?

Hmm, not that this helps me any :)

import sys
sys.stdout.encoding 'cp1252' import codecs
codecs.lookup('cp1252') (<bound method Codec.encode of <encodings.cp1252.Codec instance at
0x009D6670>>, <bound method Codec.decode of <encodings.cp1252.Codec
instance at 0x009D6698>>, <class encodings.cp1252.StreamReader at
0x009CF360>, <class encodings.cp1252.StreamWriter at 0x009CF330>)

[skip]

John> Hmm, not that this helps me any :)

import sys
sys.stdout.encoding

John> 'cp1252'

Sure it does. You can print Unicode objects which map to cp1252. I assume
that means you're on Windows or that for some perverse reason you have your
Mac's Terminal window set to cp1252. (Does it go there? I'm at work right
now so I can't check).

Skip

[John Salerno]

sk**@pobox.com wrote:

John> Hmm, not that this helps me any :)

>>>> import sys
>>>> sys.stdout.encoding

John> 'cp1252'

Sure it does. You can print Unicode objects which map to cp1252. I assume
that means you're on Windows or that for some perverse reason you have your
Mac's Terminal window set to cp1252. (Does it go there? I'm at work right
now so I can't check).

Skip

You're right, I'm on XP. I just couldn't make sense of the lookup call,
although some of the names looked like .NET classes.

[Robert Kern]

sk**@pobox.com wrote:

Robert> Because sys.stdout.encoding isn't determined by your Python
Robert> configuration, but your terminal's.

Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?

% python
Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
[GCC 3.4.1] on sunos5
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> sys.stdout.encoding '646' >>> import codecs
>>> codecs.lookup("646")

(<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii.StreamReader at 0x819aa4c>, <class encodings.ascii.StreamWriter at 0x819aa1c>)

Yes. In encodings/aliases.py in the standard library:

"""
aliases = {

# Please keep this list sorted alphabetically by value !

# ascii codec
'646' : 'ascii',

"""

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ron Garret]

In article <11**********************@i40g2000cwc.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Ron Garret wrote:
> > > I'm using an OS X terminal to ssh to a Linux machine.
> >
> > In theory it should work out of the box. OS X terminal should set
> > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > variable to Linux and python will know that your terminal is utf-8.
> > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > (all?) ssh clients don't transfer it between machines. As a workaround
> > you can set that variable on linux yourself . This should work in the
> > command line right away:
> >
> > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> >
> > Or put the following line in ~/.bashrc and logout/login
> >
> > export LANG=en_US.utf-8
>
> No joy.
>
> ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> Traceback (most recent call last):
> File "<string>", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)
> ron@www01:~$

What version of python and what shell do you run? What the following
commands print:

python -V
echo $SHELL
$SHELL --version
ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason.

Nope:

ron@www01:~$ export | grep LC
ron@www01:~$
Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:

LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"

Should be working now :)

Nope:

ron@www01:~$ LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)

rg

[Ron Garret]

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

Serge Orlov wrote:

Ron Garret wrote:

In article <11**********************@g10g2000cwb.googlegroups .com>,
"Serge Orlov" <Se*********@gmail.com> wrote:

> Ron Garret wrote:
> > > > I'm using an OS X terminal to ssh to a Linux machine.
> > >
> > > In theory it should work out of the box. OS X terminal should set
> > > enviromental variable LANG=en_US.utf-8, then ssh should transfer
> > > this
> > > variable to Linux and python will know that your terminal is utf-8.
> > > Unfortunately AFAIK OS X terminal doesn't set that variable and
> > > most
> > > (all?) ssh clients don't transfer it between machines. As a
> > > workaround
> > > you can set that variable on linux yourself . This should work in
> > > the
> > > command line right away:
> > >
> > > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > >
> > > Or put the following line in ~/.bashrc and logout/login
> > >
> > > export LANG=en_US.utf-8
> >
> > No joy.
> >
> > ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> > Traceback (most recent call last):
> > File "<string>", line 1, in ?
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
> > position 0: ordinal not in range(128)
> > ron@www01:~$
>
> What version of python and what shell do you run? What the following
> commands print:
>
> python -V
> echo $SHELL
> $SHELL --version

ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$

That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason. Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:

LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"

Should be working now :)

I've pulled myself together and installed linux in vwware player.
Apparently there is another way linux distributors can screw up. I
chose debian 3.1 minimal network install and after answering all
installation questions I found that only ascii and latin-1 english
locales were installed:
$ locale -a
C
en_US
en_US.iso88591
POSIX

In 2006, I would expect utf-8 english locale to be present even in
minimal install. I had to edit /etc/locale.gen and run locale-gen as
root. After that python started to print unicode characters.

That's it. Thanks!

rg

[Martin v. Löwis]

sk**@pobox.com wrote:

Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?

Usage of "646" as an alias for ASCII is primarily a Sun invention. When
ASCII became an international standard, its standard number became
ISO/IEC 646:1968. It's not *quite* the same as ASCII, as it leaves a
certain number of code points unassigned that ASCII defines (most
notably, the dollar sign, and the square and curly braces). What Sun
means is probably the "International Reference Version" of ISO 646,
which is (now) identical to ASCII.

Regards,
Martin