Getting the encoding of sys.stdout and sys.stdin, and changing it properly

[velle]

My headache is growing while playing arround with unicode in Python,
please help this novice. I have chosen to divide my problem into a few
questions.

Python 2.3.4 (#1, Feb 2 2005, 12:11:53)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2

1)
Does " >>>print 'hello' " simply write to sys.stdout?

2)
Exactly what does the following line return?

sys.stdout.encoding 'ISO-8859-1'

Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.

Is it the encoding of the string python "hands over" to the terminal? I
think not. In the following code i am pretty confident that the second
command changes that, and still sys.stdout.encoding is the same value.
import sys,codecs
sys.stdout.encoding 'ISO-8859-1' sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding 'ISO-8859-1'

Then what?

3)
Does raw_input() come from sys.stdin?

4)
The following script is not working, can you please tell me how to do
it right.
import codecs,sys
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdin = codecs.getreader('utf-8')(sys.stdin)
x = raw_input('write this unicode letter, Turkish che, unicode 0x00E7\t')

write this unicode letter, Turkish che, unicode 0x00E7 ç
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.

velle, Denmark

[Martin v. Löwis]

ve***@velle.dk wrote:

Exactly what does the following line return?
On your system, the result of nl_langinfo(CODESET),
see nl_langinfo(3).

On your system, this, in turn, is computed from the LANG,
LC_CHARSET, and LC_ALL environment variables.
Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.
It should be the encoding of your terminal. If you change the encoding
of your terminal, you should simultaneously also change your locale.

There is no way to determine the encoding of the terminal
programmatically, so the convention (on Unix) is that the locale setting
tells you what the terminal encoding is.

Unfortunately, there is also no way for the terminal to influence the
locale when it changes the encoding. So just don't do that, or do it
right.
Is it the encoding of the string python "hands over" to the terminal?
Not sure what you mean by "hands over". There is no way for the program
to tell the terminal what encoding to use (*), so Python cannot hand
something over. If you really mean "the encoding of the string sent
to the terminal", then ... perhaps. This encoding will only be used
when sending Unicode strings, not when sending byte strings (of those,
Python cannot know what encoding they have, and it sends them
unmodified).

sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding

'ISO-8859-1'

Then what?

Then you have two bugs. One bug is in Python: it never occurred
to me to set the .encoding attribute on a stream writer, but that
might be a good idea. You also have a bug in your code: you
shouldn't set sys.stdout to a UTF-8 writer if the underlying
stream assumes ISO-8859-1.
3)
Does raw_input() come from sys.stdin?
Yes, it does.
4)
The following script is not working, can you please tell me how to do
it right.
Not exactly sure what you did (so it is hard to tell what to change),
however, I guess you should set your terminal to UTF-8 mode
(or, better, run the terminal in UTF-8 mode to begin with).

Alternatively, you should use the encoding that the terminal uses.
write this unicode letter, Turkish che, unicode 0x00E7 ç
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.

What do you get when you do

% hexdump -C
ç<enter><ctl-d>

I expect you see

00000000 e7 0a |ç.|
00000002

which would indicate that your terminal sends latin-1.

Regards,
Martin

(*) Well, there is ISO 2022, but it doesn't really work, atleast not
in most terminals, and Python doesn't know how to use it, either.

[velle]

I have been studying your reply for many hours now, trying to figure it
all out. I am novice in many respects; eg. I did not know neither
Locale, the command locale, nl_langinfo(), hexdump, that ctrl+d is EOF,
etc. :-)

However I have found a way that solves my original problem, and I will
post solutions/explanations to this thread within this week, as soon as
I feel that the concepts have fallen into place in my head.

Thanks, velle