472,955 Members | 2,620 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,955 software developers and data experts.

Getting the encoding of sys.stdout and sys.stdin, and changing it properly

My headache is growing while playing arround with unicode in Python,
please help this novice. I have chosen to divide my problem into a few
questions.

Python 2.3.4 (#1, Feb 2 2005, 12:11:53)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2

1)
Does " >>>print 'hello' " simply write to sys.stdout?

2)
Exactly what does the following line return?
sys.stdout.encoding 'ISO-8859-1'

Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.

Is it the encoding of the string python "hands over" to the terminal? I
think not. In the following code i am pretty confident that the second
command changes that, and still sys.stdout.encoding is the same value.
import sys,codecs
sys.stdout.encoding 'ISO-8859-1' sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding 'ISO-8859-1'

Then what?

3)
Does raw_input() come from sys.stdin?

4)
The following script is not working, can you please tell me how to do
it right.
import codecs,sys
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdin = codecs.getreader('utf-8')(sys.stdin)
x = raw_input('write this unicode letter, Turkish che, unicode 0x00E7\t')

write this unicode letter, Turkish che, unicode 0x00E7
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.

velle, Denmark

Jan 3 '06 #1
2 4988
ve***@velle.dk wrote:
Exactly what does the following line return?
On your system, the result of nl_langinfo(CODESET),
see nl_langinfo(3).

On your system, this, in turn, is computed from the LANG,
LC_CHARSET, and LC_ALL environment variables.
Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.
It should be the encoding of your terminal. If you change the encoding
of your terminal, you should simultaneously also change your locale.

There is no way to determine the encoding of the terminal
programmatically, so the convention (on Unix) is that the locale setting
tells you what the terminal encoding is.

Unfortunately, there is also no way for the terminal to influence the
locale when it changes the encoding. So just don't do that, or do it
right.
Is it the encoding of the string python "hands over" to the terminal?
Not sure what you mean by "hands over". There is no way for the program
to tell the terminal what encoding to use (*), so Python cannot hand
something over. If you really mean "the encoding of the string sent
to the terminal", then ... perhaps. This encoding will only be used
when sending Unicode strings, not when sending byte strings (of those,
Python cannot know what encoding they have, and it sends them
unmodified).
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding


'ISO-8859-1'

Then what?


Then you have two bugs. One bug is in Python: it never occurred
to me to set the .encoding attribute on a stream writer, but that
might be a good idea. You also have a bug in your code: you
shouldn't set sys.stdout to a UTF-8 writer if the underlying
stream assumes ISO-8859-1.
3)
Does raw_input() come from sys.stdin?
Yes, it does.
4)
The following script is not working, can you please tell me how to do
it right.
Not exactly sure what you did (so it is hard to tell what to change),
however, I guess you should set your terminal to UTF-8 mode
(or, better, run the terminal in UTF-8 mode to begin with).

Alternatively, you should use the encoding that the terminal uses.
write this unicode letter, Turkish che, unicode 0x00E7
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.


What do you get when you do

% hexdump -C
<enter><ctl-d>

I expect you see

00000000 e7 0a |.|
00000002

which would indicate that your terminal sends latin-1.

Regards,
Martin

(*) Well, there is ISO 2022, but it doesn't really work, atleast not
in most terminals, and Python doesn't know how to use it, either.
Jan 4 '06 #2
I have been studying your reply for many hours now, trying to figure it
all out. I am novice in many respects; eg. I did not know neither
Locale, the command locale, nl_langinfo(), hexdump, that ctrl+d is EOF,
etc. :-)

However I have found a way that solves my original problem, and I will
post solutions/explanations to this thread within this week, as soon as
I feel that the concepts have fallen into place in my head.

Thanks, velle

Jan 5 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Hugh Macdonald | last post by:
I'm calling a command from within a python script and I need to be able to both catch the output (stdout and stderr) from it and also have the PID (so that I can kill it) I can do one or other...
2
by: Matthew Mueller | last post by:
I noticed in python2.3 printing unicode to an appropriate terminal actually works. But using sys.stdout.write doesn't. Ex: Python 2.3.4 (#2, May 29 2004, 03:31:27) on linux2 Type "help",...
6
by: Gandalf | last post by:
Hi All! I have a program that looks like this: # -*- coding: iso-8859-2 -*- s1 = 'nz' s2 = raw_input('Please type in "nz":') print repr(s1) print repr(s2)
0
by: lickspittle | last post by:
Hi, I have Python embedded with my other code, and when my other code opens a console and redirects stdout, stdin and stderr to it, then calls PyRun_InteractiveLoop, it immediately returns with...
6
by: Christian Convey | last post by:
Hello, I've got a program that (ideally) perpetually monitors sys.stdin for lines of text. As soon as a line comes in, my program takes some action. The problem is, it seems like a very large...
2
by: Markus Pitha | last post by:
Hello, Some time ago I tried to find a solution for preventing buffer overflows in stdin. I thought getc was the solution but today I came to a problem. I wanted to use my cognitions for a...
3
by: Neil Cerutti | last post by:
How do I cope with faulty encoding settings? I'm writing an application that needs all internal character data to be stored in iso-8859-1. It also must allow input and output using stdin and...
1
by: yc | last post by:
I have a encoding problem during using of subprocess. The input is a string with UTF-8 encoding. the code is: tokenize =...
4
by: Tobiah | last post by:
I am not sure how to capture the output of a command using subprocess without creating a temp file. I was trying this: import StringIO import subprocess file = StringIO.StringIO() ...
0
by: lllomh | last post by:
Define the method first this.state = { buttonBackgroundColor: 'green', isBlinking: false, // A new status is added to identify whether the button is blinking or not } autoStart=()=>{
2
by: DJRhino | last post by:
Was curious if anyone else was having this same issue or not.... I was just Up/Down graded to windows 11 and now my access combo boxes are not acting right. With win 10 I could start typing...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 4 Oct 2023 starting at 18:00 UK time (6PM UTC+1) and finishing at about 19:15 (7.15PM) The start time is equivalent to 19:00 (7PM) in Central...
4
NeoPa
by: NeoPa | last post by:
Hello everyone. I find myself stuck trying to find the VBA way to get Access to create a PDF of the currently-selected (and open) object (Form or Report). I know it can be done by selecting :...
3
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be using a very simple database which has Form (clsForm) & Report (clsReport) classes that simply handle making the calling Form invisible until the Form, or all...
1
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
3
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
0
NeoPa
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...
0
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.