473,287 Members | 1,580 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

Getting the encoding of sys.stdout and sys.stdin, and changing it properly

My headache is growing while playing arround with unicode in Python,
please help this novice. I have chosen to divide my problem into a few
questions.

Python 2.3.4 (#1, Feb 2 2005, 12:11:53)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2

1)
Does " >>>print 'hello' " simply write to sys.stdout?

2)
Exactly what does the following line return?
sys.stdout.encoding 'ISO-8859-1'

Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.

Is it the encoding of the string python "hands over" to the terminal? I
think not. In the following code i am pretty confident that the second
command changes that, and still sys.stdout.encoding is the same value.
import sys,codecs
sys.stdout.encoding 'ISO-8859-1' sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding 'ISO-8859-1'

Then what?

3)
Does raw_input() come from sys.stdin?

4)
The following script is not working, can you please tell me how to do
it right.
import codecs,sys
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdin = codecs.getreader('utf-8')(sys.stdin)
x = raw_input('write this unicode letter, Turkish che, unicode 0x00E7\t')

write this unicode letter, Turkish che, unicode 0x00E7 ç
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.

velle, Denmark

Jan 3 '06 #1
2 5022
ve***@velle.dk wrote:
Exactly what does the following line return?
On your system, the result of nl_langinfo(CODESET),
see nl_langinfo(3).

On your system, this, in turn, is computed from the LANG,
LC_CHARSET, and LC_ALL environment variables.
Is it the encoding of the terminal? I think not, because when I change
the encoding in my terminal the result is still the same.
It should be the encoding of your terminal. If you change the encoding
of your terminal, you should simultaneously also change your locale.

There is no way to determine the encoding of the terminal
programmatically, so the convention (on Unix) is that the locale setting
tells you what the terminal encoding is.

Unfortunately, there is also no way for the terminal to influence the
locale when it changes the encoding. So just don't do that, or do it
right.
Is it the encoding of the string python "hands over" to the terminal?
Not sure what you mean by "hands over". There is no way for the program
to tell the terminal what encoding to use (*), so Python cannot hand
something over. If you really mean "the encoding of the string sent
to the terminal", then ... perhaps. This encoding will only be used
when sending Unicode strings, not when sending byte strings (of those,
Python cannot know what encoding they have, and it sends them
unmodified).
sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
sys.stdout.encoding


'ISO-8859-1'

Then what?


Then you have two bugs. One bug is in Python: it never occurred
to me to set the .encoding attribute on a stream writer, but that
might be a good idea. You also have a bug in your code: you
shouldn't set sys.stdout to a UTF-8 writer if the underlying
stream assumes ISO-8859-1.
3)
Does raw_input() come from sys.stdin?
Yes, it does.
4)
The following script is not working, can you please tell me how to do
it right.
Not exactly sure what you did (so it is hard to tell what to change),
however, I guess you should set your terminal to UTF-8 mode
(or, better, run the terminal in UTF-8 mode to begin with).

Alternatively, you should use the encoding that the terminal uses.
write this unicode letter, Turkish che, unicode 0x00E7 ç
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib/python2.3/codecs.py", line 295, in readline
return self.decode(line, self.errors)[0]
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
unexpected end of data

When prompted, I simply enter the che with my Turkish keyboard layout.


What do you get when you do

% hexdump -C
ç<enter><ctl-d>

I expect you see

00000000 e7 0a |ç.|
00000002

which would indicate that your terminal sends latin-1.

Regards,
Martin

(*) Well, there is ISO 2022, but it doesn't really work, atleast not
in most terminals, and Python doesn't know how to use it, either.
Jan 4 '06 #2
I have been studying your reply for many hours now, trying to figure it
all out. I am novice in many respects; eg. I did not know neither
Locale, the command locale, nl_langinfo(), hexdump, that ctrl+d is EOF,
etc. :-)

However I have found a way that solves my original problem, and I will
post solutions/explanations to this thread within this week, as soon as
I feel that the concepts have fallen into place in my head.

Thanks, velle

Jan 5 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

8
by: Hugh Macdonald | last post by:
I'm calling a command from within a python script and I need to be able to both catch the output (stdout and stderr) from it and also have the PID (so that I can kill it) I can do one or other...
2
by: Matthew Mueller | last post by:
I noticed in python2.3 printing unicode to an appropriate terminal actually works. But using sys.stdout.write doesn't. Ex: Python 2.3.4 (#2, May 29 2004, 03:31:27) on linux2 Type "help",...
6
by: Gandalf | last post by:
Hi All! I have a program that looks like this: # -*- coding: iso-8859-2 -*- s1 = 'néz' s2 = raw_input('Please type in "néz":') print repr(s1) print repr(s2)
0
by: lickspittle | last post by:
Hi, I have Python embedded with my other code, and when my other code opens a console and redirects stdout, stdin and stderr to it, then calls PyRun_InteractiveLoop, it immediately returns with...
6
by: Christian Convey | last post by:
Hello, I've got a program that (ideally) perpetually monitors sys.stdin for lines of text. As soon as a line comes in, my program takes some action. The problem is, it seems like a very large...
2
by: Markus Pitha | last post by:
Hello, Some time ago I tried to find a solution for preventing buffer overflows in stdin. I thought getc was the solution but today I came to a problem. I wanted to use my cognitions for a...
3
by: Neil Cerutti | last post by:
How do I cope with faulty encoding settings? I'm writing an application that needs all internal character data to be stored in iso-8859-1. It also must allow input and output using stdin and...
1
by: yc | last post by:
I have a encoding problem during using of subprocess. The input is a string with UTF-8 encoding. the code is: tokenize =...
4
by: Tobiah | last post by:
I am not sure how to capture the output of a command using subprocess without creating a temp file. I was trying this: import StringIO import subprocess file = StringIO.StringIO() ...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.