473,695 Members | 2,527 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

"env" parameter to "popen" won't accept Unicode on Windows - minorUnicode bug

I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\li b\subprocess.py ", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.

John Nagle
Jan 15 '08 #1
11 4273
John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.
Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".en code("utf16") will help.

Regards,
Björn

--
BOFH excuse #31:

cellular telephone interference

Jan 15 '08 #2
On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n.. .@spamgourmet.c omwrote:
John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".en code("utf16") will help.
Otherwise: bugs.python.org
>
Regards,

Björn

--
BOFH excuse #31:

cellular telephone interference
Jan 15 '08 #3
On Jan 14, 6:26 pm, John Nagle <na...@animats. comwrote:
I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\li b\subprocess.py ", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.
>
John Nagle
Jan 15 '08 #4
Benjamin wrote:
On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n.. .@spamgourmet.c omwrote:
>John Nagle wrote:
>>It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess .
Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".en code("utf16") will help.
Otherwise: bugs.python.org
Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

John Nagle
Jan 15 '08 #5
John Nagle wrote:
Benjamin wrote:
>On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n.. .@spamgourmet.c omwrote:
>>John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProces s.
That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.
>>Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".en code("utf16") will help.
Otherwise: bugs.python.org
John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.
Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.
Sure thing, python will just magically convert unicode to the encoding the
program YOU invoke will expect. Right after we introduced the

solve_my_proble m()

built-in-function. Any other wishes?

If I write this simple program

------ test.py -------
import os
import sys

ENCODDINGS = ['utf-8', 'latin1']

os.env["MY_VARIABL E"].encode(ENCODIN GS[int(sys.argv[1])])
------ test.py -------
how's python supposed to know that

suprocess.call( "python test.py 0", env=dict(MY_VAR IABLE=u'foo'))

needs to be UTF-8?

Diez
Jan 15 '08 #6
Diez B. Roggisch wrote:
Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_proble m()

built-in-function. Any other wishes?
There's no reason to be rude.

Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

- Brian

Jan 15 '08 #7
Brian Smith wrote:
Diez B. Roggisch wrote:
>Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_probl em()

built-in-function. Any other wishes?

There's no reason to be rude.
If you'd know John, you'd know there is.
Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

For once, the distinction between windows and other platforms is debatable.
I admit that subprocess contains already quite a few platform specific
aspects, but it's purpose is to abstract these away as much as possible.

However, I'm not sure that just because there are wide-char windows apis
available automatically means that using UCS2/UTF-16 would succeed. A look
into the python sources (PC/_subprocess.c) reveals that someone already
thought about this, but it seems that just setting a
CREATE_UNICODE_ ENVIRONMENT in the CreateProcess-function should have been
easy enough to do it if there weren't any troubles to expect.

Additionally, passing unicode to env would also imply that os.environ should
yield unicode as well. Not sure how much code _that_ breaks.

Diez
Jan 15 '08 #8
Brian Smith wrote:
popen() knows that it is running on Windows, and it knows what
encoding Windows needs for its environment (it's either UCS2 or
UTF-16 for most Windows APIs). At least when it receives a unicode
string, it has enough information to apply the conversion
automatically, and doing so saves the caller from having to figure
out what exact encoding is to be used.
So you propose Python should employ a hidden automatism that
automagically guesses the right encoding? Why not leave it
explicitly/consistently and let the user decide? What will happen
if a future Windows changes its encoding? Will we need another
magic routine to tell it apart?

Regards,
Björn

--
BOFH excuse #353:

Second-system effect.

Jan 15 '08 #9
Diez B. Roggisch wrote:
John Nagle wrote:
>Benjamin wrote:
>>On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
mail-0306.20.chr0n.. .@spamgourmet.c omwrote:
John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProce ss.

That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.
>>>Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".en code("utf16") will help.
Otherwise: bugs.python.org

John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.
Who's this guy?
>
> Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.
I've been looking at the source code. There's "_PyPopenCreate Process"
in "posixmodule.c" . That one doesn't support passing an environment at
all; see the call to Windows CreateProcess. Is that the one that Popen uses?

Where is "win32proce ss" in the source? It ought to be in Modules, but
it's not.

John Nagle
Jan 15 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

3
3115
by: Jo Schambach | last post by:
I am trying to write a GUI with tkinter that displays the stdout from a regular C/C++ program in a text widget. The idea i was trying to use was as follows: 1) use "popen" to execute the C/C++ program 2) then use "tkinter.createfilehandler" to create a callback that would be called when the C/C++ program creates output on stdout. Somehow, I can't get this to work. here is what I have tried so far:
0
9113
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
8976
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8830
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8820
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
7655
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
4575
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2997
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
2265
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
3
1971
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.