"env" parameter to "popen" won't accept Unicode on Windows - minorUnicode bug

[John Nagle]

I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\lib\subprocess.py", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.

John Nagle

[Bjoern Schliessmann]

John Nagle wrote:

It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.

Regards,
Björn

--
BOFH excuse #31:

cellular telephone interference

[Benjamin]

On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
[e-mail]:

John Nagle wrote:

It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.

Otherwise: bugs.python.org

>
Regards,

Björn

--
BOFH excuse #31:

cellular telephone interference

[Benjamin]

On Jan 14, 6:26 pm, John Nagle <[e-mail]:

I passed a dict for the "env" variable to Popen with Unicode strings
for the dictionary values.

Got:

File "D:\Python24\lib\subprocess.py", line 706, in _execute_child
TypeError: environment can only contain strings

It turns out that the strings in the "env" parameter have to be ASCII,
not Unicode, even though Windows fully supports Unicode in CreateProcess.

>
John Nagle

[John Nagle]

Benjamin wrote:

On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
[e-mail]:

>John Nagle wrote:

>>It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.

Otherwise: bugs.python.org

Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

John Nagle

[Diez B. Roggisch]

John Nagle wrote:

Benjamin wrote:

>On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
[e-mail]:

>>John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.

>>Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.

Otherwise: bugs.python.org

John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.

Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

Sure thing, python will just magically convert unicode to the encoding the
program YOU invoke will expect. Right after we introduced the

solve_my_problem()

built-in-function. Any other wishes?

If I write this simple program

------ test.py -------
import os
import sys

ENCODDINGS = ['utf-8', 'latin1']

os.env["MY_VARIABLE"].encode(ENCODINGS[int(sys.argv[1])])
------ test.py -------
how's python supposed to know that

suprocess.call("python test.py 0", env=dict(MY_VARIABLE=u'foo'))

needs to be UTF-8?

Diez

[Brian Smith]

Diez B. Roggisch wrote:

Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_problem()

built-in-function. Any other wishes?

There's no reason to be rude.

Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

- Brian

[Diez B. Roggisch]

Brian Smith wrote:

Diez B. Roggisch wrote:

>Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_problem()

built-in-function. Any other wishes?

There's no reason to be rude.

If you'd know John, you'd know there is.

Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

For once, the distinction between windows and other platforms is debatable.
I admit that subprocess contains already quite a few platform specific
aspects, but it's purpose is to abstract these away as much as possible.

However, I'm not sure that just because there are wide-char windows apis
available automatically means that using UCS2/UTF-16 would succeed. A look
into the python sources (PC/_subprocess.c) reveals that someone already
thought about this, but it seems that just setting a
CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been
easy enough to do it if there weren't any troubles to expect.

Additionally, passing unicode to env would also imply that os.environ should
yield unicode as well. Not sure how much code _that_ breaks.

Diez

[Bjoern Schliessmann]

Brian Smith wrote:

popen() knows that it is running on Windows, and it knows what
encoding Windows needs for its environment (it's either UCS2 or
UTF-16 for most Windows APIs). At least when it receives a unicode
string, it has enough information to apply the conversion
automatically, and doing so saves the caller from having to figure
out what exact encoding is to be used.

So you propose Python should employ a hidden automatism that
automagically guesses the right encoding? Why not leave it
explicitly/consistently and let the user decide? What will happen
if a future Windows changes its encoding? Will we need another
magic routine to tell it apart?

Regards,
Björn

--
BOFH excuse #353:

Second-system effect.

[John Nagle]

Diez B. Roggisch wrote:

John Nagle wrote:

>Benjamin wrote:

>>On Jan 14, 6:26 pm, Bjoern Schliessmann <usenet-
[e-mail]:
John Nagle wrote:
It turns out that the strings in the "env" parameter have to be
ASCII, not Unicode, even though Windows fully supports Unicode in
CreateProcess.

That's of course nonsense, they don't need to be ascii, they need to be
byte-strings in whatever encoding you like.

>>>Are you sure it supports Unicode, not UTF8 or UTF16? Probably using
something like u"thestring".encode("utf16") will help.
Otherwise: bugs.python.org

John's understanding of the differences between unicode and it's encodings
is a bit blurry, to say the least.

Who's this guy?

>

> Whatever translation is necessary should be done in "popen", which
has cases for Windows and POSIX. "popen" is supposed to be cross-platform
to the extent possible. I think it's just something that didn't get fixed
when Unicode support went in.

I've been looking at the source code. There's "_PyPopenCreateProcess"
in "posixmodule.c". That one doesn't support passing an environment at
all; see the call to Windows CreateProcess. Is that the one that Popen uses?

Where is "win32process" in the source? It ought to be in Modules, but
it's not.

John Nagle

[John Nagle]

Diez B. Roggisch wrote:

Brian Smith wrote:

>Diez B. Roggisch wrote:

>>Sure thing, python will just magically convert unicode to the
encoding the program YOU invoke will expect. Right after we
introduced the

solve_my_problem()

built-in-function. Any other wishes?

There's no reason to be rude.

If you'd know John, you'd know there is.

?

>Anyway, at least on Windows it makes perfect sense for people to expect
Unicode to be handled automatically. popen() knows that it is running on
Windows, and it knows what encoding Windows needs for its environment
(it's either UCS2 or UTF-16 for most Windows APIs). At least when it
receives a unicode string, it has enough information to apply the
conversion automatically, and doing so saves the caller from having to
figure out what exact encoding is to be used.

For once, the distinction between windows and other platforms is debatable.
I admit that subprocess contains already quite a few platform specific
aspects, but it's purpose is to abstract these away as much as possible.

However, I'm not sure that just because there are wide-char windows apis
available automatically means that using UCS2/UTF-16 would succeed. A look
into the python sources (PC/_subprocess.c) reveals that someone already
thought about this, but it seems that just setting a
CREATE_UNICODE_ENVIRONMENT in the CreateProcess-function should have been
easy enough to do it if there weren't any troubles to expect.

The problem is that only the NT-derived Microsoft systems talk Unicode.
The DOS/Win16/Win9x family did not. But they did have CreateProcess.
So the current code will handle Win9x, but not Unicode.

When do we drop support for Win9x? It probably has to happen in
Python 3K, since that's Unicode-everywhere.

John Nagle

[Bjoern Schliessmann]

John Nagle wrote:

The problem is that only the NT-derived Microsoft systems
talk Unicode. The DOS/Win16/Win9x family did not. But they did
have CreateProcess. So the current code will handle Win9x, but not
Unicode.

Please explain, I don't understand. If you try using Windows system
functions in older Windows versions, u"mystring" will fail, too.
Those functions need byte strings, not Unicode string instances.
The latter have to be encoded to byte strings to pass them.

Regards,
Björn

--
BOFH excuse #70:

nesting roaches shorted out the ether cable