Bug/Weak Implementation? popen* routines can't handle simultaneous read/write?

[dmoore]

Hi folks,

I've seen the following issue come up in multiple posts to this
mailing list:

I have a python program that spawns a child process with popen or
popen2 or popen3 or popen2.popen2 etc.
the child process is interactive: it asks for input then spits out
some output, asks for more input then spits out some output. for
example, consider the trivial child program:

print "welcome"
print ">",
s=raw_input()
while s!='exit':
print "you entered:",s
print ">",
s=raw_input()

Now I may be completely wrong about this (I did play with popen for a
very long time before writing this message), but it appears that none
of the popen variants allow for a sequence of reads and writes to/from
this child. that is, if I read from the open pipe's output I will
never be able to write the input for the child because the parent
program will block on read until eof (I will have similar blocking
problems if I start with write - using readline does not seem to
help).

the standard proposed remedy I have seen on this list is to use Unix-
only select/fctl, or otherwise dig into the bowls of the win32 api, or
download some half-complete sourceforge process control project. All
well and good, but unsatisfying for writing platform independent code.

it turns out that there is at least one open source multi-platform
(read: win32/linux) api that does handle synchronous I/O with the
child: wxWidgets and wxPython using the class wxProcess. Now the
wxWidgets implementation is far from perfect, but it at least allows a
program to test for new input on the child's stdout and read stdout/
write stdout in a non-blocking way. However, I find it frustrating
that I have to import wx just to have useable interactive pipes in my
python scripts when I would expect this to be part of the native
python implementation.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

[Nick Craig-Wood]

dmoore <da**********@gmail.comwrote:

I've seen the following issue come up in multiple posts to this
mailing list:

I have a python program that spawns a child process with popen or
popen2 or popen3 or popen2.popen2 etc.
the child process is interactive: it asks for input then spits out
some output, asks for more input then spits out some output. for
example, consider the trivial child program:

print "welcome"
print ">",
s=raw_input()
while s!='exit':
print "you entered:",s
print ">",
s=raw_input()

Now I may be completely wrong about this (I did play with popen for a
very long time before writing this message), but it appears that none
of the popen variants allow for a sequence of reads and writes to/from
this child. that is, if I read from the open pipe's output I will
never be able to write the input for the child because the parent
program will block on read until eof (I will have similar blocking
problems if I start with write - using readline does not seem to
help).

You are correct.

the standard proposed remedy I have seen on this list is to use Unix-
only select/fctl, or otherwise dig into the bowls of the win32 api, or
download some half-complete sourceforge process control project.

If you are referring to pexpect I've found it works well - unix only
though. I've not noticed it being half complete.

There is also a recipe for a non-blocking subprocess - see

http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

All well and good, but unsatisfying for writing platform independent code.

it turns out that there is at least one open source multi-platform
(read: win32/linux) api that does handle synchronous I/O with the
child: wxWidgets and wxPython using the class wxProcess. Now the
wxWidgets implementation is far from perfect, but it at least allows a
program to test for new input on the child's stdout and read stdout/
write stdout in a non-blocking way.

Interesting I didn't know about that - I shall try it!

However, I find it frustrating
that I have to import wx just to have useable interactive pipes in my
python scripts when I would expect this to be part of the native
python implementation.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

The non-blocking subprocess would make a good start for a stdlib
submission.

It should really optionally use ptys under unix too otherwise you'll
never be able to script passwd etc. An interface a bit like pexpect
wpuld be useful too (ie scan for these regexps or timeout and return a
match object).

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[Terry Reedy]

"dmoore" <da**********@gmail.comwrote in message
news:11**********************@w5g2000hsg.googlegro ups.com...
| alternatives in the standard python distro) Is there some place I can
| submit this as a feature request? (Python dev?)

http://sourceforge.net/tracker/?group_id=5470

But don't hold your breath for a volunteer to respond. Discussion here is
good until you have a concrete request and support/rationale.

[Noah]

On Jun 7, 9:01 am, dmoore <[e-mail]:

popen and friends will never do what you want it to do. Down that path
lies bitter disappointment.
You need pseduo-ttys and non-blocking IO. I don't know how to do this
on Windows, but I know it's possible.
Cygwin does it.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

Try Pexpect http://pexpect.sourceforge.net/
It's been around for a long time and is quite complete and stable.

The catch is that it's UNIX-only. If you want to tease out the magic
code from wxProcess that
does non-blocking reads on win32 then I'd be happy to integrate that
into the current development branch
of Pexpect. If someone can provide a pure Python drop-in replacement
for the read_nonblocking() function
I use for UNIX systems then it would be easy. I have a whole test
framework that I can run it through to
see if it performs the same as the UNIX flavor.

I'm sure this is feasible without any C extensions -- it might require
some ctypes hacking.
I know Windows has pretty decent async IO, but I don't know what they
have as an equivalent for a pty.
Maybe it isn't necessary. A pty is only necessary on UNIX because the
standard c library, stdio, behaves
differently when it's talking to a plain pipe versus a terminal -- it
switches buffering
between block and line oriented buffer. You don't want block buffering
on interactive applications.
This is why popen eventually breaks down. No, there is no way to
select this behavior from
the calling side... unless you can trick it into dynamically linking
to your specially hacked libc.
But that's getting ahead because that's what happens on UNIX -- it
might be a non-issue on Windows.

The read_nonblocking() function I use has an interface like this:
<pre>
def read_nonblocking (self, size = 1, timeout = -1):

"""This reads at most size characters from the child
application. It
includes a timeout. If the read does not complete within the
timeout
period then a TIMEOUT exception is raised. If the end of file
is read
then an EOF exception will be raised. If a log file was set
using
setlog() then all data will also be written to the log file.

If timeout is None then the read may block indefinitely. If
timeout is -1
then the self.timeout value is used. If timeout is 0 then the
child is
polled and if there was no data immediately ready then this
will raise
a TIMEOUT exception.

The timeout refers only to the amount of time to read at least
one
character. This is not effected by the 'size' parameter, so if
you call
read_nonblocking(size=100, timeout=30) and only one character
is
available right away then one character will be returned
immediately.
It will not wait for 30 seconds for another 99 characters to
come in.

This is a wrapper around os.read(). It uses select.select() to
implement the timeout. """
</pre>

Yours,
Noah

[dmoore]

thanks for all of your responses. i'll look more closely at pexpect
(The version I originally saw was much much older). I do want a cross-
platform solution. I'm pretty busy so getting the wxProcess magic into
useful shape will take me some time (at least on the order of weeks)
so other interest parties feel free to step up.

cheers
Damien

On Jun 7, 8:59 pm, Noah <[e-mail]:

On Jun 7, 9:01 am, dmoore <[e-mail]:

popen and friends will never do what you want it to do. Down that path
lies bitter disappointment.
You need pseduo-ttys and non-blocking IO. I don't know how to do this
on Windows, but I know it's possible.
Cygwin does it.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

Try Pexpecthttp://pexpect.sourceforge.net/
It's been around for a long time and is quite complete and stable.

The catch is that it's UNIX-only. If you want to tease out the magic
code from wxProcess that
does non-blocking reads on win32 then I'd be happy to integrate that
into the current development branch
of Pexpect. If someone can provide a pure Python drop-in replacement
for the read_nonblocking() function
I use for UNIX systems then it would be easy. I have a whole test
framework that I can run it through to
see if it performs the same as the UNIX flavor.

I'm sure this is feasible without any C extensions -- it might require
some ctypes hacking.
I know Windows has pretty decent async IO, but I don't know what they
have as an equivalent for a pty.
Maybe it isn't necessary. A pty is only necessary on UNIX because the
standard c library, stdio, behaves
differently when it's talking to a plain pipe versus a terminal -- it
switches buffering
between block and line oriented buffer. You don't want block buffering
on interactive applications.
This is why popen eventually breaks down. No, there is no way to
select this behavior from
the calling side... unless you can trick it into dynamically linking
to your specially hacked libc.
But that's getting ahead because that's what happens on UNIX -- it
might be a non-issue on Windows.

The read_nonblocking() function I use has an interface like this:
<pre>
def read_nonblocking (self, size = 1, timeout = -1):

"""This reads at most size characters from the child
application. It
includes a timeout. If the read does not complete within the
timeout
period then a TIMEOUT exception is raised. If the end of file
is read
then an EOF exception will be raised. If a log file was set
using
setlog() then all data will also be written to the log file.

If timeout is None then the read may block indefinitely. If
timeout is -1
then the self.timeout value is used. If timeout is 0 then the
child is
polled and if there was no data immediately ready then this
will raise
a TIMEOUT exception.

The timeout refers only to the amount of time to read at least
one
character. This is not effected by the 'size' parameter, so if
you call
read_nonblocking(size=100, timeout=30) and only one character
is
available right away then one character will be returned
immediately.
It will not wait for 30 seconds for another 99 characters to
come in.

This is a wrapper around os.read(). It uses select.select() to
implement the timeout. """
</pre>

Yours,
Noah

[Hendrik van Rooyen]

"dmoore" <[e-mail]:

8< --------------- <description of full duplex problem------------

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

I think this is a hassle in the file implementation - I have had it also on
serial ports, where it acts as if the file "driver" is inherently half duplex.

It manifests as: "My characters don't come out"

I don't know if it is python or the underlying stuff.

I use fcntl to unblock and generally mess around to solve it, but
I don't think my solution is portable.

It is a real issue though - you are not imagining the dragon, it is
right at the door...

- Hendrik

[Nick Craig-Wood]

Noah <no**@noah.orgwrote:

On Jun 7, 9:01 am, dmoore <[e-mail]:

popen and friends will never do what you want it to do. Down that path
lies bitter disappointment.
You need pseduo-ttys and non-blocking IO. I don't know how to do this
on Windows, but I know it's possible.
Cygwin does it.

Anybody have any thoughts on this? Do I have my story straight? (the
popen variants can't handle this case and there are no other
alternatives in the standard python distro) Is there some place I can
submit this as a feature request? (Python dev?)

Try Pexpect http://pexpect.sourceforge.net/
It's been around for a long time and is quite complete and stable.

The catch is that it's UNIX-only. If you want to tease out the magic
code from wxProcess that
does non-blocking reads on win32 then I'd be happy to integrate that
into the current development branch
of Pexpect.

Windows has a really strange idea of non-blocking IO - it uses
something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
flag when you create the file/pipe. You then pass in overlap buffers
for reading writing.

I implemented this for serial ports in C a while ago. If you look at
the code in pyserial (in serialwin32.py) you'll see a very similar
implementation of non blocking IO. I assume that the same things will
work for pipes, but I've only direct experience with serial ports!

I'm sure this is feasible without any C extensions -- it might require
some ctypes hacking.

pyserial uses win32file and win32event

I know Windows has pretty decent async IO, but I don't know what they
have as an equivalent for a pty.
Maybe it isn't necessary. A pty is only necessary on UNIX because the
standard c library, stdio, behaves
differently when it's talking to a plain pipe versus a terminal -- it
switches buffering
between block and line oriented buffer. You don't want block buffering
on interactive applications.

Pty's probably aren't needed on Windows.

BTW I'd love to see pexpect working on windows and also in the
standard library. It is the proper answer to controlling other
interactive processes IMHO.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[dmoore]

On Jun 8, 12:30 pm, Nick Craig-Wood <[e-mail]:

Windows has a really strange idea of non-blocking IO - it uses
something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
flag when you create the file/pipe. You then pass in overlap buffers
for reading writing.

the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
is implicit in the calls they make)

take a look at:
http://cvs.wxwidgets.org/viewcvs.cgi...viewcvs-markup

a reasonably well-documented wxExecute function handles all the messy
win32 api calls and wraps the process in a wxProcess object and the
streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
function)

[Nick Craig-Wood]

dmoore <da**********@gmail.comwrote:

On Jun 8, 12:30 pm, Nick Craig-Wood <[e-mail]:

Windows has a really strange idea of non-blocking IO - it uses
something called overlapped io. You or in the FILE_FLAG_OVERLAPPED
flag when you create the file/pipe. You then pass in overlap buffers
for reading writing.

the wx guys appear to do it differently (unless FILE_FLAG_OVERLAPPED
is implicit in the calls they make)

take a look at:
http://cvs.wxwidgets.org/viewcvs.cgi...viewcvs-markup

a reasonably well-documented wxExecute function handles all the messy
win32 api calls and wraps the process in a wxProcess object and the
streams in wxInputStream / wxOutputStream (also see the wxExecuteDDE
function)

You are right, named pipes seem to have a non blocking mode.

Here is the lowdown from
http://msdn2.microsoft.com/en-US/library/aa365605.aspx

Both pipe clients and pipe servers can change a pipe handle's wait
mode by specifying either PIPE_WAIT or PIPE_NOWAIT in a call to the
SetNamedPipeHandleState function.

Note The nonblocking-wait mode is supported for compatibility with
Microsoft® LAN Manager version 2.0. This mode should not be used to
achieve overlapped input and output (I/O) with named
pipes. Overlapped I/O should be used instead, because it enables
time-consuming operations to run in the background after the
function returns. For more information about overlapped I/O, see
Synchronous and Overlapped Input and Output.

So it has a nonblocking mode but you shouldn't use it!

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick