Dialog with a process via subprocess.Pope n blocks forever

[bayer.justin]

Hi,

I am trying to communicate with a subprocess via the subprocess
module. Consider the following example:

>>from subprocess import Popen, PIPE
Popen("""pyth on -c 'input("hey")'" "", shell=True)

<subprocess.Pop en object at 0x729f0>

>>hey

Here hey is immediately print to stdout of my interpreter, I did not
type in the "hey". But I want to read from the output into a string,
so I do

>>x = Popen("""python -c 'input("hey\n") '""", shell=True, stdout=PIPE, bufsize=2**10)
x.stdout.read (1)

# blocks forever

Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?

Regards,
-Justin

[Hendrik van Rooyen]

<ba**********@g ooglemail.comwr ote:

Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?

I think this problem lies deeper - there has been a lot of
complaints about blocking and data getting stuck in pipes
and sockets...

I have noticed that the Python file objects seem to be
inherently half duplex, but I am not sure if it is python
or the underlying OS. (Suse 10 in my case)

You can fix it by unblocking using the fcntl module,
but then all your accesses have to be in try - except
clauses.

It may be worth making some sort of FAQ on this
subject, as it appears from time to time.

The standard advice has been to use file.flush()
after file.write(), but if you are threading and
have called file.read(n), then the flushing does
not help - this is why I say that the file object
seems to be inherently half duplex.

It makes perfect sense, of course, if the file is a
real disk file, as you have to finish the read before
you can move the heads to do the write- but for
pipes, sockets and RS-232 serial lines it does not
make so much sense.

Does anybody know where it comes from -
Python, the various OSses, or C?

- Hendrik

[bayer.justin]

Hi,

Thanks for your answer. I had a look into the fcntl module and tried
to unlock the output-file, but

>>fcntl.lockf(x .stdout, fcntl.LOCK_UN)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 9] Bad file descriptor

I wonder why it does work with the sys.stdin It's really a pity, it's
the first time python does not work as expected. =/

Flushing the stdin did not help, too.

Regards,
-Justin

[Gabriel Genellina]

En Wed, 28 Feb 2007 18:27:43 -0300, <ba**********@g ooglemail.comes cribió:

Hi,

I am trying to communicate with a subprocess via the subprocess
module. Consider the following example:

>>>from subprocess import Popen, PIPE
Popen("""pyt hon -c 'input("hey")'" "", shell=True)

<subprocess.Pop en object at 0x729f0>

>>>hey

Here hey is immediately print to stdout of my interpreter, I did not
type in the "hey". But I want to read from the output into a string,
so I do

>>>x = Popen("""python -c 'input("hey\n") '""", shell=True, stdout=PIPE,
bufsize=2**1 0)
x.stdout.rea d(1)

# blocks forever

Blocks, or is the child process waiting for you to input something in
response?

Is it possible to read to and write to the std streams of a
subprocess? What am I doing wrong?

This works for me on Windows XP. Note that I'm using a tuple with
arguments, and raw_input instead of input (just to avoid a traceback on
stderr)

pyx=Popen(("pyt hon", "-c", "raw_input('hey ')"), shell=True, stdout=PIPE)
pyx.stdout.read (1)
1234
'h'
pyx.stdout.read ()
'ey'

I typed that 1234 (response to raw_input).

You may need to use python -u, or redirect stderr too, but what your real
problem is?

--
Gabriel Genellina

[bayer.justin]

Okay, here is what I want to do:

I have a C Program that I have the source for and want to hook with
python into that. What I want to do is: run the C program as a
subprocess.
The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.

BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

Thus a useful dialogue is not possible.

Regards,
-Justin

[George Trojan]

ba**********@go oglemail.com wrote:

Okay, here is what I want to do:

I have a C Program that I have the source for and want to hook with
python into that. What I want to do is: run the C program as a
subprocess.
The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.

BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

Thus a useful dialogue is not possible.

Regards,
-Justin

Have you considered using pexpect: http://pexpect.sourceforge.net/ ?

George

[Gabriel Genellina]

En Thu, 01 Mar 2007 14:42:00 -0300, <ba**********@g ooglemail.comes cribió:

BUT If I use PIPE for both (so I can .write() on the stdin and .read()
from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

On http://docs.python.org/lib/popen2-flow-control.html there are some
notes on possible flow control problems you may encounter.
If you have no control over the child process, it may be safer to use a
different thread for reading its output.

--
Gabriel Genellina

[Hendrik van Rooyen]

<ba**********@g ooglemail.comwr ote:

Hi,

Thanks for your answer. I had a look into the fcntl module and tried
to unlock the output-file, but

>fcntl.lockf(x. stdout, fcntl.LOCK_UN)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 9] Bad file descriptor

I wonder why it does work with the sys.stdin It's really a pity, it's
the first time python does not work as expected. =/

Flushing the stdin did not help, too.

its block, not lock, and one uses file.flush() after using file.write(),
so the stdin is the wrong side - you have to push, you can't pull..

Here is the unblock function I use - it comes from the internet,
possibly from this group, but I have forgotten who wrote it.

# Some magic to make a file non blocking - from the internet

def unblock(f):
"""Given file 'f', sets its unblock flag to true."""

fcntl.fcntl(f.f ileno(), fcntl.F_SETFL, os.O_NONBLOCK)

hope this helps - note that the f is not the file's name but the
thing you get when you write :

f = open(...

- Hendrik

[Hendrik van Rooyen]

<ba**********@g ooglemail.comwr ote:

8<------------------

The C programm gets its "commands" from its stdin and sends its state
to stdout. Thus I have some kind of dialog over stdin.

So, once I start the C Program from the shell, I immediately get its
output in my terminal. If I start it from a subprocess in python and
use python's sys.stdin/sys.stdout as the subprocess' stdout/stdin I
also get it immediately.

so why don't you just write to your stdout and read from your stdin?

>
BUT If I use PIPE for both (so I can .write() on the stdin and .read()

This confuses me - I assume you mean write to the c program's stdin?

from the subprocess' stdout stream (better: file descriptor)) reading
from the subprocess stdout blocks forever. If I write something onto
the subprocess' stdin that causes it to somehow proceed, I can read
from its stdout.

This sounds like the c program is getting stuck waiting for input...

>
Thus a useful dialogue is not possible.

If you are both waiting for input, you have a Mexican standoff...

And if you are using threads, and you have issued a .read() on
a file, then a .write() to the same file, even followed by a .flush()
will not complete until after the completion of the .read().

So in such a case you have to unblock the file, and do the .read() in
a try - except clause, to "free up" the "file driver" so that the .write()
can complete.

But I am not sure if this is in fact your problem, or if it is just normal
synchronisation hassles...

- Hendrik

[bayer.justin]

If you are both waiting for input, you have a Mexican standoff...

That is not the problem. The problem is, that the buffers are not
flushed correctly. It's a dialogue, so nothing complicated. But python
does not get what the subprocess sends onto the subprocess' standard
out - not every time, anyway.

I'm quite confused, but hopefully will understand what's going on and
come back here.