popen2.Popen4: "wait()" hangs on

[Vivien Mallet]

Hello,

I use popen2.Popen4 and I experienced problems with it. Let me show you with
an example. The following script is called "lines" (it prints lines of
'X'):

---------------------
#!/usr/bin/env python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120
---------------------

In a Python console, I launch it:

---------------------
Python 2.3.4 (#2, Jul 5 2004, 09:15:05)
[GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2

import popen2
f = popen2.Popen4("lines 50000")
f.wait() ---------------------

"f.wait()" doesn't return anything. It just hangs on. "f.poll()" returns -1
all the time, even if "lines 50000" should end within one or two seconds.

My guess is that too many characters are sent to the standard output. I
found a way to deal with this problem:

---------------------
Python 2.3.4 (#2, Jul 5 2004, 09:15:05)
[GCC 3.3.4 (Debian 1:3.3.4-2)] on linux2 import popen2
f = popen2.Popen4("lines 50000")
lines = f.fromchild.readlines()
f.wait()

0
---------------------

It works! But it doesn't work in all cases. It happened to me that
"f.fromchild.readlines()" returns lines before the end of process:
"f.wait()" is then called before the end of the process and it may hang
on...

Finally, I usually write:
---------------------
lines = []
while (f.poll() == -1):
lines += f.fromchild.readlines()
---------------------
which seems to work in any case.

Is my work-around correct? Is there another one?
And why doesn't "wait()" work well? I suspect problems with full buffers or
unflushed buffers, but I am not sure.

Thank you for your help,
Vivien.

[Christopher T King]

On Mon, 26 Jul 2004, Vivien Mallet wrote:

"f.wait()" doesn't return anything. It just hangs on. "f.poll()" returns -1
all the time, even if "lines 50000" should end within one or two seconds.

My guess is that too many characters are sent to the standard output. I
found a way to deal with this problem:

Your diagnosis is nearly correct. The actual reason is that "lines" is
trying to write output, but since nothing is consuming the output, it is
forced to block on its 'print' statements. Because f.wait() is waiting
for "lines" to finish, it will keep blocking until "lines" is able to
write its output somewhere.

The solutions you give are quite correct; they consume the output before
waiting for the process to complete (I might also suggest using select()
in place of .poll(), though). What is it you're actually trying to
accomplish, though? If you're just trying to run a process in parallel,
but ignore its output, one of the spawn() family of calls might work
better (these will send the output to the same place as your main
program). If you want to discard the output entirely, then you can use a
combination of fork() and one of the exec() calls, redirecting sys.stdout
to /dev/null in the process.

Hope this helps.