os.popen3 hangs in Windows XP SP1, SP2. Python 2.5 & 2.4. Consistent test case.

[Pierre Rouleau]

Hi all,

I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.

I stumbled on a case where a piece of code works in some occasions and
hangs consistently given different data. It performs exactly the same
on 3 different computers running Windows:

- computer 1: XP, SP1, Python 2.4.3
- computer 2: XP, SP2, Python 2.4.3
- computer 3: XP, SP2, Python 2.5

I know that a bug
(http://sourceforge.net/tracker/index...70&atid=105470)

related to popen3 hanging was opened and closed as 'not-a-bug'.
However I think that the test case I have is not a application problem
(a deadlock) and is a true problem with popen3() under Windows (but I
could be wrong.)

What I have is:

- a test script: tpopen.py.
- It uses nosetests.py to execute unit tests: it runs:
- test_roman.py, a test script for :
- roman.py, a modified version of Mark Pilgrims's roman.py
module, I used to test nosetests.
#--------------
The test case I have is a short program that invokes nosetests to run a
test_roman.py:

- The script topen.py is executed on the command line with::

topen test_roman

- topen.py executes

stdin, stdout, stderr = os.popen3('nosetests --nocapture
test_roman')

- The program runs properly if it then executes the following code
right after the call to os.popen3()::

stderr_lines = list(stderr)
stdin.close()
stderr.close()
pgm_exit_code = stdout.close() or 0

- The program hangs when it tries to read stdout (**but only ** when it
is running nosetests for the test_roman.py). The code following the
call to os.popen3() is::

for line in stdout:
sys.stdout.write(line) # prints almost all nosetests
stdout lines except for some at the end!
# then it HANGS.
stderr_lines = list(stderr)
stdin.close()
stderr.close()
pgm_exit_code = stdout.close() or 0
Note that it also hangs if topen.py executes:

stdin, stdout, stderr = os.popen3('nosetests test_roman')

Or if I modify the print statements inside test_roman.py, the unit test
scrip executed by nosetests.py.
I also tried changing the order of the closing of the stream (like
doing stdin.close() right after the popen3() call). No impact.

Note that popen3() call does not hang if topen runs other unit test
scripts i have.

#---------------------

QUESTIONS:
- Can any one see a problem with the order of execution that could
cause a deadlock here (topen.py code posted below)?
- Should I post a new Python bug, providing the complete test code or,
re-open the above mentioned bug?


# tpopen.py
---------------------------------------------------------------

import sys
import os

#
-----------------------------------------------------------------------------
# p o p e n 3 ( ) -- Execute a popen3 command --
# ^^^^^^^^^^^^^^^
#
def popen3(command,
stdout_parser=None,
stderr_parser=None,
default_stdout_value=None,
default_stderr_value=None) :

pgm_exit_code = 0
stdout_value, stderr_value = default_stdout_value,
default_stderr_value
stdin, stdout, stderr = os.popen3(command)
stdin.close()
if stdout_parser:
stdout_value = stdout_parser(stdout)
if stderr_parser:
stderr_value = stderr_parser(stderr)
stderr.close()
pgm_exit_code = stdout.close() or 0
return (stdout_value, stderr_value, pgm_exit_code)
#
-----------------------------------------------------------------------------

cmd1 = 'nosetests ' + sys.argv[1]
cmd2 = 'nosetests --nocapture ' + sys.argv[1]

def print_stream(stream):
for line in stream:
sys.stdout.write(line)

print '--------- CMD 1 , no stdout print : does not hang --------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, None, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code
print '--------- CMD 2 : HANGS after printing almost all stdout -'
stdout, stderr_lines, pgm_exit_code = popen3(cmd2, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code
print '--------- CMD 1 , stdout print : HANGS -------------------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code

#
-----------------------------------------------------------------------------

--

Pierre Rouleau

[Thomas Guettler]

Pierre Rouleau wrote:

Hi all,

I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.

I had a problem like this some time ago. But it was a problem that
would happen under any operating system.

If there is output on stdout and stderr, you will get a dead lock sooner
or later.

Example: The childprocess tries to write to stderr, and the
parent process reads from stdin. The buffer of stderr will
get full. The child will block. The parent will wait for ever.

See http://docs.python.org/lib/popen2-flow-control.html

My hint: Always use popen4

You can get dead locks with popen4, too. But only if you
write to pipe.tochild.

Thomas

--
Thomas Güttler, http://www.thomas-guettler.de/ http://www.tbz-pariv.de/
E-Mail: guettli (*) thomas-guettler + de
Spam Catcher: ni**************@thomas-guettler.de

[Pierre Rouleau]

On Dec 12, 10:11 am, Thomas Guettler
<[e-mail]:

Pierre Rouleau wrote:

Hi all,

I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.I had a problem like this some time ago. But it was a problem that

would happen under any operating system.

If there is output on stdout and stderr, you will get a dead lock sooner
or later.

Example: The childprocess tries to write to stderr, and the
parent process reads from stdin. The buffer of stderr will
get full. The child will block. The parent will wait for ever.

Seehttp://docs.python.org/lib/popen2-flow-control.html

My hint: Always use popen4

You can get dead locks with popen4, too. But only if you
write to pipe.tochild.

Thanks for replying Thomas.

The reason I was using popen3() is that I need to parse the error
stream and do something else with the stdout. My operation does not
need to have concurrent operations of the parent and child. I could
have used something file::

os.system('parent log_stdout.txt 2log_stderr.txt')

and then parse the 2 files. I just wanted to avoid using the files to
avoid having to have to deal with issues related to temporary file
names if there where several process instance of the code running
simultaneously.
Now, from the reading of the above link, this means that If I want to
be able to do what I want with pipes, avoiding deadlock means that:

- In the parent program, the code should be something that looks like:

stdin, stdout, stderr = os.popen3(command)
stderr_value = list(stderr)
stdout_value = list(stdout)
pgm_exit_code = stdout.close() or 0
stdin.close()
stderr.close()

An the above would work only if one stream is written by the child at a
time and stderr closed:

for whatever: print >sys.stderr, ' the error messages'
os.close(sys.stderr.fileno())
for someother: print 'other stdout info'

In my case, since I don't control the child program, I can assume that
it does not follow the required order. I am launching nosetests which
runs other test programs. I tried closing sys.stderr in the teardown
of my test script and that removed the deadlock but caused other
problems (because sys.stderr is used later by nosetests).

So, in the end, it looks like I really don't have any choice: if I want
to safely read both stdout and stderr in a way that is child program
agnostic: I must use temporary files (or maybe use something like
select). Right?

Thanks again!

--

Pierre Rouleau