473,837 Members | 1,910 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

os.popen3 hangs in Windows XP SP1, SP2. Python 2.5 & 2.4. Consistent test case.

Hi all,

I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.

I stumbled on a case where a piece of code works in some occasions and
hangs consistently given different data. It performs exactly the same
on 3 different computers running Windows:

- computer 1: XP, SP1, Python 2.4.3
- computer 2: XP, SP2, Python 2.4.3
- computer 3: XP, SP2, Python 2.5

I know that a bug
(http://sourceforge.net/tracker/index...70&atid=105470)

related to popen3 hanging was opened and closed as 'not-a-bug'.
However I think that the test case I have is not a application problem
(a deadlock) and is a true problem with popen3() under Windows (but I
could be wrong.)

What I have is:

- a test script: tpopen.py.
- It uses nosetests.py to execute unit tests: it runs:
- test_roman.py, a test script for :
- roman.py, a modified version of Mark Pilgrims's roman.py
module, I used to test nosetests.
#--------------
The test case I have is a short program that invokes nosetests to run a
test_roman.py:

- The script topen.py is executed on the command line with::

topen test_roman

- topen.py executes

stdin, stdout, stderr = os.popen3('nose tests --nocapture
test_roman')

- The program runs properly if it then executes the following code
right after the call to os.popen3()::

stderr_lines = list(stderr)
stdin.close()
stderr.close()
pgm_exit_code = stdout.close() or 0

- The program hangs when it tries to read stdout (**but only ** when it
is running nosetests for the test_roman.py). The code following the
call to os.popen3() is::

for line in stdout:
sys.stdout.writ e(line) # prints almost all nosetests
stdout lines except for some at the end!
# then it HANGS.
stderr_lines = list(stderr)
stdin.close()
stderr.close()
pgm_exit_code = stdout.close() or 0
Note that it also hangs if topen.py executes:

stdin, stdout, stderr = os.popen3('nose tests test_roman')

Or if I modify the print statements inside test_roman.py, the unit test
scrip executed by nosetests.py.
I also tried changing the order of the closing of the stream (like
doing stdin.close() right after the popen3() call). No impact.

Note that popen3() call does not hang if topen runs other unit test
scripts i have.

#---------------------

QUESTIONS:
- Can any one see a problem with the order of execution that could
cause a deadlock here (topen.py code posted below)?
- Should I post a new Python bug, providing the complete test code or,
re-open the above mentioned bug?


# tpopen.py
---------------------------------------------------------------

import sys
import os

#
-----------------------------------------------------------------------------
# p o p e n 3 ( ) -- Execute a popen3 command --
# ^^^^^^^^^^^^^^^
#
def popen3(command,
stdout_parser=N one,
stderr_parser=N one,
default_stdout_ value=None,
default_stderr_ value=None) :

pgm_exit_code = 0
stdout_value, stderr_value = default_stdout_ value,
default_stderr_ value
stdin, stdout, stderr = os.popen3(comma nd)
stdin.close()
if stdout_parser:
stdout_value = stdout_parser(s tdout)
if stderr_parser:
stderr_value = stderr_parser(s tderr)
stderr.close()
pgm_exit_code = stdout.close() or 0
return (stdout_value, stderr_value, pgm_exit_code)
#
-----------------------------------------------------------------------------

cmd1 = 'nosetests ' + sys.argv[1]
cmd2 = 'nosetests --nocapture ' + sys.argv[1]

def print_stream(st ream):
for line in stream:
sys.stdout.writ e(line)

print '--------- CMD 1 , no stdout print : does not hang --------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, None, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code
print '--------- CMD 2 : HANGS after printing almost all stdout -'
stdout, stderr_lines, pgm_exit_code = popen3(cmd2, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code
print '--------- CMD 1 , stdout print : HANGS -------------------'
stdout, stderr_lines, pgm_exit_code = popen3(cmd1, print_stream, list)
print 'stderr_lines: ', stderr_lines
print 'program exit code : ', pgm_exit_code

#
-----------------------------------------------------------------------------

--

Pierre Rouleau

Dec 11 '06 #1
2 2697
Pierre Rouleau wrote:
Hi all,

I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.
I had a problem like this some time ago. But it was a problem that
would happen under any operating system.

If there is output on stdout and stderr, you will get a dead lock sooner
or later.

Example: The childprocess tries to write to stderr, and the
parent process reads from stdin. The buffer of stderr will
get full. The child will block. The parent will wait for ever.

See http://docs.python.org/lib/popen2-flow-control.html

My hint: Always use popen4

You can get dead locks with popen4, too. But only if you
write to pipe.tochild.

Thomas

--
Thomas Güttler, http://www.thomas-guettler.de/ http://www.tbz-pariv.de/
E-Mail: guettli (*) thomas-guettler + de
Spam Catcher: ni************* *@thomas-guettler.de

Dec 12 '06 #2


On Dec 12, 10:11 am, Thomas Guettler
<guettli.use... @thomas-guettler.dewrot e:
Pierre Rouleau wrote:
Hi all,
I have a consistent test case where os.popen3() hangs in Windows. The
system hangs when retrieving the lines from the child process stdout.
I know there were several reports related to os.popen3() hanging under
Windows in this group before.I had a problem like this some time ago. But it was a problem that
would happen under any operating system.

If there is output on stdout and stderr, you will get a dead lock sooner
or later.

Example: The childprocess tries to write to stderr, and the
parent process reads from stdin. The buffer of stderr will
get full. The child will block. The parent will wait for ever.

Seehttp://docs.python.org/lib/popen2-flow-control.html

My hint: Always use popen4

You can get dead locks with popen4, too. But only if you
write to pipe.tochild.
Thanks for replying Thomas.

The reason I was using popen3() is that I need to parse the error
stream and do something else with the stdout. My operation does not
need to have concurrent operations of the parent and child. I could
have used something file::

os.system('pare nt log_stdout.txt 2log_stderr.txt ')

and then parse the 2 files. I just wanted to avoid using the files to
avoid having to have to deal with issues related to temporary file
names if there where several process instance of the code running
simultaneously.
Now, from the reading of the above link, this means that If I want to
be able to do what I want with pipes, avoiding deadlock means that:

- In the parent program, the code should be something that looks like:

stdin, stdout, stderr = os.popen3(comma nd)
stderr_value = list(stderr)
stdout_value = list(stdout)
pgm_exit_code = stdout.close() or 0
stdin.close()
stderr.close()

An the above would work only if one stream is written by the child at a
time and stderr closed:

for whatever: print >sys.stderr, ' the error messages'
os.close(sys.st derr.fileno())
for someother: print 'other stdout info'

In my case, since I don't control the child program, I can assume that
it does not follow the required order. I am launching nosetests which
runs other test programs. I tried closing sys.stderr in the teardown
of my test script and that removed the deadlock but caused other
problems (because sys.stderr is used later by nosetests).

So, in the end, it looks like I really don't have any choice: if I want
to safely read both stdout and stderr in a way that is child program
agnostic: I must use temporary files (or maybe use something like
select). Right?

Thanks again!

--

Pierre Rouleau

Dec 12 '06 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
3115
by: | last post by:
This could possibly be a bug, but I don't understand it fully so I'm posting here first. Searching the list told me other people are having this problem too. I have created a class which inherits popen2.Popen3. The problem is that popen2.Popen3 does not define a __del__ method to handle proper closing of the pipes or destruction of the process. So I created one in my class:
13
4464
by: Russell E. Owen | last post by:
I'm trying to launch an application from Python 2.3 on Windows. The application is "ds9" (an image viewer), and is installed in C:\Program Files\ds9\ds9 On unix I just do: os.popen3("ds9") and close the returned files and all is good. (I'm not trying to communicate with the program via popen3 and so had been using os.spawnlp, but that doesn't exist on Windows.)
1
1728
by: Kenneth Pronovici | last post by:
Hi, I have a problem with a popen2 pipe hanging partway through execution of a command. This happens in my function executeCommand() that is used for every "shell" command execution in my program. Source code for this function is below. My development environment is Debian unstable with Python 2.3.5. Within executeCommand(), output of every executed command is always logged using a Python logger. If outputFile is passed in, then...
2
4176
by: Jakob Simon-Gaarde | last post by:
Follow-up on a thread from 1999 (see below) Well now it is 2005 and the operating system I'm using is Windows Server 2003, and I can still see that the same problem persists with: win32pipe.popen2() win32pipe.popen3() win32pipe.popen4() while win32pipe.popen() does almost what you want.
1
1325
by: Fuzzyman | last post by:
Hello all, I may well post this a a bug on Monday (after testing with Python 2.3) - but I thought I'd post here to see if anyone has any ideas. The basic problem is that under Python 2.4 (and windoze XP SP2) CGIHTTPServer isn't passing the CGI environment variables to scripts it runs. I've checked that the environment variables all exist in os.environ
0
6751
by: Vijay | last post by:
Prep Courses for International Certifications, CSTE & CSQA & ISEB & ISTQB &Business Analyst & SOA Certifications in HYDERABAD. After receiving overwhelming response to our last 50+ batches, SPECTRAMIND SOLUTIONS now announces a new batch of Prep Courses for CSQA & CSTE& ISEB & ISTQB & Business Analyst & SOA so as to prepare you thoroughly for the most prestigious certification exams conducted by International organizations. We...
4
2598
by: dumbkiwi | last post by:
I have written a script that uses the urllib2 module to download web pages for parsing. If there is no network interface, urllib2 hangs for a very long time before it raises an exception. I have set the socket timeout with socket.setdefaulttimeout(), however, where there is no network interface, this seems to be ignored - presumably, because without a network interface, there is nothing for the socket module to interact with.
3
2051
by: Christoph Krammer | last post by:
Hello everybody, I need to get the different frames from a GIF image in my python script and want to use the giftopnm program from netpbm to get the frames and directly convert them to pnm files. I tried to use the following code: for image in images: if (image == 'GIF'): (si, so, se) = os.popen3('giftopnm -image=all', 'b')
1
2419
by: Christoph Krammer | last post by:
Hello everybody, I try to use an external OCR tool to convert some binary image data to text. The image is in one variable, the text should be converted to another. I use the following code: (si, so, se) = os.popen3('ocrad') si.write(frame) si.close() messagetext += so.read()
0
9682
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
10877
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10570
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
0
10273
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9402
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7001
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5670
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
1
4474
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
2
4041
muto222
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.