473,396 Members | 1,804 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,396 software developers and data experts.

Trouble with popen2

I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/pyt...ly/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----
This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that.
Jul 18 '05 #1
4 1618
Rembrandt Q Einstein wrote:
I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/pyt...ly/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----
This hangs in the childerr.readlines(). Is it blocking? If so, why? In
any case, how can I be sure to read ALL data from BOTH stderr and stdout
and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that.


It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120
----

The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.
Jul 18 '05 #2
Rembrandt Q Einstein wrote:
Rembrandt Q Einstein wrote:
I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.

Here is a very well put (by someone else) posting that describes some
background:

http://mail.python.org/pipermail/pyt...ly/230837.html

I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires.
However, I need to also read stderr, so I made this sample writer and
runner:

-------
#!/usr/bin/python

import sys

for line in range(0, int(sys.argv[1])):
print "X" * 120

print >>sys.stderr, "hi"
-------
#!/usr/bin/python

import popen2

f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----
This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines()
statements, it works. But I don't want to use that until I understand
why. I suspect it's a race condition and I don't want to rely on that.

It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120
----

The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.


Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:

while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah

However, I think I might still miss some data this way.
Jul 18 '05 #3
On Mon, 27 Sep 2004 15:27:56 -0400, Rembrandt Q Einstein <hercules.rockefeller@springfield.??.us> wrote:
The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.


Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:

while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah

However, I think I might still miss some data this way.


Nope, this should work, assuming that you catch the output of select,
and act accordingly.
(at least, that is my experience)

Not sure why you need the f.poll() though, I think you can do without.

Maybe an extension to the documentation of popen would be in order, this
post pops up quite often here.

--
Albert
--
Unlike popular belief, the .doc format is not an open publically available format.
Jul 18 '05 #4
P
Rembrandt Q Einstein wrote:
I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.


This should be a FAQ if it's not.
Have a go with: http://www.pixelbeat.org/libs/subProcess.py

Pádraig.
Jul 18 '05 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Guy | last post by:
Hi I was given an exstremly useful answer to a problem that I had in windows. I'm relativly new with python, but find it exstremly useful. I'm creating a script to run test files and record...
2
by: Diez B. Roggisch | last post by:
Hi, I'm using the popen2 module to communicate with the crm114 text classificator. First I used to create a subprocess, feed the text to classify to it, close the crm's stdin file and read the...
1
by: A. Lloyd Flanagan | last post by:
OK, I've got a weird one I haven't been able to figure out yet. Admittedly I haven't had time to dig into the library source, but this behavior certainly doesn't seem right. Here's a test case: ...
1
by: | last post by:
This could possibly be a bug, but I don't understand it fully so I'm posting here first. Searching the list told me other people are having this problem too. I have created a class which...
1
by: Vivien Mallet | last post by:
Hello, I use popen2.Popen4 and I experienced problems with it. Let me show you with an example. The following script is called "lines" (it prints lines of 'X'): ---------------------...
1
by: Magnus Lycka | last post by:
I'm trying to read standard out in a process started with popen2 in a non-blocking way. (Other good ways of doing this than the one I tried are appreciated.) I've tried to dumb down my code to...
9
by: Martin P. Hellwig | last post by:
Hi all, I was doing some popen2 tests so that I'm more comfortable using it. I wrote a little python script to help me test that (testia.py): --------------------------------- someline =...
3
by: mikem76 | last post by:
How do I automatically redirect stdout and stderr when using os.popen2 to start a long running process. If the process prints a lot of stuff to stdout it will eventually stop because it runs out...
3
by: Daniel Klein | last post by:
Here's a c routine that prints a single line : #include <stdio.h> main() { printf ("Hello World!\n"); } And now the Python program (called 'po.py') that uses 'popen2' :
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.