I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some
background: http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:
-------
#!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])):
print "X" * 120
print >>sys.stderr, "hi"
-------
#!/usr/bin/python
import popen2
f = popen2.Popen3("./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.readlines()
outs += f.fromchild.readlines()
-----
This hangs in the childerr.readlines(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that. 4 1617
Rembrandt Q Einstein wrote: I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some background:
http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround (independently discovered by me) does do the job he requires. However, I need to also read stderr, so I made this sample writer and runner:
------- #!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])): print "X" * 120
print >>sys.stderr, "hi" ------- #!/usr/bin/python
import popen2
f = popen2.Popen3("./writer.py 50000", True) outs = [] errs = [] while (f.poll() == -1): errs += f.childerr.readlines() outs += f.fromchild.readlines() -----
This hangs in the childerr.readlines(). Is it blocking? If so, why? In any case, how can I be sure to read ALL data from BOTH stderr and stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements, it works. But I don't want to use that until I understand why. I suspect it's a race condition and I don't want to rely on that.
It's possible that when I have child.readlines() first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.
My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:
-------------
#!/usr/bin/python
import sys
import math
for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120
----
The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.
Rembrandt Q Einstein wrote: Rembrandt Q Einstein wrote:
I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some background:
http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround (independently discovered by me) does do the job he requires. However, I need to also read stderr, so I made this sample writer and runner:
------- #!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])): print "X" * 120
print >>sys.stderr, "hi" ------- #!/usr/bin/python
import popen2
f = popen2.Popen3("./writer.py 50000", True) outs = [] errs = [] while (f.poll() == -1): errs += f.childerr.readlines() outs += f.fromchild.readlines() -----
This hangs in the childerr.readlines(). Is it blocking? If so, why? In any case, how can I be sure to read ALL data from BOTH stderr and stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements, it works. But I don't want to use that until I understand why. I suspect it's a race condition and I don't want to rely on that.
It's possible that when I have child.readlines() first, it consumes all 50000 output lines and then err.readlines() grabs the "hi" and it just exists. When I have them the other way, err.readlines() blocks (I thought readlines() was non-blocking, though) and it just hangs.
My real external program isn't nice enough to order the output like that, so here's a more realistic writer:
------------- #!/usr/bin/python
import sys import math
for i in range(0, int(sys.argv[1])): if math.fmod(i, 100) == 0: print >>sys.stderr, "hi" print >>sys.stdout, "X" * 120 ----
The challenge is to write a program that will run this one in a subprocess and read all of both stderr and stdout.
Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:
while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah
However, I think I might still miss some data this way.
On Mon, 27 Sep 2004 15:27:56 -0400, Rembrandt Q Einstein <hercules.rockefeller@springfield.??.us> wrote: The challenge is to write a program that will run this one in a subprocess and read all of both stderr and stdout.
Ah--despite my new nick, I am an idiot. I should just use select() on fromchild and childerr. Something like this:
while f.poll() == -1: select([], [f.fromchild, f.childerr],[], 0) blah
However, I think I might still miss some data this way.
Nope, this should work, assuming that you catch the output of select,
and act accordingly.
(at least, that is my experience)
Not sure why you need the f.poll() though, I think you can do without.
Maybe an extension to the documentation of popen would be in order, this
post pops up quite often here.
--
Albert
--
Unlike popular belief, the .doc format is not an open publically available format.
Rembrandt Q Einstein wrote: I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
This should be a FAQ if it's not.
Have a go with: http://www.pixelbeat.org/libs/subProcess.py
Pádraig. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics
by: Guy |
last post by:
Hi
I was given an exstremly useful answer to a problem that I had in
windows. I'm relativly new with python, but find it exstremly useful.
I'm creating a script to run test files and record...
|
by: Diez B. Roggisch |
last post by:
Hi,
I'm using the popen2 module to communicate with the crm114 text
classificator. First I used to create a subprocess, feed the text to
classify to it, close the crm's stdin file and read the...
|
by: A. Lloyd Flanagan |
last post by:
OK, I've got a weird one I haven't been able to figure out yet.
Admittedly I haven't had time to dig into the library source, but this
behavior certainly doesn't seem right. Here's a test case:
...
|
by: |
last post by:
This could possibly be a bug, but I don't understand it fully
so I'm posting here first. Searching the list told me other
people are having this problem too.
I have created a class which...
|
by: Vivien Mallet |
last post by:
Hello,
I use popen2.Popen4 and I experienced problems with it. Let me show you with
an example. The following script is called "lines" (it prints lines of
'X'):
---------------------...
|
by: Magnus Lycka |
last post by:
I'm trying to read standard out in a process started with popen2
in a non-blocking way. (Other good ways of doing this than the
one I tried are appreciated.)
I've tried to dumb down my code to...
|
by: Martin P. Hellwig |
last post by:
Hi all,
I was doing some popen2 tests so that I'm more comfortable using it.
I wrote a little python script to help me test that (testia.py):
---------------------------------
someline =...
|
by: mikem76 |
last post by:
How do I automatically redirect stdout and stderr when using os.popen2
to start a long running process. If the process prints a lot of stuff
to stdout it will eventually stop because it runs out...
|
by: Daniel Klein |
last post by:
Here's a c routine that prints a single line :
#include <stdio.h>
main()
{
printf ("Hello World!\n");
}
And now the Python program (called 'po.py') that uses 'popen2' :
|
by: taylorcarr |
last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
|
by: Charles Arthur |
last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
|
by: aa123db |
last post by:
Variable and constants
Use var or let for variables and const fror constants.
Var foo ='bar';
Let foo ='bar';const baz ='bar';
Functions
function $name$ ($parameters$) {
}
...
|
by: ryjfgjl |
last post by:
If we have dozens or hundreds of excel to import into the database, if we use the excel import function provided by database editors such as navicat, it will be extremely tedious and time-consuming...
|
by: ryjfgjl |
last post by:
In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...
|
by: emmanuelkatto |
last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud.
Please let me know.
Thanks!
Emmanuel
|
by: BarryA |
last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
|
by: Sonnysonu |
last post by:
This is the data of csv file
1 2 3
1 2 3
1 2 3
1 2 3
2 3
2 3
3
the lengths should be different i have to store the data by column-wise with in the specific length.
suppose the i have to...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
| |