I am running an external command and I need to know a) when it is done
and b) what it wrote to both stdout and stderr. After a little
searching, I found the popen2 module and used the Popen3 class. I'm
having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some
background: http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround
(independently discovered by me) does do the job he requires. However,
I need to also read stderr, so I made this sample writer and runner:
-------
#!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])):
print "X" * 120
print >>sys.stderr, "hi"
-------
#!/usr/bin/python
import popen2
f = popen2.Popen3(" ./writer.py 50000", True)
outs = []
errs = []
while (f.poll() == -1):
errs += f.childerr.read lines()
outs += f.fromchild.rea dlines()
-----
This hangs in the childerr.readli nes(). Is it blocking? If so, why?
In any case, how can I be sure to read ALL data from BOTH stderr and
stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements,
it works. But I don't want to use that until I understand why. I
suspect it's a race condition and I don't want to rely on that. 4 1640
Rembrandt Q Einstein wrote: I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some background:
http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround (independently discovered by me) does do the job he requires. However, I need to also read stderr, so I made this sample writer and runner:
------- #!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])): print "X" * 120
print >>sys.stderr, "hi" ------- #!/usr/bin/python
import popen2
f = popen2.Popen3(" ./writer.py 50000", True) outs = [] errs = [] while (f.poll() == -1): errs += f.childerr.read lines() outs += f.fromchild.rea dlines() -----
This hangs in the childerr.readli nes(). Is it blocking? If so, why? In any case, how can I be sure to read ALL data from BOTH stderr and stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements, it works. But I don't want to use that until I understand why. I suspect it's a race condition and I don't want to rely on that.
It's possible that when I have child.readlines () first, it consumes all
50000 output lines and then err.readlines() grabs the "hi" and it just
exists. When I have them the other way, err.readlines() blocks (I
thought readlines() was non-blocking, though) and it just hangs.
My real external program isn't nice enough to order the output like
that, so here's a more realistic writer:
-------------
#!/usr/bin/python
import sys
import math
for i in range(0, int(sys.argv[1])):
if math.fmod(i, 100) == 0:
print >>sys.stderr, "hi"
print >>sys.stdout, "X" * 120
----
The challenge is to write a program that will run this one in a
subprocess and read all of both stderr and stdout.
Rembrandt Q Einstein wrote: Rembrandt Q Einstein wrote:
I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
Here is a very well put (by someone else) posting that describes some background:
http://mail.python.org/pipermail/pyt...ly/230837.html
I came to a similar conclusion as that poster and his workaround (independently discovered by me) does do the job he requires. However, I need to also read stderr, so I made this sample writer and runner:
------- #!/usr/bin/python
import sys
for line in range(0, int(sys.argv[1])): print "X" * 120
print >>sys.stderr, "hi" ------- #!/usr/bin/python
import popen2
f = popen2.Popen3(" ./writer.py 50000", True) outs = [] errs = [] while (f.poll() == -1): errs += f.childerr.read lines() outs += f.fromchild.rea dlines() -----
This hangs in the childerr.readli nes(). Is it blocking? If so, why? In any case, how can I be sure to read ALL data from BOTH stderr and stdout and not be in danger of hanging?
PS: I just found that if I swap the order of the readlines() statements, it works. But I don't want to use that until I understand why. I suspect it's a race condition and I don't want to rely on that.
It's possible that when I have child.readlines () first, it consumes all 50000 output lines and then err.readlines() grabs the "hi" and it just exists. When I have them the other way, err.readlines() blocks (I thought readlines() was non-blocking, though) and it just hangs.
My real external program isn't nice enough to order the output like that, so here's a more realistic writer:
------------- #!/usr/bin/python
import sys import math
for i in range(0, int(sys.argv[1])): if math.fmod(i, 100) == 0: print >>sys.stderr, "hi" print >>sys.stdout, "X" * 120 ----
The challenge is to write a program that will run this one in a subprocess and read all of both stderr and stdout.
Ah--despite my new nick, I am an idiot. I should just use select() on
fromchild and childerr. Something like this:
while f.poll() == -1:
select([], [f.fromchild, f.childerr],[], 0)
blah
However, I think I might still miss some data this way.
On Mon, 27 Sep 2004 15:27:56 -0400, Rembrandt Q Einstein <hercules.rocke feller@springfi eld.??.us> wrote: The challenge is to write a program that will run this one in a subprocess and read all of both stderr and stdout.
Ah--despite my new nick, I am an idiot. I should just use select() on fromchild and childerr. Something like this:
while f.poll() == -1: select([], [f.fromchild, f.childerr],[], 0) blah
However, I think I might still miss some data this way.
Nope, this should work, assuming that you catch the output of select,
and act accordingly.
(at least, that is my experience)
Not sure why you need the f.poll() though, I think you can do without.
Maybe an extension to the documentation of popen would be in order, this
post pops up quite often here.
--
Albert
--
Unlike popular belief, the .doc format is not an open publically available format.
Rembrandt Q Einstein wrote: I am running an external command and I need to know a) when it is done and b) what it wrote to both stdout and stderr. After a little searching, I found the popen2 module and used the Popen3 class. I'm having trouble with it hanging, though.
This should be a FAQ if it's not.
Have a go with: http://www.pixelbeat.org/libs/subProcess.py
Pádraig. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Guy |
last post by:
Hi
I was given an exstremly useful answer to a problem that I had in
windows. I'm relativly new with python, but find it exstremly useful.
I'm creating a script to run test files and record output, the script
already works on win32 and I've made a telnet procedure which works
quite well when you want to do the testing on one of the many unix
boxes (but its very slow). What I want to do now is run the script on
a unix system, and later...
|
by: Diez B. Roggisch |
last post by:
Hi,
I'm using the popen2 module to communicate with the crm114 text
classificator. First I used to create a subprocess, feed the text to
classify to it, close the crm's stdin file and read the results. Easy
enough.
However, the crm114 is a slow beast, and I try to increase its performanc by
using it in the "window"-mode. That means that the text fed to the
subprocess looks like this:
|
by: A. Lloyd Flanagan |
last post by:
OK, I've got a weird one I haven't been able to figure out yet.
Admittedly I haven't had time to dig into the library source, but this
behavior certainly doesn't seem right. Here's a test case:
"""Test program to demonstrate problem with popen2 module."""
import popen2
def main (argv):
mycmd = 'python2.3 -c "for i in range(100000):\n print i"'
p = popen2.Popen3(mycmd)
|
by: |
last post by:
This could possibly be a bug, but I don't understand it fully
so I'm posting here first. Searching the list told me other
people are having this problem too.
I have created a class which inherits popen2.Popen3.
The problem is that popen2.Popen3 does not define a __del__
method to handle proper closing of the pipes or destruction of
the process. So I created one in my class:
|
by: Vivien Mallet |
last post by:
Hello,
I use popen2.Popen4 and I experienced problems with it. Let me show you with
an example. The following script is called "lines" (it prints lines of
'X'):
---------------------
#!/usr/bin/env python
import sys
| |
by: Magnus Lycka |
last post by:
I'm trying to read standard out in a process started with popen2
in a non-blocking way. (Other good ways of doing this than the
one I tried are appreciated.)
I've tried to dumb down my code to see what happens, and
socket.poll seems to behave very strangely. I've tried to
use the .poll method for the poll object with and without
a timeout, but in either case, the output randomly switches
between on of the versions below. It runs fast,...
|
by: Martin P. Hellwig |
last post by:
Hi all,
I was doing some popen2 tests so that I'm more comfortable using it.
I wrote a little python script to help me test that (testia.py):
---------------------------------
someline = raw_input("something:")
if someline == 'test':
print("yup")
|
by: mikem76 |
last post by:
How do I automatically redirect stdout and stderr when using os.popen2
to start a long running process. If the process prints a lot of stuff
to stdout it will eventually stop because it runs out of buffer space.
Once I start reading the stdout file returned by os.popen2 then the
process resumes. I know that I could just specify > /dev/null when
starting the process but I'd like to know if there is a way to start a
process using os.popen2...
|
by: Daniel Klein |
last post by:
Here's a c routine that prints a single line :
#include <stdio.h>
main()
{
printf ("Hello World!\n");
}
And now the Python program (called 'po.py') that uses 'popen2' :
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |