subprocess -popen - reading stdout from child - hangs

[gregpinero]

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

>>proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=s ubprocess.PIPE)
proc.stdout.read()

But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Thanks,

Greg

[gregpinero]

On Sep 22, 11:28 pm, "[e-mail]" <[e-mail]>
wrote:

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

>proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=s ubprocess.PIPE)
proc.stdout.read()

But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Well, using this subclass of subprocess.Popen fixes the problem
(http://aspn.activestate.com/ASPN/Coo.../Recipe/440554)

I don't understand how it works though. Would anyone mind
explaining? I'm thinking there's something fundamental about Unix
processes I'm not understanding.

-Greg

[Karthik Gurusamy]

On Sep 22, 8:28 pm, "[e-mail]" <[e-mail]>
wrote:

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()

Add sys.stdout.close()

while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

>proc = subprocess.Popen('python /home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=s ubprocess.PIPE)
proc.stdout.read()

But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Since your loop.py is still alive and hasn't closed its stdout, the
caller continues to wait for EOF (it doesn't know if loop.py is done
generating all its output)

Karthik

>
Thanks,

Greg

[Lawrence D'Oliveiro]

In message <11**********************@19g2000hsx.googlegroups. com>,
gr********@gmail.com wrote:

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

>>>proc = subprocess.Popen('python
/home/chiefinnovator/loop.py',shell=True,stdin=subprocess.PIPE,stdout=s ubprocess.PIPE)
proc.stdout.read()

But it just hangs at read()

That's because you didn't tell it how much to read, so by default it tries
to read until EOF <http://docs.python.org/lib/bltin-file-objects.html>. But
since the subprocess is still running and hasn't closed its stdout, the
pipe has not reached EOF.

[Lawrence D'Oliveiro]

In message <11*********************@d55g2000hsg.googlegroups. com>,
gr********@gmail.com wrote:

Well, using this subclass of subprocess.Popen fixes the problem
(http://aspn.activestate.com/ASPN/Coo.../Recipe/440554)

I don't understand how it works though. Would anyone mind
explaining? I'm thinking there's something fundamental about Unix
processes I'm not understanding.

The "select" calls are checking that the subprocess is ready to read before
writing to its stdin, or that it has something to write before reading from
its stdout.

The O_NONBLOCK fcntl simply says not to wait if there is nothing more to be
read, just return what was already read.

[gregpinero]

On Sep 23, 2:58 am, Karthik Gurusamy <[e-mail]:

On Sep 22, 8:28 pm, "[e-mail]" <[e-mail]>
wrote:

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()

Add sys.stdout.close()

Adding sys.stdout.close() and removing sys.stdout.flush() seems to
make it work. But can the while loop still use sys.stdout later on?
Do I have to reopen it?

Thanks,

Greg

[Karthik Gurusamy]

On Sep 24, 2:22 pm, "[e-mail]" <[e-mail]>
wrote:

On Sep 23, 2:58 am, Karthik Gurusamy <[e-mail]:

On Sep 22, 8:28 pm, "[e-mail]" <[e-mail]>
wrote:

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()

Add sys.stdout.close()

Adding sys.stdout.close() and removing sys.stdout.flush() seems to
make it work. But can the while loop still use sys.stdout later on?
Do I have to reopen it?

Once you close a file-object, you cannot use it. You'll get exception
if you try.
I quickly tried the fdopen on id 1 (stdout in unix like OS) and it
seems to work

>>import sys
sys.stdout.close()
import os
print 'hi'

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file

>>sys.stdout=os.fdopen(1, 'w')
print 'hi'

hi

>>>

BUT you may want to re-think your design. Note that your caller
process will anyway stop reading further from your loop.py process
the moment it sees the "first" EOF. So even if you enable loop.py to
generate more output (thru' the above fdopen), the caller process is
not going to see the newly generated data.

If your requirement is to collect all output from loop.py then you
can't do it if loop.py has an infinite loop across its data generation
points (ie it can generate data after the infinite loop -- which
anyway doesn't sense).

If all you want is not to get blocked, try one of the select solutions
or read a small amount at a time (which you can guarantee to be
available). Yet another solution would be you could set up an alarm
and get out of the blocking read if the alarm fires.

Karthik

>
Thanks,

Greg