This has been driving me insane for the last hour or so. I have search
everywhere, and nothing works. I am trying to use the subprocess module
to run a program and get its output line by line. But, it always waits
for the process to terminate and then return the output all at once.

Can someone please show me some code that actually works for this sort
of thing? It doesn't even have to use the subprocess module. Don't
worry if the code isn't compatible with Windows. My program is targeted
at Linux/UNIX users.

Thanks!

cypher543 wrote:

This has been driving me insane for the last hour or so. I have search
everywhere, and nothing works. I am trying to use the subprocess module
to run a program and get its output line by line. But, it always waits
for the process to terminate and then return the output all at once.
>
Can someone please show me some code that actually works for this sort
of thing? It doesn't even have to use the subprocess module. Don't
worry if the code isn't compatible with Windows. My program is targeted
at Linux/UNIX users.
>
Thanks!
>

try:

Python 2.5c1 (r25c1:51305, Aug 17 2006, 17:07:04)
[GCC 3.3.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.

For more info on how to do stdin and other things check out:
http://docs.python.org/lib/module-subprocess.html

Hope this helps.

Adonis

cypher543 wrote:

This has been driving me insane for the last hour or so. I have search
everywhere, and nothing works. I am trying to use the subprocess module
to run a program and get its output line by line. But, it always waits
for the process to terminate and then return the output all at once.

Right, you need to use subprocess.PIPE and polling of the process and
the pipe to get it to do what you want. Hopefully this makes sense, I
also hope it'll work on Linux but can't imagine why it wouldn't.

Save this code to a file, name unimportant. Executing the file will
fire the top clause of the if statement, the bit that runs the "parent"
part. That starts the script again as a subprocess, which triggers the
else clause. Hope it makes sense.

Depending on your environment, you might need 'shell=True' in your Popen
args. On Windows, that is required if you want to prevent a console
window from popping up if you're running in a GUI app.

-tom!

--

import subprocess
import sys
import time

if len(sys.argv) == 1:
# no command line arg means, "we're the parent process."
print 'starting parent process.'
myName = sys.argv[0]

# launch the subprocess with an additional parameter, to trigger
# else clause below.
p = subprocess.Popen(
( 'python', '-u', myName, 'x' ),
stdout=subprocess.PIPE
)

while p.poll() == None:
data = p.stdout.readline()
if data:
print data,

print 'process ended with return code', p.returncode

else:
# assume the command-line arg means "run the background process"
print 'starting subprocess'

for i in range(10):
time.sleep(.25)
print i

sys.exit(14)

Tom Plunket wrote:

while p.poll() == None:
data = p.stdout.readline()
if data:
print data,

If you change that print to something more decorated, like,

print 'process said:', data,

Then it might be more obvious where/how the print statements are getting
routed.

Keep in mind that readline() will return one line at a time, and that
line will end with a newline, although the last line you get (at EOF)
may not have one.

again, good luck. Hope this helps,
-tom!

--

Thank you for the examples, but I have tried all of that before. No
matter what I do, my program always hangs while it waits for the
process to exit and then it prints all of the output at once. I've
tried read(), readline(), readlines(), communicate(), etc and it is
always the same.

self.buildPID = subprocess.Popen(["python", "tobeforked.py"], stdout =
subprocess.PIPE)
while self.buildPID.poll() == None:
output = self.buildPID.stdout.readline()
self.consoleLogBuffer.insert(self.consoleLogBuffer .get_end_iter(),
output)
self.consoleLog.scroll_to_mark(self.consoleLogBuff er.get_insert(), 0)

Keep in mind that I'm not required to use subprocess. But I have also
tried os.fork and the pty module. They both produce the exact same
results.

On Dec 29, 3:38 am, Tom Plunket <t...@fancy.orgwrote:

Tom Plunket wrote:

>
print 'process said:', data,
>
Then it might be more obvious where/how the print statements are getting
routed.
>
Keep in mind that readline() will return one line at a time, and that
line will end with a newline, although the last line you get (at EOF)
may not have one.
>
again, good luck. Hope this helps,
-tom!
>
--

cypher543 wrote:

Thank you for the examples, but I have tried all of that before.

Did you try my example specifically?

No matter what I do, my program always hangs while it waits for the
process to exit and then it prints all of the output at once.
>
self.buildPID = subprocess.Popen(["python", "tobeforked.py"], ...

By default, python will execute in buffered mode if it's attached to a
pipe. Start it with the '-u' command line option and you may be good to
go. (`python -h` for more info on this, man pages may also discuss it.)

-tom!

--

At Friday 29/12/2006 12:22, cypher543 wrote:

>Thank you for the examples, but I have tried all of that before. No
>matter what I do, my program always hangs while it waits for the
>process to exit and then it prints all of the output at once. I've
>tried read(), readline(), readlines(), communicate(), etc and it is
>always the same.

Did you *actually* tried what Tom Plunket posted? Two tiny chars make
a difference.


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

Gabriel Genellina wrote:

Did you *actually* tried what Tom Plunket posted? Two tiny chars make
a difference.

The sad irony is that before taking off for vacation I was struggling at
work with the same problem in some sense. I couldn't figure out why for
some processes I got all of the output right away and for others it all
got queued up 'til the process ended. Working out a simple example for
this thread showed the light: my calling of my_process.stdout.read() was
blocking 'til EOF. Oops.

-tom!

--

Yes, I did try your example. But, after talking on the #python IRC
channel, I realized that it wasn't the process that was blocking, it
was my GUI. I had to fire an event using gobject.io_add_watch()
whenever data was received from the child process. The event then read
from the process and added a line to my gtk.TextView.

Thank you for your suggestions, though.

On Dec 30, 12:12 am, Tom Plunket <t...@fancy.orgwrote:

Gabriel Genellina wrote:

work with the same problem in some sense. I couldn't figure out why for
some processes I got all of the output right away and for others it all
got queued up 'til the process ended. Working out a simple example for
this thread showed the light: my calling of my_process.stdout.read() was
blocking 'til EOF. Oops.
>
-tom!
>
--