Capturing stdout without waiting for the process end

[Luigi]

Hi to all!

I'd like to execute an external program capturing the stdout/stderr
messages at "real-time". I mean that I don't want to wait for the end
of the process. If I write a code like this:

import os
import sys

class Runner:

def run(self, arg):
try:
fin, fout = os.popen4(arg)
self.outData = fout.readlines( )
self.outStatus = fout.close()

except Exception, err:
print err

def printOut(self):
print "###"
print self.outStatus
print "###"
for line in self.outData:
print line
r = Runner()
r.run("ls /tmp")
r.printOut()

I can print out (in this case in the os.stdout, but it may be
elsewhere) the whole external program output only once it ends. How can
I do to intercept the external program output during the processing?

Thank you in advance

Luigi

[akameswaran]

What OS are you doing this on?

I had an issue similar to this and it was due to default buffering
behavior of my tty's. If I recall correctly I executed a bunch of
settty to eliminate the buffering behavior of stdout. The set the
terminal back to it's original setting when my program was done.

Anand

[Luigi]

I use Linux.

So, if I modify my sdtout behaviour, when I execute:

fin, fout = os.popen4(arg)

this is executed asyncronously? And how can I intercept a fout.write()
event?

The question is that I have a C program (by third part) that streams
the logs into the stderr and stdout devices. I need to create an
envelopment that captures the outputs and puts them in a file
generating log events (for a real-time view).

Thanks

Luigi

[Diez B. Roggisch]

> fin, fout = os.popen4(arg)

this is executed asyncronously? And how can I intercept a fout.write()
event?

You don't intercept anything, you read from the child process stdin, which
from your POV is a readable stream.

Diez

[akameswaran]

Diez is correct, the C program is writting to stdout, you are reading
from stdout. Default bahavior in most modern Unix like systems is to
buffer std out. I stumbled on this a long time ago, so I am trying to
remember the details.

What I think is happening here, you call the child process, it does
it's thing and puts thing into stdout.

How long does the child process run typically?

What also is the actual behavior? How is it wrong, exactly? Are you
getting the full output from the C program? Part of it? I still
suspect a buffering problem of some sort, but I am not clear on what is
going wrong.

[Donn Cave]

In article <11************ **********@z34g 2000cwc.googleg roups.com>,
"Luigi" <lu*********@li bero.it> wrote:

The question is that I have a C program (by third part) that streams
the logs into the stderr and stdout devices. I need to create an
envelopment that captures the outputs and puts them in a file
generating log events (for a real-time view).

As suggested in another followup, the C program's
output will probably be "block buffered" when its
output is a pipe. In this case, you'll get output
only when the buffer is full, or when the program
exits.

If the program can be modified, it only needs to
flush stdout after each output event. Otherwise,
you need to use a device that looks like a tty,
just so the C stdio library will switch to line
buffering as it generally does with terminals.

This is called a pseudotty. It's a little more
difficult to use than a pipe, but you can probably
get something going with openpty or forkpty from
the os/posix module, or there may still be 3rd
party packages for this.

Donn Cave, do**@u.washingt on.edu