popen question

[Robert Latest]

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Thanks,

robert

[Robert Latest]

Marc 'BlackJack' Rintsch wrote:

Both processes have to make their communication ends unbuffered or line
buffered.

Yeah, I figured something like that.

And do whatever is needed to output the numbers from ``slow``
unbuffered or line buffered.

Hm, "slow" of course is just a little test program I wrote for this purpose.
In reality I want to call another program whose behavior I can't influence
(well, technically I could because it's open-source, but let's assume it to
be a black box for now).

If 'slow' or some other program does buffered output, how come I can see
its output line-by-line in the shell?

robert

[Hrvoje Niksic]

Robert Latest <bo*******@yahoo.comwrites:

If 'slow' or some other program does buffered output, how come I can
see its output line-by-line in the shell?

stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.

In reality I want to call another program whose behavior I can't
influence (well, technically I could because it's open-source, but
let's assume it to be a black box for now).

To test whether your black box buffers output to pipe, simply start it
like this:

$ ./slow | cat

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).

[Robert Latest]

Hrvoje Niksic wrote:

stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.

Makes sense.

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).

Do I really? After all, the shell itself doesn't hack stdio, does it?
Anyway, I'm taking this over to comp.unix.programmer since it really isn't a
python problem.

Thanks,
robert

[Robert Latest]

pexpect is the solution. Seems to wrap quite a bit of dirty pseudo-tty
hacking.

robert

[Karthik Gurusamy]

On Jan 8, 1:20 am, Robert Latest <[e-mail]:

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

I've seen this problem and it took me a while to figure out what is
happening.
As other posts, I too first suspected it's a problem related to line/
full buffering on the sender side (./slow here).

It turned out that the "for line in child:" line in the iterator is
the culprit. The iterator does something like a child.readlines()
underneath (in it's __iter__ call) instead of a more logical single
line read.

Change your reading to force line-by-line read
e.g.
While True:
line = child.readline()
if not line: break
print line

Karthik

>
Thanks,

robert