469,332 Members | 6,612 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,332 developers. It's quick & easy.

popen question

Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?

Thanks,

robert
Jan 8 '08 #1
5 1086
Marc 'BlackJack' Rintsch wrote:
Both processes have to make their communication ends unbuffered or line
buffered.
Yeah, I figured something like that.
And do whatever is needed to output the numbers from ``slow``
unbuffered or line buffered.
Hm, "slow" of course is just a little test program I wrote for this purpose.
In reality I want to call another program whose behavior I can't influence
(well, technically I could because it's open-source, but let's assume it to
be a black box for now).

If 'slow' or some other program does buffered output, how come I can see
its output line-by-line in the shell?

robert
Jan 8 '08 #2
Robert Latest <bo*******@yahoo.comwrites:
If 'slow' or some other program does buffered output, how come I can
see its output line-by-line in the shell?
stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.
In reality I want to call another program whose behavior I can't
influence (well, technically I could because it's open-source, but
let's assume it to be a black box for now).
To test whether your black box buffers output to pipe, simply start it
like this:

$ ./slow | cat

If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).
Jan 8 '08 #3
Hrvoje Niksic wrote:
stdio uses different buffering strategies depending on the output
type. When the output is a TTY, line buffering is used; when the
output goes to a pipe or file, it is fully buffered.
Makes sense.
If you see lines one by one, you are in luck, and you can fix things
on the Python level simply by avoiding buffering in popen. If not,
you will need to resort to more advanced hackery (e.g. fixing stdio
using LD_PRELOAD).
Do I really? After all, the shell itself doesn't hack stdio, does it?
Anyway, I'm taking this over to comp.unix.programmer since it really isn't a
python problem.

Thanks,
robert
Jan 8 '08 #4
pexpect is the solution. Seems to wrap quite a bit of dirty pseudo-tty
hacking.

robert
Jan 8 '08 #5
On Jan 8, 1:20 am, Robert Latest <boblat...@yahoo.comwrote:
Hello,

look at this function:

--------------
def test():
child = os.popen('./slow')
for line in child:
print line
-------------

The program "slow" just writes the numbers 0 through 9 on stdout, one line a
second, and then quits.

I would have expected the python program to spit out a numbers one by one,
instead I see nothing for 10 seconds and then the whole output all at once.

How can I get and process the pipe's output at the pace it is generated?
I've seen this problem and it took me a while to figure out what is
happening.
As other posts, I too first suspected it's a problem related to line/
full buffering on the sender side (./slow here).

It turned out that the "for line in child:" line in the iterator is
the culprit. The iterator does something like a child.readlines()
underneath (in it's __iter__ call) instead of a more logical single
line read.

Change your reading to force line-by-line read
e.g.
While True:
line = child.readline()
if not line: break
print line

Karthik
>
Thanks,

robert
Jan 9 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Stuart McGraw | last post: by
17 posts views Thread by bastiaannaber | last post: by
5 posts views Thread by damacy | last post: by
3 posts views Thread by Jesse | last post: by
12 posts views Thread by Eric_Dexter | last post: by
reply views Thread by suresh191 | last post: by
reply views Thread by Marylou17 | last post: by
1 post views Thread by Marylou17 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.