470,591 Members | 1,609 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,591 developers. It's quick & easy.

line-by-line output from a subprocess

I am using the subprocess module to invoke a command-line utility and
process the output.

However, I would like to process the output line-by-line as it is
generated rather than
running the subprocess to completion and THEN processing the results.
So, for instance,
I'd like to write code like this:

all_files = []
import subprocess
subp = subprocess.Popen('dir', stdout=subprocess.PIPE)
for line in subp.stdout:
sys.stdout.print(line)
all_files.append(line)

....and have it print the lines to stdout one-by-one until the entire
list is
collected. (Pretend that 'dir' is VERY slow.)

The communicate() method in subprocess blocks until the subprocess has
exited... I'm sure there is some good reason for this behavior. But how
does
one achieve what I am trying to do?

(PS: this is on Windows... although I'd rather use an OS-agnostic
solution if
one exists.)

-- Michael Chermside


************************************************** ***************************
This email may contain confidential or privileged information. If you believe
you have received the message in error, please notify the sender and delete
the message without copying or disclosing it.
************************************************** ***************************

Jul 19 '05 #1
2 1706
Okay, so the reason what you're trying to do doesn't work is that the
readahead buffer used by the file iterator is 8192 bytes, which clearly
might be too much. It also might be because the output from the
application you're running is buffered, so you might have to do
something about that as well.

Anyway, if the output from the child application is unbuffered, writing
a generator like this would work:

def iterread(fobj):
stdout = fobj.stdout.read(1) # or what you like
data = ""
while stdout:
data += stdout
while "\n" in data:
line, data = data.split("\n", 1)
yield line
stdout = fobj.stdout.read(1)
if data:
yield data,

Jul 19 '05 #2
Jp Calderone wrote:
Or, doing the same thing, but with less code:


Hmm ... What have I been smoking?

Jul 19 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

1 post views Thread by Steve Quezadas | last post: by
5 posts views Thread by Grant Edwards | last post: by
12 posts views Thread by Eric_Dexter | last post: by
9 posts views Thread by Phoe6 | last post: by
12 posts views Thread by bhunter | last post: by
7 posts views Thread by skunkwerk | last post: by
6 posts views Thread by John Mechaniks | last post: by
3 posts views Thread by topazcode | last post: by
5 posts views Thread by thedsadude | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.