470,644 Members | 1,232 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,644 developers. It's quick & easy.

subprocess module and blocking

I'm using a polling loop in a thread that looks approximately like this

while 1:
p = find_a_process()
rc = p.poll()
if rc is not None:
out, err = p.communicate()
#deal with output etc
sleep(1)

the process p is opened using

p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=cwd)

stdin is actually never written to.

I notice that under both win32 and freebsd that things are fine provided
that the subprocess doesn't write too much to stdout/stderr. However,
the subprocess seems to lock often if too much is written (under freebsd
I see a process state of POLL). I assume that the subprocess is filling
up the pipe and then failing to wake up again. I had expected that
subprocess would take care of this for me, but possibly I'm being
utterly clueless and stupid. What should I do to avoid blocking in the
subprocess?
--
Robin Becker
Jul 19 '05 #1
2 1450
Robin Becker <ro***@SPAMREMOVEjessikat.fsnet.co.uk> writes on Sun, 12 Jun 2005 09:22:52 +0000:
I'm using a polling loop in a thread that looks approximately like this

while 1:
p = find_a_process()
rc = p.poll()
if rc is not None:
out, err = p.communicate()
#deal with output etc
sleep(1)
....
I notice that under both win32 and freebsd that things are fine
provided that the subprocess doesn't write too much to
stdout/stderr. However, the subprocess seems to lock often if too much
is written (under freebsd I see a process state of POLL). I assume
that the subprocess is filling up the pipe and then failing to wake up
again. I had expected that subprocess would take care of this for me,


You just found out that this is not the case.

The warning attached to "communicate"s docstring might have
you averted: "Note: the data read is buffered in memory...
do not use for large size".

If subprocess would do what you expect, it would need to
buffer the output (to make room in the pipes again).
But, for large data, this could have dramatic consequences.

Thus, you should use "select" on the pipes to find out
when to read data.
Dieter
Jul 19 '05 #2
Dieter Maurer wrote:
......

You just found out that this is not the case.
thanks I suppose I was being a moron.

The warning attached to "communicate"s docstring might have
you averted: "Note: the data read is buffered in memory...
do not use for large size".

If subprocess would do what you expect, it would need to
buffer the output (to make room in the pipes again).
But, for large data, this could have dramatic consequences.

Thus, you should use "select" on the pipes to find out
when to read data.

Unfortunately select isn't an option for windows (I think that's only for
sockets). I guess I need to read repeatedly from the stdout etc to get the
output. Clearly the poll call can't return a status until we've finished
reading.

Dieter

--
Robin Becker

Jul 19 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

17 posts views Thread by Michael McGarry | last post: by
4 posts views Thread by Nicolas Fleury | last post: by
4 posts views Thread by Marc Carter | last post: by
3 posts views Thread by Tom Plunket | last post: by
3 posts views Thread by Kevin Walzer | last post: by
12 posts views Thread by bhunter | last post: by
8 posts views Thread by rdabane | last post: by
1 post views Thread by Korara | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.