Re: Popen pipe hang

[Jean-Paul Calderone]

On Mon, 12 May 2008 17:35:44 -0700 (PDT), schickb <sc*****@gmail.comwrote:

>I'm trying to pipe data that starts life in an array('B') object
through several processes. The code below is a simplified example. The
data makes it through, but the wait() always hangs. Is there a better
way to indicate src.stdin has reach EOF?

from subprocess import Popen, PIPE
from array import array

arr = array('B')
arr.fromstring("hello\n")

src = Popen( ["cat"], stdin=PIPE, stdout=PIPE)
dst = Popen( ["cat"], stdin=src.stdout)
arr.tofile(src.stdin)
src.stdin.close()
dst.wait()

Alas, you haven't actually closed src's standard input. Though you did
close src.stdin, you didn't close the copy of it which was inherited by
dst! So though the file descriptor is no longer open in your main process,
it remains open due to the reference dst has to it. You can fix this by
having Popen close all file descriptors except 0, 1, and 2 before it execs
cat - pass close_fds=True to the 2nd Popen call and you should get the
behavior you want.

Jean-Paul

[schickb]

On May 12, 7:35 pm, Jean-Paul Calderone <[e-mail]:

from subprocess import Popen, PIPE
from array import array

arr = array('B')
arr.fromstring("hello\n")

src = Popen( ["cat"], stdin=PIPE, stdout=PIPE)
dst = Popen( ["cat"], stdin=src.stdout)
arr.tofile(src.stdin)
src.stdin.close()
dst.wait()

Alas, you haven't actually closed src's standard input. Though you did
close src.stdin, you didn't close the copy of it which was inherited by
dst! So though the file descriptor is no longer open in your main process,
it remains open due to the reference dst has to it. You can fix this by
having Popen close all file descriptors except 0, 1, and 2 before it execs
cat - pass close_fds=True to the 2nd Popen call and you should get the
behavior you want.

Thanks, that did the trick. Although I assume that by passing
close_fds=True the second Popen is actually closing src.stdout (rather
than src.stdin as mentioned)? With close_fds=True, is Python buffering
the data even though bufsize=0 is the default? Otherwise I don't see
how it could close the fd before executing the process.