Problem with select.poll and popen2

[Magnus Lycka]

I'm trying to read standard out in a process started with popen2
in a non-blocking way. (Other good ways of doing this than the
one I tried are appreciated.)

I've tried to dumb down my code to see what happens, and
socket.poll seems to behave very strangely. I've tried to
use the .poll method for the poll object with and without
a timeout, but in either case, the output randomly switches
between on of the versions below. It runs fast, there is
certainly no 1000 ms delay between printed full stops.

I suspect Python does things in the background with the
pipes that confuse select/poll.

import sys
import select
import popen2
import os

CMD = 'date'

#fi, fo = os.popen2(CMD) # Same result with this...
fo, fi = popen2.popen2(CMD)
p = select.poll()
p.register(fo, select.POLLOUT)
while 1:
print '.',
sys.stdout.flush()
l = p.poll(1000)
for (o, evt) in l:
if o == fo.fileno() and evt == select.POLLOUT:
print 'Reading', fo
c = fo.read(1)
print c,

$ python clpy.py
.. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
T . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
u . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
e . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
A . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
u . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
g . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
3 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
0 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
1 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
5 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
: . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
4 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
5 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
: . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
3 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
8 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
C . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
E . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
S . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
T . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
2 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
0 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
0 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
5 . Reading <open file '(fdopen)', mode 'r' at 0x92f9070>

.. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[etc]

$ python clpy.py
.. Reading <open file '(fdopen)', mode 'r' at 0x92f9070>
T . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[etc]

$ python clpy.py
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
[etc]

[Magnus Lycka]

Magnus Lycka wrote:

I'm trying to read standard out in a process started with popen2
in a non-blocking way. (Other good ways of doing this than the
one I tried are appreciated.)

I'm starting to get on top of this. First of all, I was confused
by POLLIN and POLLOUT, since it's IN or OUT of the polling process,
so we want the POLLIN filter for the output of the spawned process...

Secondly, I thought that I'd get a fd, event pair for each kind of
event, but events are bitwise ORed together, and in my dumb downed
version I had used a command (date) that ends quickly, and then there
is a stream of POLLHUP that comes out of poll, even though I didn't
register for that. (Is it supposed to be like that?)

The following pair of programs (sender.py first) seems to behave as
I intended. Perhaps they can be of use for someone. After all, it's
not only sockets that we might want to read in an unblocking way,
but it seems most examples are geared towards that.
import time, sys
while 1:
print 'HELLO'
sys.stdout.flush()
time.sleep(2)

.....

import sys
import select
import os

CMD = 'python sender.py'

fi, fo = os.popen2(CMD)
fi.close()
p = select.poll()
p.register(fo, select.POLLIN)
while 1:
l = p.poll(500)
for (o, evt) in l:
if o == fo.fileno() and evt & select.POLLIN:
c = fo.read(10)
print c,
else:
print "Timeout!"