Polling, Fifos, and Linux

[Jeremy Moles]

This is my first time working with some of the more lower-level python
"stuff." I was wondering if someone could tell me what I'm doing wrong
with my simple test here?

Basically, what I need is an easy way for application in userspace to
simply echo values "down" to this fifo similar to the way proc files are
used. Is my understanding of fifo's and their capabilities just totally
off base?

I've implemented something very similar in the past in C which works
fine; however, this app immediately takes up 100% of my CPU after the
first read from the fifo, regardless of whatever options I pass to
os.open(). Am I not actually reading the data out of the fifo? Is that
what's causing it to poll infinite thereafter?

Any help would be greatly appreciate, though I'll keep reading and
experimenting in the meantime.

#!/usr/bin/env python

import select
import os

poller = select.poll()
fifo = os.open("fifo", os.O_RDONLY | os.O_NONBLOCK)

poller.register(fifo, select.POLLIN)

while True:
p = poller.poll()
# only have one file
string = os.read(p[0][0], 1)
if len(string):
print string

[Jacob Page]

Jeremy Moles wrote:

This is my first time working with some of the more lower-level python
"stuff." I was wondering if someone could tell me what I'm doing wrong
with my simple test here?

Basically, what I need is an easy way for application in userspace to
simply echo values "down" to this fifo similar to the way proc files are
used. Is my understanding of fifo's and their capabilities just totally
off base?

You shouldn't need to use select.poll(), unless I'm missing something.
I was able to get the following to work:

-=-=-

import os

fifo = os.open("fifo", os.O_RDONLY | os.O_NONBLOCK)

while True:
string = os.read(fifo, 1)
if len(string):
print string
# Perhaps add a delay under an else

-=-=-

The Python script, when run, does nothing until you put data into the
fifo from another process. Then it immediately spits the data out,
character by character.

I'm assuming that you've already created the fifo and that it's in the
current working directory.

[Andreas Kostyrka]

On Thu, Jul 07, 2005 at 10:21:19PM -0700, Jacob Page wrote:

Jeremy Moles wrote:

This is my first time working with some of the more lower-level python
"stuff." I was wondering if someone could tell me what I'm doing wrong
with my simple test here?

Basically, what I need is an easy way for application in userspace to
simply echo values "down" to this fifo similar to the way proc files are
used. Is my understanding of fifo's and their capabilities just totally
off base?
You shouldn't need to use select.poll(), unless I'm missing something.
I was able to get the following to work:

Ok, you miss something ;) The program you proposed does busy waiting
and without a time.sleep call will consume 100% CPU time :(

Actually, with a named fifo the situation gets even nastier:

import os, select, time

fifo = os.open("fifo", os.O_RDONLY)

while True:
print "SELECT", select.select([fifo],[],[])
string = os.read(fifo, 1)
if len(string):
print string
else:
nf = os.open("fifo", os.O_RDONLY)
os.close(fifo)
fifo = nf
# Perhaps add a delay under an else

The problem is, that select (and poll) show a End-Of-File condition by returning
ready to read. But on a FIFO, when the first client terminates, the reading
end goes into a EOF state till somebody else reopens the fifo for writing.

[This bit of wisdom comes Advanced Programming in the UNIX Environment by
W.R. Stevens p. 400: 'If we encounter the end of file on a descriptor, that
descriptor is considered readbale by select.']

closing the old descriptor must be done after opening a new one, or else you
get a tiny moment where a O_WRONLY client is not able to open the file.
This way there is always a reading client of the fifo.

Andreas


-=-=-

import os

fifo = os.open("fifo", os.O_RDONLY | os.O_NONBLOCK)

while True:
string = os.read(fifo, 1)
if len(string):
print string
# Perhaps add a delay under an else

-=-=-

The Python script, when run, does nothing until you put data into the
fifo from another process. Then it immediately spits the data out,
character by character.

I'm assuming that you've already created the fifo and that it's in the
current working directory.
--
http://mail.python.org/mailman/listinfo/python-list

[Donn Cave]

In article <ma***************************************@python. org>,
Andreas Kostyrka <an*****@kostyrka.org> wrote:

On Thu, Jul 07, 2005 at 10:21:19PM -0700, Jacob Page wrote:

Jeremy Moles wrote:

This is my first time working with some of the more lower-level python
"stuff." I was wondering if someone could tell me what I'm doing wrong
with my simple test here?

Basically, what I need is an easy way for application in userspace to
simply echo values "down" to this fifo similar to the way proc files are
used. Is my understanding of fifo's and their capabilities just totally
off base?
You shouldn't need to use select.poll(), unless I'm missing something.
I was able to get the following to work:

Ok, you miss something ;) The program you proposed does busy waiting
and without a time.sleep call will consume 100% CPU time :(
I don't doubt that it would, but that's because he (like the
original poster) open the file with O_NONBLOCK. From my point
of view that's a self-inflicted injury, but if you start from
the assumption that O_NONBLOCK is needed for some reason, then
the poll makes sense. In normal blocking mode, select+read is
identical to plain read for any kind of file that supports select.
Actually, with a named fifo the situation gets even nastier:

import os, select, time

fifo = os.open("fifo", os.O_RDONLY)

while True:
print "SELECT", select.select([fifo],[],[])
string = os.read(fifo, 1)
if len(string):
print string
else:
nf = os.open("fifo", os.O_RDONLY)
os.close(fifo)
fifo = nf
# Perhaps add a delay under an else

The problem is, that select (and poll) show a End-Of-File condition by
returning
ready to read. But on a FIFO, when the first client terminates, the reading
end goes into a EOF state till somebody else reopens the fifo for writing.

[This bit of wisdom comes Advanced Programming in the UNIX Environment by
W.R. Stevens p. 400: 'If we encounter the end of file on a descriptor, that
descriptor is considered readbale by select.']

closing the old descriptor must be done after opening a new one, or else you
get a tiny moment where a O_WRONLY client is not able to open the file.
This way there is always a reading client of the fifo.

OK, but in more detail, what happens in these two scenarios?
In your version, what happens when the writer opens a pipe
pipe that the reader is about to close? Who reads the data?

On the other hand, if you close the pipe first, what happens
to the writer who happens to try to open the pipe at that moment?

Luckily, as far as I know, we don't have to worry about the
first one, since if data could be lost in this way it would
be much more complicated to close a file descriptor without
running this risk.

But I don't see the second one as much of a problem either.
The writer blocks - so?

Now, what would really be useful is a way for the writer to
detect whether open will block, and potentially time out.

Donn Cave, do**@u.washington.edu

[Remi Villatel]

Jeremy Moles wrote:

This is my first time working with some of the more lower-level python
"stuff." I was wondering if someone could tell me what I'm doing wrong
with my simple test here?

Basically, what I need is an easy way for application in userspace to
simply echo values "down" to this fifo similar to the way proc files are
used. Is my understanding of fifo's and their capabilities just totally
off base?

Funny coincidence... I was also trying some FIFO stuff today. I came to a
very simple solution for just one writer and one listener. I make this stuff
send text (strings) from one Konsole to another Konsole.

The FIFO was created from Bash with:

$ mkfifo -m 666 fifo

listener.py
------------------
#! /usr/bin/env python

fifo = open("fifo","r")

while True:
print fifo.readline()[:-1]
# [:-1] because of the newline added by print.
#
------------------

writer.py
------------------
#! /usr/bin/env python

fifo = open("fifo", "a") # Mode "a" or "w"

try:
fifo.write("This string goes down the pipe.\n")
# Newline is important because the listener uses readline().
fifo.flush()
# Output is buffered.
except IOError:
pass
#
------------------

If you kill the writer, the listener remains quiet until somebody writes
into the pipe. The same happens if there is no writer.

If you kill the listener, the writer reports a broken pipe when it tries to
flush().

The writer can close and open the pipe to its liking, the listener doesn't care.

The only problem is that the writer freezes when it opens the pipe until
there is a listener at the other end.

Any way, it's very simple and it uses 0% of the CPU.

Just my 2 Euro cents,

--
==================
Remi Villatel
maxilys_@_tele2.fr
==================