By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
434,701 Members | 1,938 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 434,701 IT Pros & Developers. It's quick & easy.

Can read() be non-blocking?

P: n/a
This issue has been raised a couple of times I am sure. But I have yet
to find a satisfying answer.

I am reading from a subprocess and this subprocess sometimes hang, in
which case a call to read() call will block indefinite, keeping me from
killing it.

The folloing sample code illustrates the problem:

proc = subprocess.Popen(['/usr/bin/foo', '/path/to/some/file'],
stdout=subprocess.PIPE)
output = StringIO.StringIO()
while True:
r = select.select([proc.stdout.fileno()], [], [], 5)[0]
if r:
# NOTE: This will block since it reads until EOF
data = proc.stdout.read()
if not data:
break # EOF from process has been reached
else:
output.write(data)
else:
os.kill(proc.pid, signal.SIGKILL)
proc.wait()

<Process the output...>

As the NOTE: comment above suggests the call to read() will block here.

I see two solutions:

1. Read one byte at a time, meaning call read(1).
2. Read non-blocking.

I think reading one byte at a time is a waste of CPU, but I cannot find
a way to read non-blocking.

Is there a way to read non-blocking? Or maybe event a better way in
generel to handle this situation?

Thanks

Thomas

Nov 6 '08 #1
Share this Question
Share on Google+
5 Replies


P: n/a
In message <ma**************************************@python.o rg>, Thomas
Christensen wrote:
r = select.select([proc.stdout.fileno()], [], [], 5)[0]
if r:
# NOTE: This will block since it reads until EOF
data = proc.stdout.read()
No, it will read what data is available.
Nov 7 '08 #2

P: n/a
In message <gf**********@lust.ihug.co.nz>, Lawrence D'Oliveiro wrote:
In message <ma**************************************@python.o rg>, Thomas
Christensen wrote:
> r = select.select([proc.stdout.fileno()], [], [], 5)[0]
if r:
# NOTE: This will block since it reads until EOF
data = proc.stdout.read()

No, it will read what data is available.
Sorry, maybe not. But you can set O_NOBLOCK on the fd.
Nov 7 '08 #3

P: n/a
On Nov 7, 6:54*am, Thomas Christensen <thom...@thomaschristensen.org>
wrote:
This issue has been raised a couple of times I am sure. *But I have yet
to find a satisfying answer.

I am reading from a subprocess and this subprocess sometimes hang, in
which case a call to read() call will block indefinite, keeping me from
killing it.

The folloing sample code illustrates the problem:

* proc = subprocess.Popen(['/usr/bin/foo', '/path/to/some/file'],
* * * * * * * * * * * * * stdout=subprocess.PIPE)
* output = StringIO.StringIO()
* while True:
* * * r = select.select([proc.stdout.fileno()], [], [], 5)[0]
* * * if r:
* * * * * # NOTE: This will block since it reads until EOF
* * * * * data = proc.stdout.read()
* * * * * if not data:
* * * * * * * break *# EOF from process has been reached
* * * * * else:
* * * * * * * output.write(data)
* * * else:
* * * * * os.kill(proc.pid, signal.SIGKILL)
* proc.wait()

* <Process the output...>

As the NOTE: comment above suggests the call to read() will block here.

I see two solutions:

1. Read one byte at a time, meaning call read(1).
2. Read non-blocking.

I think reading one byte at a time is a waste of CPU, but I cannot find
a way to read non-blocking.

Is there a way to read non-blocking? *Or maybe event a better way in
generel to handle this situation?

Thanks

* * * * * * * * Thomas
As far as I know, you can use '''fctnl''' to make a file handle non-
blocking.

But :

1. I don't know if it works on standard io
2. If it works in python
Nov 7 '08 #4

P: n/a
On Nov 6, 2:54*pm, Thomas Christensen <thom...@thomaschristensen.org>
wrote:
This issue has been raised a couple of times I am sure. *But I have yet
to find a satisfying answer.

I am reading from a subprocess and this subprocess sometimes hang, in
which case a call to read() call will block indefinite, keeping me from
killing it.

The folloing sample code illustrates the problem:

* proc = subprocess.Popen(['/usr/bin/foo', '/path/to/some/file'],
* * * * * * * * * * * * * stdout=subprocess.PIPE)
* output = StringIO.StringIO()
* while True:
* * * r = select.select([proc.stdout.fileno()], [], [], 5)[0]
* * * if r:
* * * * * # NOTE: This will block since it reads until EOF
* * * * * data = proc.stdout.read()
* * * * * if not data:
* * * * * * * break *# EOF from process has been reached
* * * * * else:
* * * * * * * output.write(data)
* * * else:
* * * * * os.kill(proc.pid, signal.SIGKILL)
* proc.wait()

* <Process the output...>

As the NOTE: comment above suggests the call to read() will block here.

I see two solutions:

1. Read one byte at a time, meaning call read(1).
2. Read non-blocking.

I think reading one byte at a time is a waste of CPU, but I cannot find
a way to read non-blocking.

Is there a way to read non-blocking? *Or maybe event a better way in
generel to handle this situation?
From what I understand, you want a way to abort waiting on a blocking
read if the process is hung.
There are some challenges about how you decide if the process is hung
or just busy doing work without generating output for a while (or may
be the system is busy and the process didn't get enough CPU due to
other CPU hungry processes).
Assuming you have a valid way to figure this out, one option is to
have a timeout on the read.
If the timeout exceeds, you abort the read call. No, the read doesn't
provide a timeout, you can build one using alarm.

def alarm_handler(*args):
""" This signal stuff may not work in non unix env """
raise Exception("timeout")

signal.signal(signal.SIGALRM, alarm_handler)

try:
signal.alarm(timeout) # say timeout=60 for a max wait of 1
minute
data = proc.stdout.read()
except Exception, e:
if not str(e) == 'timeout': # something else went wrong ..
raise
# got the timeout exception from alarm .. proc is hung; kill it

Karthik
>
Thanks

* * * * * * * * Thomas
Nov 7 '08 #5

P: n/a
On Nov 7, 9:09*am, Lawrence D'Oliveiro <l...@geek-
central.gen.new_zealandwrote:
In message <gf07sh$in...@lust.ihug.co.nz>, Lawrence D'Oliveiro wrote:
In message <mailman.3600.1226012406.3487.python-l...@python.org>, Thomas
Christensen wrote:
* * * r = select.select([proc.stdout.fileno()], [], [], 5)[0]
* * * if r:
* * * * * # NOTE: This will block since it reads until EOF
* * * * * data = proc.stdout.read()
No, it will read what data is available.

Sorry, maybe not. But you can set O_NOBLOCK on the fd.
Set O_NONBLOCK on proc.fileno() and try using os.read() on that
descriptor.

-srp
Nov 7 '08 #6

This discussion thread is closed

Replies have been disabled for this discussion.