By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,173 Members | 796 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,173 IT Pros & Developers. It's quick & easy.

How to perform a nonblocking read from a process

P: n/a
Hi,
I'm trying to perform following operation from inside the python
script
1. Open a shell ( start a process )
2. Send command1 to the process
3. Get output from the process
4. Send command2 to the process
5. Get output from the process
.......
Following is sample code :

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n')
o,e = p2.stdout.readline()
print o,e
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

All the help is appreciated ..

Thanks,
-Rahul.

Jun 27 '08 #1
Share this Question
Share on Google+
4 Replies


P: n/a
On Jun 4, 3:20 am, rdab...@gmail.com wrote:
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..
Threads are the simplest remedy for blocking i/o.

Jun 27 '08 #2

P: n/a
On Jun 3, 7:53 pm, sturlamolden <sturlamol...@yahoo.nowrote:
On Jun 4, 3:20 am, rdab...@gmail.com wrote:
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

Threads are the simplest remedy for blocking i/o.

Threads are the simplest remedy for blocking i/o.
I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...
Jun 27 '08 #3

P: n/a
On Jun 4, 8:45 am, rdab...@gmail.com wrote:
I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...

Are you planning on doing thousands of simultaneous asynchronous I/O
calls? How many processes are you communicating with? Take a look at
the twisted framwork (twistedmatrix.org) or perhaps i/o completion
ports on Windows (e.g. win32file.CreateIoCompletionPort in PyWin32).
But if you need that king of scalability, I suggest you start by
reconsidering the design.

If you are communicating with only one process, you don't need
thousands of threads, just one. Keep the thread idle until you need it
again. Here is a simple worker thread that allows you to do multiple,
different function calls in a background thread (you can shut it down
by passing None to setTask).
import threading
import Queue

class WorkerThread(threading.Thread):

def __init__(self):
self.taskQueue = Queue.Queue(0)
self.resQueue = Queue.Queue(0)
self.setDaemon(True)

def run(self):
while 1:
fun, args, kwarg = self.taskQueue.get()
if (fun is None): break
self.resQueue.put(fun(*args,**kwarg))

def setTask(self, fun, *args, **kwarg):
self.taskQueue.set((fun, args, kwarg))

def getResult(self):
return self.resQueue.get()

def probeResult(self):
return not self.resQueue.empty()
Rewrite of your examples:

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
worker = WorkerThread()
for i in range(10):
worker.setTask(p2.stdin.write, 'print 10'+'\n')
worker.setTask(p2.stdout.readline)
worker.getResult() # wait for write to finish
o,e = worker.getResult() # wait for read to finish
print o,e

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
writer = WorkerThread()
reader = WorkerThread()
for i in range(10):
writer.setTask(p2.stdin.write, 'print 10'+'\n')
reader.setTask(p2.stdout.readline)
o,e = reader.getResult() # wait for read to finish
print o,e
Jun 27 '08 #4

P: n/a
rd*****@gmail.com <rd*****@gmail.comwrote:
Hi,
I'm trying to perform following operation from inside the python
script
1. Open a shell ( start a process )
2. Send command1 to the process
3. Get output from the process
4. Send command2 to the process
5. Get output from the process
......
Following is sample code :

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n')
o,e = p2.stdout.readline()
print o,e

It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..
If you are working under linux/some-unix-like-os then use the python
expect module which is for exactly this sort of thing.

http://www.noah.org/wiki/Pexpect

Pexpect is a pure Python expect-like module. Pexpect makes Python a
better tool for controlling other applications.

Pexpect is a pure Python module for spawning child applications;
controlling them; and responding to expected patterns in their
output. Pexpect works like Don Libes' Expect. Pexpect allows your
script to spawn a child application and control it as if a human
were typing commands.

Pexpect can be used for automating interactive applications such as
ssh, ftp, passwd, telnet, etc. It can be used to a automate setup
scripts for duplicating software package installations on different
servers. It can be used for automated software testing. Pexpect is
in the spirit of Don Libes' Expect, but Pexpect is pure
Python. Unlike other Expect-like modules for Python, Pexpect does
not require TCL or Expect nor does it require C extensions to be
compiled. It should work on any platform that supports the standard
Python pty module. The Pexpect interface was designed to be easy to
use.

You'll never get it to work with subprocess like this because of the
buffering.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jun 27 '08 #5

This discussion thread is closed

Replies have been disabled for this discussion.