473,287 Members | 3,295 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,287 software developers and data experts.

How to perform a nonblocking read from a process

Hi,
I'm trying to perform following operation from inside the python
script
1. Open a shell ( start a process )
2. Send command1 to the process
3. Get output from the process
4. Send command2 to the process
5. Get output from the process
.......
Following is sample code :

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n')
o,e = p2.stdout.readline()
print o,e
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

All the help is appreciated ..

Thanks,
-Rahul.

Jun 27 '08 #1
4 4473
On Jun 4, 3:20 am, rdab...@gmail.com wrote:
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..
Threads are the simplest remedy for blocking i/o.

Jun 27 '08 #2
On Jun 3, 7:53 pm, sturlamolden <sturlamol...@yahoo.nowrote:
On Jun 4, 3:20 am, rdab...@gmail.com wrote:
It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..

Threads are the simplest remedy for blocking i/o.

Threads are the simplest remedy for blocking i/o.
I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...
Jun 27 '08 #3
On Jun 4, 8:45 am, rdab...@gmail.com wrote:
I've to admit I'm a newbie to this kind of programming...
what if I have to run thousands of these commands...it doesn't make
sense to create
thousands of threads..
Is there a way that above mentioned piece of code be made to worked...

Are you planning on doing thousands of simultaneous asynchronous I/O
calls? How many processes are you communicating with? Take a look at
the twisted framwork (twistedmatrix.org) or perhaps i/o completion
ports on Windows (e.g. win32file.CreateIoCompletionPort in PyWin32).
But if you need that king of scalability, I suggest you start by
reconsidering the design.

If you are communicating with only one process, you don't need
thousands of threads, just one. Keep the thread idle until you need it
again. Here is a simple worker thread that allows you to do multiple,
different function calls in a background thread (you can shut it down
by passing None to setTask).
import threading
import Queue

class WorkerThread(threading.Thread):

def __init__(self):
self.taskQueue = Queue.Queue(0)
self.resQueue = Queue.Queue(0)
self.setDaemon(True)

def run(self):
while 1:
fun, args, kwarg = self.taskQueue.get()
if (fun is None): break
self.resQueue.put(fun(*args,**kwarg))

def setTask(self, fun, *args, **kwarg):
self.taskQueue.set((fun, args, kwarg))

def getResult(self):
return self.resQueue.get()

def probeResult(self):
return not self.resQueue.empty()
Rewrite of your examples:

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
worker = WorkerThread()
for i in range(10):
worker.setTask(p2.stdin.write, 'print 10'+'\n')
worker.setTask(p2.stdout.readline)
worker.getResult() # wait for write to finish
o,e = worker.getResult() # wait for read to finish
print o,e

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
writer = WorkerThread()
reader = WorkerThread()
for i in range(10):
writer.setTask(p2.stdin.write, 'print 10'+'\n')
reader.setTask(p2.stdout.readline)
o,e = reader.getResult() # wait for read to finish
print o,e
Jun 27 '08 #4
rd*****@gmail.com <rd*****@gmail.comwrote:
Hi,
I'm trying to perform following operation from inside the python
script
1. Open a shell ( start a process )
2. Send command1 to the process
3. Get output from the process
4. Send command2 to the process
5. Get output from the process
......
Following is sample code :

from subprocess import *
p2 = Popen('python',stdin=PIPE,stdout=PIPE,universal_ne wlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n')
o,e = p2.stdout.readline()
print o,e

It seems that stdout.readline() is a blocking read and it just gets
stuck their..
How to fix this ..
If you are working under linux/some-unix-like-os then use the python
expect module which is for exactly this sort of thing.

http://www.noah.org/wiki/Pexpect

Pexpect is a pure Python expect-like module. Pexpect makes Python a
better tool for controlling other applications.

Pexpect is a pure Python module for spawning child applications;
controlling them; and responding to expected patterns in their
output. Pexpect works like Don Libes' Expect. Pexpect allows your
script to spawn a child application and control it as if a human
were typing commands.

Pexpect can be used for automating interactive applications such as
ssh, ftp, passwd, telnet, etc. It can be used to a automate setup
scripts for duplicating software package installations on different
servers. It can be used for automated software testing. Pexpect is
in the spirit of Don Libes' Expect, but Pexpect is pure
Python. Unlike other Expect-like modules for Python, Pexpect does
not require TCL or Expect nor does it require C extensions to be
compiled. It should work on any platform that supports the standard
Python pty module. The Pexpect interface was designed to be easy to
use.

You'll never get it to work with subprocess like this because of the
buffering.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jun 27 '08 #5

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: KimTaehwan | last post by:
I want to be throw exception when one thread wait to read no data from a remote source for long time. This can be done in pre-1.4 enviroment like the following. ...
1
by: Peter Ammon | last post by:
I would like to read from a pipe in which data may arrive slowly. From experimenting, it looks like os.read() will block until it returns the maximum amount of data you asked for, in the second...
1
by: David Dvali | last post by:
Hello. What is nonblocking sockets and how can I ues it? Can you show me some seamlpe please?
2
by: David Helgason | last post by:
I was just wondering whether this was either: - supported, or - doable, either - using the normal poll-read-done? loop - a settable timeout It would be awfully nice. David Helgason, Over...
6
by: Rajat Katyal | last post by:
Hi: In postgres documentation its written that if we execute query as PERFORM query inside our stored procedure; then the special variable FOUND is set totrue if the query produced at least one...
9
by: dc2000 | last post by:
Hello everyone: I am normally not an advocate of increasing the priority of a running process or of a thread but it looks like I have to choice. I'm writing a small app that plays a line-in...
3
by: noswitchport | last post by:
Hi, I used C to write simple "udp" (not tcp) socket programming. The client will send the string to the server and server will just echo that string back to client. Everything went fine until I...
2
by: yawnmoth | last post by:
For learning purposes, I'd like to create a PHP script that'll output the system time every minute and, at the same time, will process "command line" commands as I type them into STDIN. eg. If...
0
by: mh | last post by:
So, I'm parsing a log file that's being written out in real time. <logfile> <entry><timestamp>123</timestamp><details>foo</details> </entry>...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: Aftab Ahmad | last post by:
Hello Experts! I have written a code in MS Access for a cmd called "WhatsApp Message" to open WhatsApp using that very code but the problem is that it gives a popup message everytime I clicked on...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.