473,545 Members | 1,638 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Help need with subprocess communicate

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl' ,stdin=PIPE,std out=PIPE)
o,e = p2.communicate( input='qdl_help \n qdl_read \n
qdl_reg_group_l ist ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
Jun 27 '08 #1
8 9145
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rd*****@gmail.c om wrote:
>I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl' ,stdin=PIPE,std out=PIPE)
o,e = p2.communicate( input='qdl_help \n qdl_read \n
qdl_reg_group_ list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Use
stdin.write(com mand + '\n')
to 'send' data to the sub-process.

Use
stdout.readline ()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newli nes = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newli nes) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Jun 27 '08 #2
On Jun 3, 5:42 pm, Daniel Klein <dani...@feathe rbrain.netwrote :
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rdab...@gmail.c om wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output
But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......
Following is my code:
from subprocess import *
p2 = Popen('qdl_tcl' ,stdin=PIPE,std out=PIPE)
o,e = p2.communicate( input='qdl_help \n qdl_read \n
qdl_reg_group_l ist ')
Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Use
stdin.write(com mand + '\n')
to 'send' data to the sub-process.

Use
stdout.readline ()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newli nes = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newli nes) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

following is a simple example of problem..please try running it ...

from subprocess import *
p2 =
Popen('python', shell=True,stdi n=PIPE,stdout=P IPE,universal_n ewlines=True)
for i in range(10):
p2.stdin.write( 'print 10'+'\n') # Write Command
o = p2.stdout.readl ine() # Read Command
print o
I appreciate all your help...

Thanks,
-Rahul Dabane.
Jun 27 '08 #3
On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.c om declaimed the
following in comp.lang.pytho n:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netc om.com wulfr...@bestia ria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria. com)
HTTP://www.bestiaria.com/

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
Jun 27 '08 #4
On Jun 4, 9:56*am, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.c om declaimed the
following in comp.lang.pytho n:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

* * * * I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * wlfr...@ix.netc om.com * * * * * * *wulfr...@besti aria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria. com)
* * * * * * * * HTTP://www.bestiaria.com/
starting your application with "python -u" sets it to unbuffered
binary stdout and stderr, hopefully that helps.
Jun 27 '08 #5
On Jun 3, 11:04 pm, rdab...@gmail.c om wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
[...]
Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl' ,stdin=PIPE,std out=PIPE)
o,e = p2.communicate( input='qdl_help \n qdl_read \n
qdl_reg_group_l ist ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Here's what you need: http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti

Jun 27 '08 #6
Ali
On Jun 3, 10:04*pm, rdab...@gmail.c om wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl' ,stdin=PIPE,std out=PIPE)
o,e = p2.communicate( input='qdl_help \n qdl_read *\n
qdl_reg_group_l ist ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
It sounds like this may help: http://www.noah.org/wiki/Pexpect

(pure python expect-like functionality)

Ali
Jun 27 '08 #7
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rd*****@gmail.c om wrote:
>On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
>On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.c om declaimed the
following in comp.lang.pytho n:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netc om.com wulfr...@bestia ria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria. com)
HTTP://www.bestiaria.com/


Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
I'm not so sure it is a buffer problem. To test this out I first created a
'p2.py' script...

from subprocess import *
import os
p=Popen('Consol eApplication1.e xe',stdin=PIPE, stdout=PIPE,uni versal_newlines =True)
print p.stdout.readli ne()[:-1] # strip \n from end of line
p.stdin.write(' hi' + os.linesep)
print p.stdout.readli ne()[:-1]
p.stdin.write(' bye' + os.linesep)
print p.stdout.readli ne()[:-1]
p.stdin.close()
p.stdout.close( )

I then created the following VB console application (this is the 'process'
that is being 'Popen'd and is in my %PATH%)...

Module Module1
Dim x As String
Sub Main()
Console.WriteLi ne("Process started...")
x = Console.ReadLin e()
Console.WriteLi ne(x)
x = Console.ReadLin e()
Console.WriteLi ne(x)
End Sub
End Module

Here is the output when I run it...

C:\home\python> python p2.py
Process started...
hi
bye

Note that I didn't have to 'flush()' anything.

I got the same thing working with a C program. I don't know why it won't
work with a similar python script...

import sys
sys.stdout.writ e('process started...\n')
r = sys.stdin.readl ine()
sys.stdout.writ e(r + '\n')
s = sys.stdin.readl ine()
sys.stdout.writ e(s + '\n')

I called this 'p3.py'. When I plug this into the 'p2.py' script I get
nothing, it just hangs. So maybe there is something else I am missing.

I normally don't do things this way cos there are os size limits to what you
can send/recv, so I use my own protocol (similar to netstrings) for
communication.

Daniel Klein
Jun 27 '08 #8
On Jun 4, 9:56 am, Dennis Lee Bieber <wlfr...@ix.net com.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.c om declaimed the
following in comp.lang.pytho n:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
Which is why the Cookbook recipe I pointed to in my other message
appears to be a superior alternative. Its author works around this
problem by subclassing Popen to use non blocking I/O.

I used it to drive ClearCase's cleartool interactive tool and it works
very well. The only problem I encountered is that it doesn't mix well
with framework that have their own event loop (wxPython in my case),
but you can't really expect that to work.

Cheers,
Nicola Musatti
Jun 27 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
9858
by: James Hong | last post by:
Help please, I try to sending an email from my html page using the java applet. but it give error on most of the PC only very few work, what is the error i make the java applet show as below ********************************** package Celcom.Client;
0
2379
by: Steven Green | last post by:
I need some help to get a training grant. If you are responsible for hiring Oracle professionals please read on. I want to become an Oracle DBA and am seeking to take a DBA certificate program. In order to obtain the grant to pay for the training I need to show a demand for the skills I want to study. The department of employment and...
4
5574
by: Scott Holland | last post by:
HELP - Need to connect to DB2 database on AIX from NT server. Also AS/400 from NT Server -- I am experienced in ORACLE and a novice at DB2. What tools would be the equivalent of Net*8 or SQLNET in ORACLE that would allow me to connect to a database on a remote box (either AIX or AS/400)? I have read that DB2 Connect appears to be the...
3
2801
by: amanda | last post by:
Hope someone can help me with this - I've been staring at it stupidly for hours now, convinced there must be an easy way to achieve the results I want: I have a very large table recording every sale made by one of 90 salespeople (2 years of data), and the date the sale was made. I need a report that shows a list of the salespeople down the...
4
2839
by: Web_PDE_Eric | last post by:
I don't know where to go, or what to buy, so plz re-direct me if I'm in the wrong place. I want to do high performance integration of partial differential eqns in n dimensions (n=0,1,2,3..etc) I want to do (fast) graphic output of the results. I used to use C. I want to upgrade my computer. Do I get dual core? Does C# support dual-core?...
0
1791
by: riflefire | last post by:
Hello Everyone, I am using the new MITE c# text game engine to code up a MUD style game. I need a lot of help as i am new to C#. The reason i need the help is that the engine requires c# scripts to be written to run the mud. It also has a remote gui, code editor and other neat features. The mud i envision is a Science Fiction with a sub genre of...
1
1181
by: mavester | last post by:
Hi all, i need some help in order for me to startup my project. Here is the device and dscription i stated below: s/w needed: vs2003 or vs 2005 language: any code behind for which is suitable Device1: PC with serial port Device2: GSM modem (Maestro 100) Description1: PC initiate a msg to GSM modem to send a sms to mobile phone....
0
769
by: rdabane | last post by:
I'm trying to perform following type of operation from inside a python script. 1. Open an application shell (basically a tcl ) 2. Run some commands on that shell and get outputs from each command 3. Close the shell I could do it using communicate if I concatenate all my commands ( separated by newline ) and read all the output in the...
2
1156
by: Clinton M James | last post by:
Hi All, I have set up an access database that I communicate with through Excel and VBA. The purpose is to do auditing of staff. Because there are several audits, each audi has differing criteria to mark on, therefore the vba program I have made is dynamic in that the user can construct as many marking criteria as needed.
0
7459
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7653
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
0
7803
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7411
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7749
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
5965
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
1
5322
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
4942
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3439
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.