473,289 Members | 1,953 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,289 software developers and data experts.

Help need with subprocess communicate

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
Jun 27 '08 #1
8 9127
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rd*****@gmail.com wrote:
>I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Jun 27 '08 #2
On Jun 3, 5:42 pm, Daniel Klein <dani...@featherbrain.netwrote:
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output
But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......
Following is my code:
from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')
Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

following is a simple example of problem..please try running it ...

from subprocess import *
p2 =
Popen('python',shell=True,stdin=PIPE,stdout=PIPE,u niversal_newlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n') # Write Command
o = p2.stdout.readline() # Read Command
print o
I appreciate all your help...

Thanks,
-Rahul Dabane.
Jun 27 '08 #3
On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netcom.com wulfr...@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria.com)
HTTP://www.bestiaria.com/

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
Jun 27 '08 #4
On Jun 4, 9:56*am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

* * * * I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * wlfr...@ix.netcom.com * * * * * * *wulfr...@bestiaria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria.com)
* * * * * * * * HTTP://www.bestiaria.com/
starting your application with "python -u" sets it to unbuffered
binary stdout and stderr, hopefully that helps.
Jun 27 '08 #5
On Jun 3, 11:04 pm, rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
[...]
Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Here's what you need: http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti

Jun 27 '08 #6
Ali
On Jun 3, 10:04*pm, rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read *\n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
It sounds like this may help: http://www.noah.org/wiki/Pexpect

(pure python expect-like functionality)

Ali
Jun 27 '08 #7
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rd*****@gmail.com wrote:
>On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
>On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netcom.com wulfr...@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria.com)
HTTP://www.bestiaria.com/


Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
I'm not so sure it is a buffer problem. To test this out I first created a
'p2.py' script...

from subprocess import *
import os
p=Popen('ConsoleApplication1.exe',stdin=PIPE,stdou t=PIPE,universal_newlines=True)
print p.stdout.readline()[:-1] # strip \n from end of line
p.stdin.write('hi' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.write('bye' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.close()
p.stdout.close()

I then created the following VB console application (this is the 'process'
that is being 'Popen'd and is in my %PATH%)...

Module Module1
Dim x As String
Sub Main()
Console.WriteLine("Process started...")
x = Console.ReadLine()
Console.WriteLine(x)
x = Console.ReadLine()
Console.WriteLine(x)
End Sub
End Module

Here is the output when I run it...

C:\home\python>python p2.py
Process started...
hi
bye

Note that I didn't have to 'flush()' anything.

I got the same thing working with a C program. I don't know why it won't
work with a similar python script...

import sys
sys.stdout.write('process started...\n')
r = sys.stdin.readline()
sys.stdout.write(r + '\n')
s = sys.stdin.readline()
sys.stdout.write(s + '\n')

I called this 'p3.py'. When I plug this into the 'p2.py' script I get
nothing, it just hangs. So maybe there is something else I am missing.

I normally don't do things this way cos there are os size limits to what you
can send/recv, so I use my own protocol (similar to netstrings) for
communication.

Daniel Klein
Jun 27 '08 #8
On Jun 4, 9:56 am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
Which is why the Cookbook recipe I pointed to in my other message
appears to be a superior alternative. Its author works around this
problem by subclassing Popen to use non blocking I/O.

I used it to drive ClearCase's cleartool interactive tool and it works
very well. The only problem I encountered is that it doesn't mix well
with framework that have their own event loop (wxPython in my case),
but you can't really expect that to work.

Cheers,
Nicola Musatti
Jun 27 '08 #9

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: James Hong | last post by:
Help please, I try to sending an email from my html page using the java applet. but it give error on most of the PC only very few work, what is the error i make the java applet show as below ...
0
by: Steven Green | last post by:
I need some help to get a training grant. If you are responsible for hiring Oracle professionals please read on. I want to become an Oracle DBA and am seeking to take a DBA certificate program....
4
by: Scott Holland | last post by:
HELP - Need to connect to DB2 database on AIX from NT server. Also AS/400 from NT Server -- I am experienced in ORACLE and a novice at DB2. What tools would be the equivalent of Net*8 or...
3
by: amanda | last post by:
Hope someone can help me with this - I've been staring at it stupidly for hours now, convinced there must be an easy way to achieve the results I want: I have a very large table recording every...
4
by: Web_PDE_Eric | last post by:
I don't know where to go, or what to buy, so plz re-direct me if I'm in the wrong place. I want to do high performance integration of partial differential eqns in n dimensions (n=0,1,2,3..etc) I...
0
by: riflefire | last post by:
Hello Everyone, I am using the new MITE c# text game engine to code up a MUD style game. I need a lot of help as i am new to C#. The reason i need the help is that the engine requires c# scripts to...
1
by: mavester | last post by:
Hi all, i need some help in order for me to startup my project. Here is the device and dscription i stated below: s/w needed: vs2003 or vs 2005 language: any code behind for which is suitable ...
0
by: rdabane | last post by:
I'm trying to perform following type of operation from inside a python script. 1. Open an application shell (basically a tcl ) 2. Run some commands on that shell and get outputs from each command...
2
by: Clinton M James | last post by:
Hi All, I have set up an access database that I communicate with through Excel and VBA. The purpose is to do auditing of staff. Because there are several audits, each audi has differing...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: MeoLessi9 | last post by:
I have VirtualBox installed on Windows 11 and now I would like to install Kali on a virtual machine. However, on the official website, I see two options: "Installer images" and "Virtual machines"....
0
by: DolphinDB | last post by:
The formulas of 101 quantitative trading alphas used by WorldQuant were presented in the paper 101 Formulaic Alphas. However, some formulas are complex, leading to challenges in calculation. Take...
0
by: Aftab Ahmad | last post by:
So, I have written a code for a cmd called "Send WhatsApp Message" to open and send WhatsApp messaage. The code is given below. Dim IE As Object Set IE =...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: marcoviolo | last post by:
Dear all, I would like to implement on my worksheet an vlookup dynamic , that consider a change of pivot excel via win32com, from an external excel (without open it) and save the new file into a...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.