By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
443,660 Members | 1,102 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 443,660 IT Pros & Developers. It's quick & easy.

Help need with subprocess communicate

P: n/a
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
Jun 27 '08 #1
Share this Question
Share on Google+
8 Replies


P: n/a
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rd*****@gmail.com wrote:
>I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Jun 27 '08 #2

P: n/a
On Jun 3, 5:42 pm, Daniel Klein <dani...@featherbrain.netwrote:
On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output
But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......
Following is my code:
from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')
Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

following is a simple example of problem..please try running it ...

from subprocess import *
p2 =
Popen('python',shell=True,stdin=PIPE,stdout=PIPE,u niversal_newlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n') # Write Command
o = p2.stdout.readline() # Read Command
print o
I appreciate all your help...

Thanks,
-Rahul Dabane.
Jun 27 '08 #3

P: n/a
On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netcom.com wulfr...@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria.com)
HTTP://www.bestiaria.com/

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
Jun 27 '08 #4

P: n/a
On Jun 4, 9:56*am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

* * * * I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * wlfr...@ix.netcom.com * * * * * * *wulfr...@bestiaria.com
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * web-a...@bestiaria.com)
* * * * * * * * HTTP://www.bestiaria.com/
starting your application with "python -u" sets it to unbuffered
binary stdout and stderr, hopefully that helps.
Jun 27 '08 #5

P: n/a
On Jun 3, 11:04 pm, rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell
[...]
Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.
Here's what you need: http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti

Jun 27 '08 #6

P: n/a
Ali
On Jun 3, 10:04*pm, rdab...@gmail.com wrote:
I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read *\n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.
It sounds like this may help: http://www.noah.org/wiki/Pexpect

(pure python expect-like functionality)

Ali
Jun 27 '08 #7

P: n/a
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rd*****@gmail.com wrote:
>On Jun 3, 11:23 pm, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
>On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
wlfr...@ix.netcom.com wulfr...@bestiaria.com
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: web-a...@bestiaria.com)
HTTP://www.bestiaria.com/


Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?
I'm not so sure it is a buffer problem. To test this out I first created a
'p2.py' script...

from subprocess import *
import os
p=Popen('ConsoleApplication1.exe',stdin=PIPE,stdou t=PIPE,universal_newlines=True)
print p.stdout.readline()[:-1] # strip \n from end of line
p.stdin.write('hi' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.write('bye' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.close()
p.stdout.close()

I then created the following VB console application (this is the 'process'
that is being 'Popen'd and is in my %PATH%)...

Module Module1
Dim x As String
Sub Main()
Console.WriteLine("Process started...")
x = Console.ReadLine()
Console.WriteLine(x)
x = Console.ReadLine()
Console.WriteLine(x)
End Sub
End Module

Here is the output when I run it...

C:\home\python>python p2.py
Process started...
hi
bye

Note that I didn't have to 'flush()' anything.

I got the same thing working with a C program. I don't know why it won't
work with a similar python script...

import sys
sys.stdout.write('process started...\n')
r = sys.stdin.readline()
sys.stdout.write(r + '\n')
s = sys.stdin.readline()
sys.stdout.write(s + '\n')

I called this 'p3.py'. When I plug this into the 'p2.py' script I get
nothing, it just hangs. So maybe there is something else I am missing.

I normally don't do things this way cos there are os size limits to what you
can send/recv, so I use my own protocol (similar to netstrings) for
communication.

Daniel Klein
Jun 27 '08 #8

P: n/a
On Jun 4, 9:56 am, Dennis Lee Bieber <wlfr...@ix.netcom.comwrote:
On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rdab...@gmail.com declaimed the
following in comp.lang.python:
Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
Which is why the Cookbook recipe I pointed to in my other message
appears to be a superior alternative. Its author works around this
problem by subclassing Popen to use non blocking I/O.

I used it to drive ClearCase's cleartool interactive tool and it works
very well. The only problem I encountered is that it doesn't mix well
with framework that have their own event loop (wxPython in my case),
but you can't really expect that to work.

Cheers,
Nicola Musatti
Jun 27 '08 #9

This discussion thread is closed

Replies have been disabled for this discussion.