Help need with subprocess communicate

[rdabane]

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.

[Daniel Klein]

On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), rd*****@gmail.com wrote:

>I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein

[rdabane]

On Jun 3, 5:42 pm, Daniel Klein <[e-mail]:

On Tue, 3 Jun 2008 14:04:10 -0700 (PDT), [e-mail] wrote:

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Use
stdin.write(command + '\n')
to 'send' data to the sub-process.

Use
stdout.readline()
to 'receive' data from the sub-process.

But to use this requires you open the subprocess with:

universal_newlines = True

It assumes that 'command' will be sent with '\n' and received data will come
in a line at a time. Your Python program needs to know what to expect; you
are in control.

Alternatively, you can use std.write() and stdout.read() (without
universal_newlines) but this means you need to create your own IPC protocol
(like netstrings).

Hope this helps,

Daniel Klein

Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

following is a simple example of problem..please try running it ...

from subprocess import *
p2 =
Popen('python',shell=True,stdin=PIPE,stdout=PIPE,u niversal_newlines=True)
for i in range(10):
p2.stdin.write('print 10'+'\n') # Write Command
o = p2.stdout.readline() # Read Command
print o
I appreciate all your help...

Thanks,
-Rahul Dabane.

[rdabane]

On Jun 3, 11:23 pm, Dennis Lee Bieber <[e-mail]:

On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), [e-mail] declaimed the
following in comp.lang.python:

Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
[e-mail] [e-mail]
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: [e-mail])
HTTP://www.bestiaria.com/

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

[Chris]

On Jun 4, 9:56*am, Dennis Lee Bieber <[e-mail]:

On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), [e-mail] declaimed the
following in comp.lang.python:

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

* * * * I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.
--
* * * * Wulfraed * * * *Dennis Lee Bieber * * * * * * * KD6MOG
* * * * [e-mail] * * * * * * *[e-mail]
* * * * * * * * HTTP://wlfraed.home.netcom.com/
* * * * (Bestiaria Support Staff: * * * * * * * [e-mail])
* * * * * * * * HTTP://www.bestiaria.com/

starting your application with "python -u" sets it to unbuffered
binary stdout and stderr, hopefully that helps.

[Nicola Musatti]

On Jun 3, 11:04 pm, [e-mail] wrote:

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

[...]

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read \n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Here's what you need: http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti

[Ali]

On Jun 3, 10:04*pm, [e-mail] wrote:

I'm trying to perform following type of operation from inside a python
script.
1. Open an application shell (basically a tcl )
2. Run some commands on that shell and get outputs from each command
3. Close the shell

I could do it using communicate if I concatenate all my commands
( separated by newline ) and read all the output in the end. So
basically I could do following sequence:
1. command1 \n command2 \n command 3 \n
2. Read all the output

But I want to perform it interactively.
1. command1
2. read output
3. command2
4. read output ......

Following is my code:

from subprocess import *
p2 = Popen('qdl_tcl',stdin=PIPE,stdout=PIPE)
o,e = p2.communicate(input='qdl_help \n qdl_read *\n
qdl_reg_group_list ')

Please suggest a way to perform it interactively with killing the
process each time I want to communicate with it.

Thanks in advance,
-Rahul.

It sounds like this may help: http://www.noah.org/wiki/Pexpect

(pure python expect-like functionality)

Ali

[Daniel Klein]

On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), rd*****@gmail.com wrote:

>On Jun 3, 11:23 pm, Dennis Lee Bieber <[e-mail]:

>On Tue, 3 Jun 2008 18:04:40 -0700 (PDT), [e-mail] declaimed the
following in comp.lang.python:

Hi Daniel,
Thanks for your reply..
I've done exactly as you suggested...but I'm still having problem with
the read...it just gets stuck in
the read ( I think because its a blocking read...)

And it is likely blocking because the subprocess is doing buffered
output -- ie, nothing is available to be read because the output has not
been flushed.

This is a problem with most programs when run as a subprocess -- it
is common for stdout, when routed to a pipe or file, to behave as a
buffered stream that only flushes when some x-bytes have been written;
unlike stdout to a console which gets flushed on each new-line.
--
Wulfraed Dennis Lee Bieber KD6MOG
[e-mail] [e-mail]
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: [e-mail])
HTTP://www.bestiaria.com/

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

I'm not so sure it is a buffer problem. To test this out I first created a
'p2.py' script...

from subprocess import *
import os
p=Popen('ConsoleApplication1.exe',stdin=PIPE,stdou t=PIPE,universal_newlines=True)
print p.stdout.readline()[:-1] # strip \n from end of line
p.stdin.write('hi' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.write('bye' + os.linesep)
print p.stdout.readline()[:-1]
p.stdin.close()
p.stdout.close()

I then created the following VB console application (this is the 'process'
that is being 'Popen'd and is in my %PATH%)...

Module Module1
Dim x As String
Sub Main()
Console.WriteLine("Process started...")
x = Console.ReadLine()
Console.WriteLine(x)
x = Console.ReadLine()
Console.WriteLine(x)
End Sub
End Module

Here is the output when I run it...

C:\home\python>python p2.py
Process started...
hi
bye

Note that I didn't have to 'flush()' anything.

I got the same thing working with a C program. I don't know why it won't
work with a similar python script...

import sys
sys.stdout.write('process started...\n')
r = sys.stdin.readline()
sys.stdout.write(r + '\n')
s = sys.stdin.readline()
sys.stdout.write(s + '\n')

I called this 'p3.py'. When I plug this into the 'p2.py' script I get
nothing, it just hangs. So maybe there is something else I am missing.

I normally don't do things this way cos there are os size limits to what you
can send/recv, so I use my own protocol (similar to netstrings) for
communication.

Daniel Klein

[Nicola Musatti]

On Jun 4, 9:56 am, Dennis Lee Bieber <[e-mail]:

On Tue, 3 Jun 2008 23:48:38 -0700 (PDT), [e-mail] declaimed the
following in comp.lang.python:

Is there way to configure the stdout buffer size so that it flushes
earlier..
Is there a way to make above mentioned piece code working?

I believe there is a command line argument that will set Python into
an unbuffered mode... BUT, unless the spawned process/program has some
similar option, you have no control over its output.

Which is why the Cookbook recipe I pointed to in my other message
appears to be a superior alternative. Its author works around this
problem by subclassing Popen to use non blocking I/O.

I used it to drive ClearCase's cleartool interactive tool and it works
very well. The only problem I encountered is that it doesn't mix well
with framework that have their own event loop (wxPython in my case),
but you can't really expect that to work.

Cheers,
Nicola Musatti