471,616 Members | 1,350 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,616 software developers and data experts.

Interprocess communication woes

Hi Python Gurus,

I am writing a GUI app (on linux) using pygtk which would launch some
external applications and display their stdout and stderr inside the
output window of my application synchronously. I am using the
subprocess module's Popen to launch the external programs and to
capture their stdout and stderr. The problem is that, for some
external programs that I launch inside my interface, I am not able to
capture and display the stdout as the program *runs*.

After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script

#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()

Run this program as follows
$ test.py "ping -c 10 www.google.com"

Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}

if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this

$python test.py "./timer"

you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.

Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.

Thanks!
Murali.

Jul 19 '07 #1
2 1691
Murali <mu**************@gmail.comwrote:
After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script

#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()

Run this program as follows
$ test.py "ping -c 10 www.google.com"

Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}

if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this

$python test.py "./timer"

you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.

Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.
Buffering is your problem.

If you add a fflush(stdout); after the printf(...); you'll find the
c++ program works as you expect.

It is just a fact of life of the C stdio system. If it is connected
to a terminal then it will turn off buffering. If it is connected
anything else (eg a pipe via subprocess) then it will buffer stuff as
you've seen.

So you can

a) modify the c++ prog to add fflush() in or use setvbuf()
b) use the pexpect module - http://pexpect.sourceforge.net/
c) use the pty module (unix only)

The pexpect module will connect to the subprogram with pseudo-ttys,
fooling the program, and the C library, into thinking that it is
speaking to a terminal and turn off buffering. Pexpect doesn't work
on windows.

The fact that ping works is because it uses fflush() - you can see
this if you "ltrace" it.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Jul 19 '07 #2
On Jul 19, 4:30 am, Nick Craig-Wood <n...@craig-wood.comwrote:
Murali <murali.dhanak...@gmail.comwrote:
After some investigation, I found out that this problem had nothing to
do with my GUI app not getting refreshed and I was able to reproduce
this problem with normal python scripts. Here is one such script
#File test.py
from subprocess import Popen
from subprocess import PIPE
import sys
if __name__ == '__main__':
prog = sys.argv[1]
proc = Popen(prog, shell = True, stdout = PIPE, bufsize = 0)
out = proc.stdout
while proc.poll() is None:
print out.readline()
Run this program as follows
$ test.py "ping -c 10www.google.com"
Now, you would see the responses as if you just launched ping on the
command line. Now, lets look at another program. Here is a simple C++
program that prints numbers 1 to 10 at the passage of every second
(sort of a stopwatch)
#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
main ( )
{
timeval t1, t2;
gettimeofday(&t1, NULL);
int prev = -1;
int cur = 0;
while (true)
{
gettimeofday(&t2,NULL);
if(t2.tv_sec - t1.tv_sec 10)
break;
else
{
cur = t2.tv_sec-t1.tv_sec;
if (cur != prev)
{
printf("%d\r\n",cur);
prev = cur;
}
}
}
}
if you would build this program and call it lets say timer ($ g++ -o
timer timer.cpp) and run it with our python script like this
$python test.py "./timer"
you would see that every time you run the program your results vary
and on top of this the stdout of the timer program gets displayed all
at once presumably when the timer program has completed execution.
Why this discrepancy between the ping and timer programs? Is my
test.py script correct? Is there a better or a preferred method for
doing interprocess communication in Python.

Buffering is your problem.

If you add a fflush(stdout); after the printf(...); you'll find the
c++ program works as you expect.

It is just a fact of life of the C stdio system. If it is connected
to a terminal then it will turn off buffering. If it is connected
anything else (eg a pipe via subprocess) then it will buffer stuff as
you've seen.

So you can
Thanks Nick. fflush fixed it. Thanks for your pointers on pexpect and
pty module too.

Murali.

a) modify the c++ prog to add fflush() in or use setvbuf()
b) use the pexpect module -http://pexpect.sourceforge.net/
c) use the pty module (unix only)

The pexpect module will connect to the subprogram with pseudo-ttys,
fooling the program, and the C library, into thinking that it is
speaking to a terminal and turn off buffering. Pexpect doesn't work
on windows.

The fact that ping works is because it uses fflush() - you can see
this if you "ltrace" it.

--
Nick Craig-Wood <n...@craig-wood.com--http://www.craig-wood.com/nick

Jul 20 '07 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

2 posts views Thread by Jeroen | last post: by
7 posts views Thread by Daniel | last post: by
4 posts views Thread by Charles Packer | last post: by
5 posts views Thread by guy | last post: by
3 posts views Thread by James Aguilar | last post: by
7 posts views Thread by Michael Butscher | last post: by
reply views Thread by Murali | last post: by
1 post views Thread by XIAOLAOHU | last post: by
reply views Thread by leo001 | last post: by
1 post views Thread by ZEDKYRIE | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.