using subprocess for non-terminating command

[Phoe6]

Hi all,
Consider this scenario, where in I need to use subprocess to execute a
command like 'ping 127.0.0.1' which will have a continuous non-
terminating output in Linux.

# code

>>>import subprocess
process = subprocess.Popen('ping 127.0.0.1', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

>>>print process.pid # returns pid
print process.poll() # returns None!!! this is strange.
print process.stdout.read()

# This hangs at this point.
How should I handle these kind of commands (ping 127.0.0.1) with
subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Thanks,
Senthil

[zacherates]

How should I handle these kind of commands (ping 127.0.0.1) with

subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Ping, for one, allows you to set an upper bound on how long it runs
(the -c option). This is probably the cleanest approach if it's
available.

You can also send the subprocess signals if you need it to exit
(although, this is a unix thing so I'm not sure how portable it is).
You could emulate having a timeout on child.stdout.read by registering
a callback with Timer to kill the child.

Cheers,
Aaron

[O.R.Senthil Kumaran]

* zacherates <za********@gmail.com[2007-07-04 12:09:03]:

How should I handle these kind of commands (ping 127.0.0.1) with
subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Ping, for one, allows you to set an upper bound on how long it runs
(the -c option). This is probably the cleanest approach if it's
available.

Yes, I am aware of the ping -c option. But again even that does not help.
try
process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
shell=True)
process.stdout.read() # This will hang again.

I am not sure, why subprocess is behaving so.

You can also send the subprocess signals if you need it to exit
(although, this is a unix thing so I'm not sure how portable it is).

Yes, I have tried to kill and then get the standard output result.
But the result has been the same. I could not read the Popen returned file
object.

You could emulate having a timeout on child.stdout.read by registering
a callback with Timer to kill the child.

I dont know how to do this. I shall give it a try ( by looking around ) and
trying.

--
O.R.Senthil Kumaran
http://uthcode.sarovar.org

[Jerry Hill]

On 7/4/07, O.R.Senthil Kumaran <or*******@users.sourceforge.netwrote:

Yes, I am aware of the ping -c option. But again even that does not help.
try
process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
shell=True)
process.stdout.read() # This will hang again.

When I try that, it doesn't hang. Instead, I get the output of the
ping command pruinted to the screen, then the following exception:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'read'

That's because you tied stdin to a pipe in your Popen call, but then
tried to read from stdout. Try this instead:

>>process = subprocess.Popen("ping -c 10 127.0.0.1",

stdout=subprocess.PIPE, shell=True)

>>process.stdout.readlines()

['PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.\n', '64 bytes from
127.0.0.1: icmp_seq=1 ttl=64 time=0.049 ms\n', '64 bytes from
127.0.0.1: icmp_seq=2 ttl=64 time=0.033 ms\n', '64 bytes from
127.0.0.1: icmp_seq=3 ttl=64 time=0.040 ms\n', '64 bytes from
127.0.0.1: icmp_seq=4 ttl=64 time=0.033 ms\n', '64 bytes from
127.0.0.1: icmp_seq=5 ttl=64 time=0.033 ms\n', '64 bytes from
127.0.0.1: icmp_seq=6 ttl=64 time=0.030 ms\n', '64 bytes from
127.0.0.1: icmp_seq=7 ttl=64 time=0.032 ms\n', '64 bytes from
127.0.0.1: icmp_seq=8 ttl=64 time=0.028 ms\n', '64 bytes from
127.0.0.1: icmp_seq=9 ttl=64 time=0.030 ms\n', '64 bytes from
127.0.0.1: icmp_seq=10 ttl=64 time=0.039 ms\n', '\n', '--- 127.0.0.1
ping statistics ---\n', '10 packets transmitted, 10 received, 0%
packet loss, time 8991ms\n', 'rtt min/avg/max/mdev =
0.028/0.034/0.049/0.009 ms\n']

--
Jerry

[Steve Holden]

O.R.Senthil Kumaran wrote:

* zacherates <za********@gmail.com[2007-07-04 12:09:03]:

>>How should I handle these kind of commands (ping 127.0.0.1) with
subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Ping, for one, allows you to set an upper bound on how long it runs
(the -c option). This is probably the cleanest approach if it's
available.

Yes, I am aware of the ping -c option. But again even that does not help.
try
process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE,
shell=True)
process.stdout.read() # This will hang again.

I am not sure, why subprocess is behaving so.

>You can also send the subprocess signals if you need it to exit
(although, this is a unix thing so I'm not sure how portable it is).

Yes, I have tried to kill and then get the standard output result.
But the result has been the same. I could not read the Popen returned file
object.

>You could emulate having a timeout on child.stdout.read by registering
a callback with Timer to kill the child.

I dont know how to do this. I shall give it a try ( by looking around ) and
trying.

Is it possible your ping implementation outputs to stderr?

regards
Steve
--
Steve Holden [phone] [phone]
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
--------------- Asciimercial ------------------
Get on the web: Blog, lens and tag the Internet
Many services currently offer free registration
----------- Thank You for Reading -------------

[O.R.Senthil Kumaran]

* Jerry Hill <ma*********@gmail.com[2007-07-04 11:23:33]:

>
That's because you tied stdin to a pipe in your Popen call, but then
tried to read from stdout. Try this instead:

My mistake. I had just 'typed' the command in the mail itself and forgot to
include the stdin, stdout, and stderr and mentioned it as hung based on some
recollection.

>

>process = subprocess.Popen("ping -c 10 127.0.0.1",

stdout=subprocess.PIPE, shell=True)

>process.stdout.readlines()

I tried it again and found that giving the -c 10 returns a well defined
output.
Only when the program has executed and the output available, subprocess can
read through PIPE's stdout it seems ( not at any other time).
With killing, I loose the output.

>>process = subprocess.Popen('ping 10 127.0.0.1', stdin=subprocess.PIPE,

stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)

>>process.pid

3475

>>import os
import signal
os.kill(process.pid,signal.SIGINT)
process.stdout.read()

''

>># required output is lost!

--
O.R.Senthil Kumaran
http://uthcode.sarovar.org

[zacherates]

Only when the program has executed and the output available, subprocess can

read through PIPE's stdout it seems ( not at any other time).
With killing, I loose the output.

This is untrue.

>>process.stdout.read() # Blocks until end of stream.
process.stdout.read(1) # Reads one character, only blocks if that character is unavailable.

As such you can read the needed chars from the child's STDOUT one at a
time. For example:

import os
import signal
import subprocess
import threading
import sys

stop = False
ping = subprocess.Popen('ping 127.0.0.1', shell = True, stdout =
subprocess.PIPE)

def kill():
global stop
stop = True
os.kill(ping.pid, signal.SIGTERM)

threading.Timer(5, kill).start()

while not stop:
sys.stdout.write(ping.stdout.read(1))

This solution let's you read from the stdout of a program that may
never terminate and time out after a certain amount of time but it's
not pretty. It's unix specific and introduces threads into a program
that doesn't need them. I'd go with trying to limit the time the
child runs through command line options if at all possible.

Cheers,
Aaron

[prikar20]

On Jul 4, 12:29 pm, "O.R.Senthil Kumaran"
<[e-mail]:

* Jerry Hill <[e-mail][2007-07-04 11:23:33]:

That's because you tied stdin to a pipe in your Popen call, but then
tried to read from stdout. Try this instead:

My mistake. I had just 'typed' the command in the mail itself and forgot to
include the stdin, stdout, and stderr and mentioned it as hung based on some
recollection.

>>process = subprocess.Popen("ping -c 10 127.0.0.1",

stdout=subprocess.PIPE, shell=True)

>>process.stdout.readlines()

I tried it again and found that giving the -c 10 returns a well defined
output.
Only when the program has executed and the output available, subprocess can
read through PIPE's stdout it seems ( not at any other time).
With killing, I loose the output.

>process = subprocess.Popen('ping 10 127.0.0.1', stdin=subprocess.PIPE,

stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)

I think you meant ping -c 10 (and not ping 10). If you pass first arg
as string, are you sure you didn't get an exception? I get one if I
use 'ping -c 10 <ip-addr>' -- because looks like subprocess is
searching for an executable named as "ping -c 10 ..." (not just
'ping').
So I sent in a sequence and it all worked as expected (I didn't have
shell=True, but it shouldn't matter in getting the required output).

Try the sequence as first arg.

>>process = subprocess.Popen('ping -c 10 127.0.0.1', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/auto/xxxkarthikxxx/python251/lib/python2.5/subprocess.py",
line 593, in __init__
errread, errwrite)
File "/auto/xxxkarthikxxx/python251/lib/python2.5/subprocess.py",
line 1079, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

>>process = subprocess.Popen('ping -c 10 127.0.0.1'.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

>>print process

<subprocess.Popen object at 0xb7580aac>

>>print process.pid

13435

>>print process.stdout.read()

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.025 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.006 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.005 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.004 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.011 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.007 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.020 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.006 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.006 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.006 ms

--- 127.0.0.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 0.004/0.009/0.025/0.007 ms, pipe 2

>>>

-- Karthik

>

>process.pid

3475

>import os
import signal
os.kill(process.pid,signal.SIGINT)
process.stdout.read()

''

># required output is lost!

--
O.R.Senthil Kumaranhttp://uthcode.sarovar.org

[Karthik Gurusamy]

On Jul 4, 4:38 am, Phoe6 <[e-mail]:

Hi all,
Consider this scenario, where in I need to use subprocess to execute a
command like 'ping 127.0.0.1' which will have a continuous non-
terminating output in Linux.

# code

>>import subprocess
process = subprocess.Popen('ping 127.0.0.1', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print process.pid # returns pid
print process.poll() # returns None!!! this is strange.

It's expected behavior. It means the child process is still running.

>>print process.stdout.read()

# This hangs at this point.

This too is expected behavior. 'ping <ip-addr>' runs forever
generating continuous output. It doesn't stop by itself.

read() reads until there is data available in the file object. Thus it
doesn't ever finish since the child never stops generating data.

If you do read(n), then it reads n bytes and returns.

How should I handle these kind of commands (ping 127.0.0.1) with
subprocess module. I am using subprocess, instead of os.system because
at anypoint in time, I need access to stdout and stderr of execution.

Using subprocess is good. Just ensure your child stops data generation
at some point. For ping, you can use '-c <num>' or some other
application, you can try closing it's stdin (e.g. cat, bc, gdb)

Thanks,
Karthik

Thanks,
Senthil

[Jerry Hill]

On 7/4/07, O.R.Senthil Kumaran <or*******@users.sourceforge.netwrote:

Only when the program has executed and the output available, subprocess can
read through PIPE's stdout it seems ( not at any other time).
With killing, I loose the output.

I think you have to read the data from the process's stdout before you
kill it. If you want to process the output of the program before it's
done, do something like this:

import subprocess
process = subprocess.Popen("ping 127.0.0.1", stdout=subprocess.PIPE, shell=True)
for i in xrange(10):
line = process.stdout.readline()
print "line:", repr(line)

Then you can decide to kill the subprocess when you have enough data,
or whatever.

Also, even though the output from ping appears to be pretty easy to
read, there can be problems with buffering when you connect a pipe to
the stdout of some programs. The pexpect faq talks a little bit about
that - http://pexpect.sourceforge.net/#faq . If you're working under
a unix system, pexpect is really useful for automating interactive
programs that are otherwise difficult to work with.

--
Jerry