[Python 2.4/2.5] subprocess module is sorely deficient?

[Harishankar]

Hi,

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)

2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Paul Boddie]

On 22 Apr, 12:52, Harishankar <[e-mail]:

>
Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

If you want some hints about using subprocesses with non-blocking I/O,
you might find some in my jailtools and pprocess projects:

http://www.python.org/pypi/jailtools
http://www.python.org/pypi/pprocess

Although these projects involve things which are not exactly cross-
platform, the communications mechanisms should be portable, perhaps
with a bit of effort (since I don't recall whether the poll library
function is available on Windows, so you might have to use the select
function instead). It can be awkward sustaining non-blocking
communications with processes if they use buffered I/O, and the only
way I could make Python-based subprocesses work in jailtools was to
invoke them with the unbuffered option (-u).

My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.

The subprocess module may be an improvement over the popen2 module and
various os module functions, but it's still rather arcane.

Paul

[Harishankar]

On Tuesday 22 Apr 2008 17:06:26 Paul Boddie wrote:

On 22 Apr, 12:52, Harishankar <[e-mail]:

Is there any way to use non-blocking Popen objects using subprocess? and
2 - is there a way to kill the subprocess in a platform independent
manner in a purely Pythonic way? I thought initially that this problem is
simple enough, but over the last couple of days I've been really
struggling to find any answer. I've been through dozens of mailing list
archives in to find a solution. Unfortunately none of the solutions seem
to fit my needs.

If you want some hints about using subprocesses with non-blocking I/O,
you might find some in my jailtools and pprocess projects:

http://www.python.org/pypi/jailtools
http://www.python.org/pypi/pprocess

Thank you. I will take a look at those. Actually I feel a mechanism like this
should be built-in to Python in the future.

Although these projects involve things which are not exactly cross-
platform, the communications mechanisms should be portable, perhaps
with a bit of effort (since I don't recall whether the poll library
function is available on Windows, so you might have to use the select
function instead). It can be awkward sustaining non-blocking
communications with processes if they use buffered I/O, and the only
way I could make Python-based subprocesses work in jailtools was to
invoke them with the unbuffered option (-u).

My only solution seems to be to offer the end user the mencoder command
line and make them execute it manually and be done with it but that seems
a rather weak solution.

The subprocess module may be an improvement over the popen2 module and
various os module functions, but it's still rather arcane.

Yes. I am quite sure there must be an elegant solution to the subprocess
handling/management. Problem is I've been at this for three days and I'm
getting quite bleary eyed trying to pore through a lot of documentation ;-)

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Nicola Musatti]

On Apr 22, 12:52 pm, Harishankar <[e-mail]:

Hi,

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)

I suggest you check out this: http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti

[Nick Craig-Wood]

Harishankar <v.***********@gmail.comwrote:

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)

2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

You are correct on both of those points. Subprocess isn't for
interacting with subprocesses - this should be written in large
letters in the help!

Is there any way to use non-blocking Popen objects using
subprocess?

There is a recipe in the cookbook

http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Which I've used and it works.

you can also (if on unix) use

http://www.noah.org/wiki/Pexpect

I think the best solution would be to port Pexpect to windows which
wouldn't be that difficult according to my reading of the code. If
only I had more free time!

and 2 - is there a way to kill the subprocess in a platform
independent manner in a purely Pythonic way? I thought initially
that this problem is simple enough, but over the last couple of
days I've been really struggling to find any answer. I've been
through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

No...

This is the best I came up with to add to the subprocess recipe above

import os
from subprocess import *
from subprocess import mswindows
from time import sleep

if mswindows:
import win32api
else:
import signal

class PopenNB(Popen):
# - see cookbook recipe for rest of stuff
# ...
def kill(self, killpg=False):
"""
Kill the running process
"""
pid = self.pid
if mswindows:
# Kill the process using win32api and pid - ignore errors
try:
PROCESS_TERMINATE = 1
handle = win32api.OpenProcess(PROCESS_TERMINATE, False, pid)
win32api.TerminateProcess(handle, -1)
win32api.CloseHandle(handle)
except pywintypes.error:
pass
else:
# Kill the process by sending the pid / process group a
signal
if killpg:
try:
pgid = os.getpgid(pid)
except OSError:
killpg = False
try:
if killpg:
os.killpg(pgid, signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except OSError:
return
sleep(1.0)
try:
if killpg:
os.killpg(pgid, signal.SIGKILL)
else:
os.kill(pid, signal.SIGKILL)
except OSError:
return

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[Harishankar]

On Tuesday 22 Apr 2008 18:00:02 Nick Craig-Wood wrote:

There is a recipe in the cookbook

http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Which I've used and it works.

Thanks. I found that recipe too. I was hoping I could cook up something
similar without having to use the module win32api, but looks like that's not
the case.

>
you can also (if on unix) use

http://www.noah.org/wiki/Pexpect

I'm on Linux. Debian. Pexpect would do the job fine too. The only thing is
it's a third party module and so would reduce the portability of my
application. But never mind. I guess I have to make a compromise one way or
the other.

import os
from subprocess import *
from subprocess import mswindows
from time import sleep

if mswindows:
import win32api
else:
import signal

class PopenNB(Popen):
# - see cookbook recipe for rest of stuff
# ...
def kill(self, killpg=False):
"""
Kill the running process
"""
pid = self.pid
if mswindows:
# Kill the process using win32api and pid - ignore errors
try:
PROCESS_TERMINATE = 1
handle = win32api.OpenProcess(PROCESS_TERMINATE, False,
pid) win32api.TerminateProcess(handle, -1)
win32api.CloseHandle(handle)
except pywintypes.error:
pass
else:
# Kill the process by sending the pid / process group a
signal
if killpg:
try:
pgid = os.getpgid(pid)
except OSError:
killpg = False
try:
if killpg:
os.killpg(pgid, signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except OSError:
return
sleep(1.0)
try:
if killpg:
os.killpg(pgid, signal.SIGKILL)
else:
os.kill(pid, signal.SIGKILL)
except OSError:
return

Thanks for this bit of code. It should probably be adaptible to my needs.

By the way, the win32api seems to be a nonstandard module (i.e. not present in
the main distribution).

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Harishankar]

On Tuesday 22 Apr 2008 17:54:00 Nicola Musatti wrote:

I suggest you check out this:
http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Cheers,
Nicola Musatti
--
http://mail.python.org/mailman/listinfo/python-list

An interesting solution. Thanks a lot for the link.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Tim Golden]

Harishankar wrote:

On Tuesday 22 Apr 2008 18:00:02 Nick Craig-Wood wrote:

>There is a recipe in the cookbook

http://aspn.activestate.com/ASPN/Coo.../Recipe/440554

Which I've used and it works.

Thanks. I found that recipe too. I was hoping I could cook up something
similar without having to use the module win32api...

Well if you want to, you can reproduce the same effect by using ctypes
which *is* in the standard library. But why reinvent the wheel?

By the way, the win32api seems to be a nonstandard module (i.e. not present in
the main distribution).

Correct. It's part of the pywin32 extensions, one of many useful packages available
to the discerning Python programmer who doesn't feel in some way bound to
whatever comes bundled with the standard library.

TJG

[Harishankar]

On Tuesday 22 Apr 2008 19:02:17 Tim Golden wrote:

Well if you want to, you can reproduce the same effect by using ctypes
which *is* in the standard library. But why reinvent the wheel?

The reason is once again, rightly or wrongly I feel that using non-standard
extensions could make it:

1. Difficult to distribute the application as I am not able to package the
third-party extension with distutils.
2. Difficult to predict its behaviour with future versions of Python.

Correct. It's part of the pywin32 extensions, one of many useful packages
available to the discerning Python programmer who doesn't feel in some way
bound to whatever comes bundled with the standard library.

TJG

I wouldn't feel "bound" if I restricted the program to myself. But if I want
to distribute it (as I intend to) I have to think of others as well.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[A.T.Hofkamp]

On 2008-04-22, Harishankar <v.***********@gmail.comwrote:

Hi,

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

1. Create non-blocking pipes which can be read in a separate thread (I am

I don't know about threading, but you can read/write streams in a non-blocking
way in Python (under Linux) with the os.read/os.write functions (these map
directly to read(2)/write(2) primitives). You should first check that something
can be read/written. Use eg select.select() for this. Several GUI toolkits also
allow monitoring of file handles.

Whether this also works at other OSes, I don't know.
Alternatively, you can create your own pipes, make them non-blocking, and give
the file handles to subprocess.

2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

The concept of sub-process is platform dependent already.

Usually however, closing the child input stream is sufficient for most child
programs to decide that they can quit.

Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.

Interfacing with the rest of the world implies you are going to need services
provided by the OS at one time or another. Python is rather transparent here
and gives you quick access to the OS services.

While this is bad for uniformity, it is good to let people make their own
choices in optimally using the OS services. Python helps here by giving
light-weight access to the underlying OS, making explicit what part is OS
dependent.

(and if still in doubt, why do you think were all the solutions you found 'not
fitting'?).

My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.

This is always a good fallback.
Sincerely,
Albert

[Mark Wooding]

Nick Craig-Wood <ni**@craig-wood.comwrote:

Harishankar <v.***********@gmail.comwrote:

> 1. Create non-blocking pipes which can be read in a separate thread
[...]

You are correct on both of those points.

I must be missing something. What's wrong with spawning the subprocess
with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/
whatever, making your end nonblocking with fcntl.fcntl and then using
os.read/os.write in the obvious ways?

-- [mdw]

[sturlamolden]

On Apr 22, 12:52 pm, Harishankar <[e-mail]:

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to

Have you looked at the processing module in cheese shop?

[Nick Craig-Wood]

Mark Wooding <md*@distorted.org.ukwrote:

Nick Craig-Wood <ni**@craig-wood.comwrote:

Harishankar <v.***********@gmail.comwrote:

1. Create non-blocking pipes which can be read in a separate thread
[...]

You are correct on both of those points.

I must be missing something. What's wrong with spawning the subprocess
with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/
whatever, making your end nonblocking with fcntl.fcntl and then using
os.read/os.write in the obvious ways?

Nothing apart from the fact it doesn't work on windows. The buffering
will cause you grief too. If you want to do this properly under unix
use pexpect not subprocess.

http://www.noah.org/wiki/Pexpect

Proper non blocking IO is an absolute nightmare under Windows in my
experience! It really isn't the Windows way so you are fighting the
system the whole time.

--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[Christian Heimes]

Nick Craig-Wood schrieb:

Nothing apart from the fact it doesn't work on windows. The buffering
will cause you grief too. If you want to do this properly under unix
use pexpect not subprocess.

http://www.noah.org/wiki/Pexpect

Proper non blocking IO is an absolute nightmare under Windows in my
experience! It really isn't the Windows way so you are fighting the
system the whole time.

Nick is correct. The subproces tries to work around the issues with
threads. But it's no more than an ugly workaround fir Windows' short
comings on async file IO. It's a shame Windows implements the select()
syscall in wsock32 and limits its usage to sockets.

By the way I'm willing to dedicate some time to help enhancing the
subprocess. Everybody is invited to submit patches and I'll review and
check them into the trunk and py3k ASAP. Any help is appreciated:
enhancements for async IO, doc updates, more examples ...

Christian
Python core developer

[=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=]

2. Kill the subprocess in a platform independent manner (i.e. no third party

modules and no hacks).

What's wrong with the .terminate method of the Popen object?

Regards,
Martin

[Christian Heimes]

Martin v. Löwis schrieb:

>2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

What's wrong with the .terminate method of the Popen object?

It's brand new ;)

Christian

[Christian Heimes]

Martin v. Löwis schrieb:

>2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).

What's wrong with the .terminate method of the Popen object?

It's brand new ;)

Christian

[Mike Hansen]

I think the best solution would be to port Pexpect to windows which

wouldn't be that difficult according to my reading of the code. If
only I had more free time!

Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to
interface with all sorts of other systems. We recently received
funding from Microsoft to do a native port of Sage (and all of its
components to Windows. Part of this will most likely be a port of
pexpect to Windows.

--Mike

[Heikki Toivonen]

Harishankar wrote:

Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:

At OSAF we used a slightly modified killableprocess module with a
wrapper to deal with complexities of various redirections in
cross-platform way. I actually blogged about this a week ago so rather
than rehash the issues I'll point you to the article which contains
links to all the pieces we used:

http://www.heikkitoivonen.net/blog/2...dered-harmful/

--
Heikki Toivonen

[Harishankar]

On Wednesday 23 Apr 2008 02:25:14 Christian Heimes wrote:

Nick Craig-Wood schrieb:

Nothing apart from the fact it doesn't work on windows. The buffering
will cause you grief too. If you want to do this properly under unix
use pexpect not subprocess.

http://www.noah.org/wiki/Pexpect

Proper non blocking IO is an absolute nightmare under Windows in my
experience! It really isn't the Windows way so you are fighting the
system the whole time.

Nick is correct. The subproces tries to work around the issues with
threads. But it's no more than an ugly workaround fir Windows' short
comings on async file IO. It's a shame Windows implements the select()
syscall in wsock32 and limits its usage to sockets.

By the way I'm willing to dedicate some time to help enhancing the
subprocess. Everybody is invited to submit patches and I'll review and
check them into the trunk and py3k ASAP. Any help is appreciated:
enhancements for async IO, doc updates, more examples ...

Christian
Python core developer

Thanks a lot to everybody who's been following this discussion. Very
interesting indeed.

I'm currently thinking of working around this problem by actually opening a
new terminal window and running the command from there, thus allowing the
user full control over the process.

Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Harishankar]

On Wednesday 23 Apr 2008 08:27:01 Heikki Toivonen wrote:

At OSAF we used a slightly modified killableprocess module with a
wrapper to deal with complexities of various redirections in
cross-platform way. I actually blogged about this a week ago so rather
than rehash the issues I'll point you to the article which contains
links to all the pieces we used:

http://www.heikkitoivonen.net/blog/2...m-considered-h
armful/

killableprocess.py looks like a good solution indeed. I actually came across
your website in my searches. I just wanted to be absolutely sure that it
worked because you had mentioned that it has some drawbacks.

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org

[Christian Heimes]

Harishankar schrieb:

Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?

No, there isn't. It usually not possible to create a graphical terminal
window on a remote server.

Christian

[Nick Craig-Wood]

Mike Hansen <mh*****@gmail.comwrote:

I think the best solution would be to port Pexpect to windows which
wouldn't be that difficult according to my reading of the code. If
only I had more free time!

Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to
interface with all sorts of other systems. We recently received
funding from Microsoft to do a native port of Sage (and all of its
components to Windows. Part of this will most likely be a port of
pexpect to Windows.

Hooray!
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick

[Harishankar]

On Wednesday 23 Apr 2008 14:46:20 Christian Heimes wrote:

Harishankar schrieb:

Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?

No, there isn't. It usually not possible to create a graphical terminal
window on a remote server.

Christian

Ah, well, since my application is a desktop tool and it requires a GUI I'm
doing something like this:

However, I have to then force the user to use xterm (which is a popular/common
X Terminal)

if (sys.platform.startswith ('win'):
# launch the windows cmd.exe with the command
...
else:
# warn the user that xterm is required and then launch xterm
...

--
Regards,
V. Harishankar

http://hari.literaryforums.org
http://harishankar.org