Using subprocess module to launch a shell shell script that itselfforks a process

[Samuel A. Falvo II]

I have a shell script script.sh that launches a Java process in the
background using the &-operator, like so:

#!/bin/bash
java ... arguments here ... &

In my Python code, I want to invoke this shell script using the
Subprocess module. Here is my code:

def resultFromRunning_(command):
"""Invokes a shell command, and returns the stdout response.

Args:
command: A string containing the complete shell command to
run.

Results:
A string containing the output of the command executed.

Raises:
ValueError if a non-zero return code is returned from the
shell.
OSError if command isn't found, inappropriate permissions,
etc.
"""

L = log4py.Logger().get_instance()
L.info("Executing: " + command)

p = subprocess.Popen(
command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)

outputChannel = p.stdout
output = outputChannel.read()

result = p.wait()
if result:
raise(ShellError(command, result, output))

L.info("Result = " + str(output))
return output

When running the aforementioned code, it kicks off the shell script,
and, the shell script kicks off the Java process. However, the Python
code never returns from outputChannel.read() until I explicitly kill
the Java process myself via the kill shell command.

I've researched this issue on Google and various other websites, and
maybe I'm missing the obvious, but I cannot seem to find any
documentation relevant to this problem. Lots of references to bugs
filed in the past, that appear to be fixed, or to websites talking
about how the Popen module has a 64K limit on its data queue size, but
nothing relevent to my situation.

Can anyone inform me or point me to the appropriate documentation on
how to properly invoke a shell command such that any spawned children
processes don't cause Python to hang on me? I assume it has something
to do with process groups, but I'm largely ignorant of how to control
those.

Thanks in advance.

[Gabriel Genellina]

En Tue, 07 Oct 2008 21:43:41 -0300, Samuel A. Falvo II
<sa*******@gmail.comescribió:

I have a shell script script.sh that launches a Java process in the
background using the &-operator, like so:

#!/bin/bash
java ... arguments here ... &

In my Python code, I want to invoke this shell script using the
Subprocess module. Here is my code:

def resultFromRunning_(command):
"""Invokes a shell command, and returns the stdout response.

Args:
command: A string containing the complete shell command to
run.

Results:
A string containing the output of the command executed.

Raises:
ValueError if a non-zero return code is returned from the
shell.
OSError if command isn't found, inappropriate permissions,
etc.
"""

L = log4py.Logger().get_instance()
L.info("Executing: " + command)

p = subprocess.Popen(
command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)

outputChannel = p.stdout
output = outputChannel.read()

result = p.wait()
if result:
raise(ShellError(command, result, output))

L.info("Result = " + str(output))
return output

When running the aforementioned code, it kicks off the shell script,
and, the shell script kicks off the Java process. However, the Python
code never returns from outputChannel.read() until I explicitly kill
the Java process myself via the kill shell command.

Is your shell script doing something else, apart from invoking the java
process? If not, you could just invoke java directly from Python. Also,
you set stdin=PIPE - is your java process expecting some input? you're not
writing anything to stdin.
Anyway, it's better to use the communicate method instead (it uses select
to read from both stdout and stderr):

p = subprocess.Popen(...)
output = p.communicate()[0]
result = p.returncode
if result: ...

See
http://docs.python.org/library/subpr...en.communicate

--
Gabriel Genellina

[Samuel A. Falvo II]

On Oct 7, 6:23*pm, "Gabriel Genellina" <[e-mail]:

Is your shell script doing something else, apart from invoking the java *
process?

Obviously, yes. The script is some 150 lines long. But the hang-up
occurs because of the forked Java process, not the other lines.

If not, you could just invoke java directly from Python. Also, *
you set stdin=PIPE - is your java process expecting some input? you're not *
writing anything to stdin.

It does not expect input from stdin. However, this does not affect
any OTHER scripts or commands I run.

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. For all other
types of commands, it works 100% as expected.

Anyway, it's better to use the communicate method instead (it uses select*
to read from both stdout and stderr):

That doesn't help me.

See *http://docs.python.org/library/subpr...ss.Popen.commu...

I have.

[Samuel A. Falvo II]

On Oct 8, 11:24*am, "Samuel A. Falvo II" <[e-mail]:

It does not expect input from stdin. *However, this does not affect
any OTHER scripts or commands I run.

OK, so, I'm very confused as to why this would matter.

I removed the stdin=PIPE argument, and this works. Many thanks for
bringing this to my attention.

But, my question now is, WHY is this an issue? If the launched
process doesn't read from its stdin, why would it block? Hence my
question here:

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. *For all other
types of commands, it works 100% as expected.

Meaning, if I launched the Java process directly, it works fine. If I
launch it from the shell script WITHOUT background execution, it works
fine. But when I launch it WITH background execution (e.g., with the
& suffix), then it blocks.

Any ideas, so I can write this into my log for future reference?

Thanks.

[Samuel A. Falvo II]

On Oct 8, 11:31*am, "Samuel A. Falvo II" <[e-mail]:

I removed the stdin=PIPE argument, and this works. *Many thanks for
bringing this to my attention.

OK, I am confused. After observing a bug where the code works "every
other time", like clockwork, I've used strace to figure out what is
going on. In the times when "it works," it's only the shell script
complaining that it's already running.

So, I'm right back to square one. Even _without_ setting stdin=PIPE,
it fails to unblock. I'm left utterly bewildered.

The only thing I can think of is that the JVM's stdout is tied to the
shell script's stdout, because that's the only way it can remain open
upon the termination of the child process. I suppose, at this point,
the next place to look is in shell script syntax to find out how to
detach the JVM from the script's process group.

[Sean DiZazzo]

On Oct 8, 11:24*am, "Samuel A. Falvo II" <[e-mail]:

On Oct 7, 6:23*pm, "Gabriel Genellina" <[e-mail]:

Is your shell script doing something else, apart from invoking the java*
process?

Obviously, yes. *The script is some 150 lines long. *But the hang-up
occurs because of the forked Java process, not the other lines.

If not, you could just invoke java directly from Python. Also, *
you set stdin=PIPE - is your java process expecting some input? you're not *
writing anything to stdin.

It does not expect input from stdin. *However, this does not affect
any OTHER scripts or commands I run.

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. *For all other
types of commands, it works 100% as expected.

Anyway, it's better to use the communicate method instead (it uses select *
to read from both stdout and stderr):

That doesn't help me.

See *http://docs.python.org/library/subpr...ss.Popen.commu...

I have.

You should be nicer to Gabriel. He is a guru.

[Gabriel Genellina]

En Wed, 08 Oct 2008 15:24:39 -0300, Samuel A. Falvo II
<sa*******@gmail.comescribió:

On Oct 7, 6:23*pm, "Gabriel Genellina" <[e-mail]:

>you set stdin=PIPE - is your java process expecting some input? you're
not writing anything to stdin.

It does not expect input from stdin. However, this does not affect
any OTHER scripts or commands I run.

But it *does* affect how subprocess handles the child process internally.

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. For all other
types of commands, it works 100% as expected.

Don't conclude too fast... I've tested a variant (using a Perl script as a
replacement instead of a Java program), and it worked fine:

<log>
gabriel@sleipnir:~$ cat foo.sh
#!/bin/sh
perl foo.pl&
echo End of foo.sh

gabriel@sleipnir:~$ cat foo.pl
#!/usr/bin/perl

for ($i=5; $i>0; $i--) {
print "$i seconds remaining...\n";
sleep(1);
}
print "End of foo.pl\n";

gabriel@sleipnir:~$ cat foo.py
#!/usr/bin/python2.5

import subprocess

command = "./foo.sh"

p = subprocess.Popen(command,
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)
print "py: before p.communicate"
output = p.communicate()[0]
print "py: after p.communicate"
print "py: returncode=",p.returncode
print "py: read",len(output),"bytes"
print "py: output=", repr(output)
print "py: End of foo.py"
gabriel@sleipnir:~$ ./foo.py
py: before p.communicate
py: after p.communicate
py: returncode= 0
py: read 143 bytes
py: output= 'End of foo.sh\n5 seconds remaining...\n4 seconds
remaining...\n3 seconds remaining...\n2 seconds remaining...\n1 seconds
remaining...\nEnd of foo.pl\n'
py: End of foo.py
gabriel@sleipnir:~$ uname -a
Linux debian 2.6.18-4-486 #1 Mon Mar 26 16:39:10 UTC 2007 i686 GNU/Linux
gabriel@sleipnir:~$
</log>

This was tested both with python 2.4.4 and 2.5.2
I don't think it's a Python problem, but something related to java and how
it handles stdin/stdout. Try with another program.

But, my question now is, WHY is this an issue? If the launched
process doesn't read from its stdin, why would it block?

Several reasons - the child process might send enough text to stderr to
fill its buffer, and if the parent process doesn't read from the other end
of the pipe, the child blocks. That's why I suggested to use communicate
instead of stdout.read() - it takes care of such cases.

The only thing I can think of is that the JVM's stdout is tied to the
shell script's stdout, because that's the only way it can remain open
upon the termination of the child process. I suppose, at this point,
the next place to look is in shell script syntax to find out how to
detach the JVM from the script's process group.

Can't you redirect to a file instead?
java foo.jar >/tmp/foo.log 2>&1 &

--
Gabriel Genellina

[Derek Martin]

On Tue, Oct 07, 2008 at 05:43:41PM -0700, Samuel A. Falvo II wrote:

p = subprocess.Popen(
command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)

outputChannel = p.stdout
output = outputChannel.read()

You call read with no arguments. This is highly problematic in the
context of interprocess communication, unless you can be 100% positive
that none of the children will write anywhere besides STDOUT, and
won't try to read from STDIN in the meanwhile.

Python's read() with no args reads until the end of file, which in IPC
contexts is bad... Normally the child process won't close stdout
until it exits. So, if it did any other output in between, say, to
STDERR, the child will block waiting for the parent to read STDERR,
meanwhile the parent is blocked waiting for input from the child's
STDOUT, which results in a deadlock. Both processes sleep forever.
The exact same thing can happen if either the shell script or a
process started by the shell script tries to read from STDIN.

Since Java is launched by the shell script, it inherits the shell
script's STDIN, STDOUT, and STDERR file descriptors (i.e. the two
processes share the same STDIO). Thus if the java process writes to
STDERR, that also could be causing your deadlock.
On Wed, Oct 08, 2008 at 11:24:39AM -0700, Samuel A. Falvo II wrote:

On Oct 7, 6:23*pm, "Gabriel Genellina" <[e-mail]:

Is your shell script doing something else, apart from invoking the java*
process?

Obviously, yes.

It's far from obvious. I can't count the number of scripts I've seen
whose sole purpose was to launch a Java program (with the right
environment)... You would do well to avoid being dismissive and
listen to those who respond with help, when you are the one who
obviously doesn't understand the behavior you're getting, and you're
the one asking for help.

The script is some 150 lines long. But the hang-up
occurs because of the forked Java process, not the other lines.

I'm betting it's because the Java program is writing warnings to
STDERR (more data than will fit in the STDIO buffer), which you're not
reading...

If not, you could just invoke java directly from Python. Also, *
you set stdin=PIPE - is your java process expecting some input? you're not *
writing anything to stdin.

It does not expect input from stdin. However, this does not affect
any OTHER scripts or commands I run.

Irrelevant... Unless "any OTHER scripts" encompases all possible
combinations of process interactions, and you can demontstrate that it
does so, this proves nothing.

Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. For all other
types of commands, it works 100% as expected.

These are not facts which are in evidence. We don't know what your
script is doing, and it's clear that you yourself are not able to
explain the behavior you are seeing, therefore there is no reason for
us to conclude that the above statements are true and correct. Most
likely, they are not.

Anyway, it's better to use the communicate method instead (it uses select *
to read from both stdout and stderr):

That doesn't help me.

Please explain why it doesn't.

The most likely cause of the behavior you are seeing is the deadlock I
described, above. Using select() (i.e. using communicate()) should
generally fix about half the cases... The rest would be fixed by
redirecting STDIN of the child (or at least the Java process) from
/dev/null, most likely.

Then of course, there could be other explanations. But this being
overwhelmingly the most likely one, if you don't try those solutions,
there's no point in trying to help you further...

--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFI7mPcdjdlQoHP510RAoxzAJ9IhLxvbW7TyViMHG5zLM gm4bKsbgCgncD1
72+6DU1/j0q7jb9HV+66wsw=
=M4yo
-----END PGP SIGNATURE-----