Hi,
I have a problem with a popen2 pipe hanging partway through execution of
a command. This happens in my function executeCommand() that is used
for every "shell" command execution in my program. Source code for this
function is below. My development environment is Debian unstable with
Python 2.3.5.
Within executeCommand(), output of every executed command is always
logged using a Python logger. If outputFile is passed in, then output
will also be written into the outputFile, which might have been created
using open(), gzip.GzipFile() or bz2.BZ2File(), etc. This functionality
exists so I have an equivalent of command-line redirection of stdout,
i.e. command > file.
If ignoreStderr=False, I use popen2.Popen4 so that stderr and stdout are
intermingled. If ignoreStderr=True, I use popen2.Popen3 with
capturestderr=True so stderr doesn't appear in the output. This
functionality exists so I have an equivalent of command-line redirection
of stderr, i.e. command 2>/dev/null.
I am using popen2 rather than popen because I want to avoid all of the
issues around shell interpolation of arguments, etc.
This whole thing has been working pretty well for a while. Then lately,
I started using it on commands that produce a large amount of output,
and it's begun to hang partway through execution. A good way to
reproduce this behavior on my system is to use command "ls -laR /". The
function hangs whever the output file reaches a certain size (around 20
MB). When I interrupt the program, the stack trace shows that it's
stuck at pipe.fromchild.readline().
After some digging, I've decided that this behavior probably occurs
because I am ignoring the pipe.childerr file object. Indeed, if I call
pipe.childerr.close() right after opening the pipe, my "ls" command that
had been hanging completes normally. However, other commands which
actually attempt to write to stderr don't seem to like this very much.
What is the right way to discard stderr when working with a pipe? I
want to consistently throw it away, and I don't see a good way to do
this with the popen2 implementation.
Thanks for the help,
KEN
------------------------------- snip -------------------------------
def executeCommand(command, args, returnOutput=False,
ignoreStderr=False, outputFile=None):
"""
@param command: Shell command to execute in a list like [ "ls", ]
@param args: List of arguments to the command, like [ "-laR", "/", ]
@param returnOutput: Indicates whether to return the output of the command
@param outputFile: File object that all output should be written to.
@return: Tuple of C{(result, output)}.
"""
output = []
fields = command[:]
fields.extend(args)
if ignoreStderr:
pipe = popen2.Popen3(fields, capturestderr=True)
else:
pipe = popen2.Popen4(fields)
pipe.tochild.close()
while True:
line = pipe.fromchild.readline()
if not line: break
if returnOutput: output.append(line)
if outputFile is not None: outputFile.write(line)
outputLogger.info(line[:-1])
if returnOutput:
return (pipe.wait(), output)
else:
return (pipe.wait(), None)
------------------------------- snip -------------------------------
--
Kenneth J. Pronovici <pr******@ieee.org>
