does Popen.communicate() close all parent-side file descriptors?

[Roger Davis]

Hi,

I am brand-new to Python but am an experienced C/Unix programmer. I am rewriting in Python some old shell scripts that do lots of stuff like

set output = `cat this | grep that | whatever ...`

My sense is that this should be done using subprocess.Popen() since, although I am using Python 2.4 (Linux) and 2.6 (MacOS), all of the other library interfaces for this sort of thing are being deprecated/removed in future releases.

I have found a number of examples that show how to do this, including a very appropriate one on Doug Hellmann's website that shows how to chain a bunch of Popen()s together to mimic the pipeline example shown above. I have three questions, however:

(1) Does Popen.communicate() close all parent-side file descriptors opened by the Popen() call when the child exits? I am connecting the child's stdout to the parent, e.g.,

comm= subprocess.Popen(args, stdout=subprocess.PIPE)

This is being done in a loop run thousands of times so I need to make sure that each file descriptor opened in the parent to communicate with a child is closed as soon as I am finished with it.

(2) If I do not use Popen.communicate() (or if it does not close file descriptors), exactly which file descriptors do I need to close and how? I would assume the only things I need to close are any of the stdio streams for which I specified =PIPE in the Popen() call. For the above example would I just do

comm.stdout.close()

and not worry about anything else?

(3) Documentation on Popen.returncode is a bit vague. Is this the child's exit code only or is it the combination of exit code and signal status that is more fully documented in os.wait()?

Thanks!

[dwblas]

Python's closing of file handles is spotty. I ran into the same problem while using a class' __del__ method to close files. Python closes Python objects, and a file pointer/handle is not a Python object so it gets garbage collected but when is not necessarily known. I would say to use the Python equivalent of
`cat this | grep that | whatever ...`
instead of invoking a bash shell.

for fname in os.listdir(path_name):
    fp = open(os.path.join(path_name, fname)):
    for rec in fp:
        if search_string in rec:
            print rec
    fp.close() 

[Roger Davis]

Thanks for the reply, dwblas. I actually provided a poor example (`cat this | grep that | whatever`), there are many instances where I need to communicate with external programs to do things that Python cannot do internally.

I received a pretty comprehensive answer to this question from Chris Torek over on comp.lang.python who apparently researched the subprocess code before responding. According to him these parent-side pipe file descriptors are explicitly closed by Popen.communicate(). In cases where that routine is not used (and I have several), Popen()'s delete method will close them when it is called by the garbage collector. This seems borne out by my own testing, although since GC is unreliable I think I will be explicitly calling Popen.stdXXX.close() myself whenever I am done with the pipe on the parent side.

Thanks!

[dwblas]

Just wanted to make you aware of pexpect