By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,035 Members | 1,388 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,035 IT Pros & Developers. It's quick & easy.

does Popen.communicate() close all parent-side file descriptors?

P: n/a

I am brand-new to Python but am an experienced C/Unix programmer. I am rewriting in Python some old shell scripts that do lots of stuff like

set output = `cat this | grep that | whatever ...`

My sense is that this should be done using subprocess.Popen() since, although I am using Python 2.4 (Linux) and 2.6 (MacOS), all of the other library interfaces for this sort of thing are being deprecated/removed in future releases.

I have found a number of examples that show how to do this, including a very appropriate one on Doug Hellmann's website that shows how to chain a bunch of Popen()s together to mimic the pipeline example shown above. I have three questions, however:

(1) Does Popen.communicate() close all parent-side file descriptors opened by the Popen() call when the child exits? I am connecting the child's stdout to the parent, e.g.,

comm= subprocess.Popen(args, stdout=subprocess.PIPE)

This is being done in a loop run thousands of times so I need to make sure that each file descriptor opened in the parent to communicate with a child is closed as soon as I am finished with it.

(2) If I do not use Popen.communicate() (or if it does not close file descriptors), exactly which file descriptors do I need to close and how? I would assume the only things I need to close are any of the stdio streams for which I specified =PIPE in the Popen() call. For the above example would I just do


and not worry about anything else?

(3) Documentation on Popen.returncode is a bit vague. Is this the child's exit code only or is it the combination of exit code and signal status that is more fully documented in os.wait()?

Oct 13 '10 #1
Share this Question
Share on Google+
3 Replies

Expert 100+
P: 624
Python's closing of file handles is spotty. I ran into the same problem while using a class' __del__ method to close files. Python closes Python objects, and a file pointer/handle is not a Python object so it gets garbage collected but when is not necessarily known. I would say to use the Python equivalent of
`cat this | grep that | whatever ...`
instead of invoking a bash shell.
Expand|Select|Wrap|Line Numbers
  1. for fname in os.listdir(path_name):
  2.     fp = open(os.path.join(path_name, fname)):
  3.     for rec in fp:
  4.         if search_string in rec:
  5.             print rec
  6.     fp.close() 
Oct 14 '10 #2

P: n/a
Thanks for the reply, dwblas. I actually provided a poor example (`cat this | grep that | whatever`), there are many instances where I need to communicate with external programs to do things that Python cannot do internally.

I received a pretty comprehensive answer to this question from Chris Torek over on comp.lang.python who apparently researched the subprocess code before responding. According to him these parent-side pipe file descriptors are explicitly closed by Popen.communicate(). In cases where that routine is not used (and I have several), Popen()'s delete method will close them when it is called by the garbage collector. This seems borne out by my own testing, although since GC is unreliable I think I will be explicitly calling Popen.stdXXX.close() myself whenever I am done with the pipe on the parent side.

Oct 14 '10 #3

Expert 100+
P: 624
Just wanted to make you aware of pexpect
Oct 14 '10 #4

Post your reply

Sign in to post your reply or Sign up for a free account.