By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,035 Members | 1,388 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,035 IT Pros & Developers. It's quick & easy.

does Popen.communicate() close all parent-side file descriptors?

P: n/a
Hi,

I am brand-new to Python but am an experienced C/Unix programmer. I am rewriting in Python some old shell scripts that do lots of stuff like

set output = `cat this | grep that | whatever ...`

My sense is that this should be done using subprocess.Popen() since, although I am using Python 2.4 (Linux) and 2.6 (MacOS), all of the other library interfaces for this sort of thing are being deprecated/removed in future releases.

I have found a number of examples that show how to do this, including a very appropriate one on Doug Hellmann's website that shows how to chain a bunch of Popen()s together to mimic the pipeline example shown above. I have three questions, however:

(1) Does Popen.communicate() close all parent-side file descriptors opened by the Popen() call when the child exits? I am connecting the child's stdout to the parent, e.g.,

comm= subprocess.Popen(args, stdout=subprocess.PIPE)

This is being done in a loop run thousands of times so I need to make sure that each file descriptor opened in the parent to communicate with a child is closed as soon as I am finished with it.

(2) If I do not use Popen.communicate() (or if it does not close file descriptors), exactly which file descriptors do I need to close and how? I would assume the only things I need to close are any of the stdio streams for which I specified =PIPE in the Popen() call. For the above example would I just do

comm.stdout.close()

and not worry about anything else?

(3) Documentation on Popen.returncode is a bit vague. Is this the child's exit code only or is it the combination of exit code and signal status that is more fully documented in os.wait()?

Thanks!
Oct 13 '10 #1
Share this Question
Share on Google+
3 Replies


Expert 100+
P: 624
Python's closing of file handles is spotty. I ran into the same problem while using a class' __del__ method to close files. Python closes Python objects, and a file pointer/handle is not a Python object so it gets garbage collected but when is not necessarily known. I would say to use the Python equivalent of
`cat this | grep that | whatever ...`
instead of invoking a bash shell.
Expand|Select|Wrap|Line Numbers
  1. for fname in os.listdir(path_name):
  2.     fp = open(os.path.join(path_name, fname)):
  3.     for rec in fp:
  4.         if search_string in rec:
  5.             print rec
  6.     fp.close() 
Oct 14 '10 #2

P: n/a
Thanks for the reply, dwblas. I actually provided a poor example (`cat this | grep that | whatever`), there are many instances where I need to communicate with external programs to do things that Python cannot do internally.

I received a pretty comprehensive answer to this question from Chris Torek over on comp.lang.python who apparently researched the subprocess code before responding. According to him these parent-side pipe file descriptors are explicitly closed by Popen.communicate(). In cases where that routine is not used (and I have several), Popen()'s delete method will close them when it is called by the garbage collector. This seems borne out by my own testing, although since GC is unreliable I think I will be explicitly calling Popen.stdXXX.close() myself whenever I am done with the pipe on the parent side.

Thanks!
Oct 14 '10 #3

Expert 100+
P: 624
Just wanted to make you aware of pexpect
Oct 14 '10 #4

Post your reply

Sign in to post your reply or Sign up for a free account.