Hi,
I am brand-new to Python but am an experienced C/Unix programmer. I am rewriting in Python some old shell scripts that do lots of stuff like
set output = `cat this | grep that | whatever ...`
My sense is that this should be done using subprocess.Pope n() since, although I am using Python 2.4 (Linux) and 2.6 (MacOS), all of the other library interfaces for this sort of thing are being deprecated/removed in future releases.
I have found a number of examples that show how to do this, including a very appropriate one on Doug Hellmann's website that shows how to chain a bunch of Popen()s together to mimic the pipeline example shown above. I have three questions, however:
(1) Does Popen.communica te() close all parent-side file descriptors opened by the Popen() call when the child exits? I am connecting the child's stdout to the parent, e.g.,
comm= subprocess.Pope n(args, stdout=subproce ss.PIPE)
This is being done in a loop run thousands of times so I need to make sure that each file descriptor opened in the parent to communicate with a child is closed as soon as I am finished with it.
(2) If I do not use Popen.communica te() (or if it does not close file descriptors), exactly which file descriptors do I need to close and how? I would assume the only things I need to close are any of the stdio streams for which I specified =PIPE in the Popen() call. For the above example would I just do
comm.stdout.clo se()
and not worry about anything else?
(3) Documentation on Popen.returncod e is a bit vague. Is this the child's exit code only or is it the combination of exit code and signal status that is more fully documented in os.wait()?
Thanks!
3 3498 dwblas 626
Recognized Expert Contributor
Python's closing of file handles is spotty. I ran into the same problem while using a class' __del__ method to close files. Python closes Python objects, and a file pointer/handle is not a Python object so it gets garbage collected but when is not necessarily known. I would say to use the Python equivalent of
`cat this | grep that | whatever ...`
instead of invoking a bash shell. - for fname in os.listdir(path_name):
-
fp = open(os.path.join(path_name, fname)):
-
for rec in fp:
-
if search_string in rec:
-
print rec
-
fp.close()
Thanks for the reply, dwblas. I actually provided a poor example (`cat this | grep that | whatever`), there are many instances where I need to communicate with external programs to do things that Python cannot do internally.
I received a pretty comprehensive answer to this question from Chris Torek over on comp.lang.pytho n who apparently researched the subprocess code before responding. According to him these parent-side pipe file descriptors are explicitly closed by Popen.communica te(). In cases where that routine is not used (and I have several), Popen()'s delete method will close them when it is called by the garbage collector. This seems borne out by my own testing, although since GC is unreliable I think I will be explicitly calling Popen.stdXXX.cl ose() myself whenever I am done with the pipe on the parent side.
Thanks!
dwblas 626
Recognized Expert Contributor
Just wanted to make you aware of pexpect Sign in to post your reply or Sign up for a free account.
Similar topics |
by: J Peterman |
last post by:
I need to do this exercise, but am having problems.
I need to write a program that firstly, sleeps for 5 seconds, then reads
a line of input from file descriptor 0 and then writes the line back to
file descriptor 1.
Apparrantly, this program would block forever unless you type a line of
text on the keyboard, but I can't get a working program to try.
Next, this program needs to be modified so that it would write its input
|
by: JG |
last post by:
Hi all,
Does anyone know how the implementations on Linux and Windows handle
synchronization between a read and write FD open to the same file.
For example, if I have 2 FD open to file X.txt. 1 I use for reading,
the other for writing. If I write to position 125 on the write_FD,
call flush, and then turn around and read from the read_FD position
125, am I guaranteed to get the result I just wrote?
|
by: upendrao |
last post by:
Hi,
I have a requirement in which I need to count the number
of file descriptors associated with a file. To be more clear...A file
can be opened by several processes say in read only mode. I would like
to count the number of such file opens on particular file at a given
instant of time. Any C library function or unix command that can solve
my problem please...?
|
by: Joe Lester |
last post by:
I'm trying to figure out what the optimal Postgres configuration would
be for my server (with 200 connecting clients, even though I'd really
like to get it up to 500).
I've got a 700 MHz eMac running Mac OS 10.3.2 (Panther) with 512 MB of
RAM. I've messed around with some settings but I'm still getting an
occasional "out of file descriptor" error, especially when performing a
VACUUM. Like so...
2004-04-13 23:30:05 LOG: out of file...
|
by: Didier Trosset |
last post by:
I am using the following code. It is used to spawn a new process (using
popen) and to change the file handles so that subsequent writes to
standard output with printf goes into this child process standard input.
import os
child_stdin = os.popen("cat -", "w")
old_stdout = os.dup(1)
os.close(child_stdin.fileno())
| |
by: xhy_China |
last post by:
How to close an file in an running process from another
process?
For example:
There are two running processes,named procIn and
procFrom. A file
named needCloseFile was opened in procIn. Now I want to
close the
needCloseFile from procFrom by programming,but left
procIn running normally.
What can I do???
|
by: jfigueiras |
last post by:
>I have a problem with the module subprocess!
As many other programs...
I'm not sure what you mean by "non-standard file descriptors". The
other program is free to open, read, write, etc any file he wants -
are you trying to trap any file operation it may want to do?
You *could* do such things -google for "code injection" and "API
hooking"- but I doubt it's what you really want.
|
by: Yang |
last post by:
Hi, I'm experiencing a problem when trying to close the file descriptor
for a socket, creating another socket, and then closing the file
descriptor for that second socket. I can't tell if my issue is about
Python or POSIX.
In the following, the first time through, everything works. On the
second connection, though, the same file descriptor as the first
connection may be re-used, but for some reason, trying to do
os.read/close on that...
|
by: DJ Dharme |
last post by:
Hi all,
I am writing a multi-threaded application in c++ running on
solaris. I have a file which is updated by a single thread by
appending data into the file and at same time the other threads are
reading the content written into the file. Can anybody tell me is
there a performance or any other gain (except for the multex locking)
by using different file descriptors in each thread for the same file
rather than using a single FD with a...
|
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |