472,353 Members | 1,473 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,353 software developers and data experts.

popen hangs sporadically


I have a script that calls the function write_tmpfile, which looks
something like this:

def write_tmpfile(f, tmpfile):
# set-up code omitted
in_f = popen("""grep -v '^\\[eof\\]$' %s |\
grep '[^[:space:]]' |\
sort -u""" % f)
out_f = open(tmpfile, 'w')
while 1:
line = in_f.readline()
if not line: break
# i omit the code that munges line

The script calls this function several thousand times. (The average
size of the input file f is 70K lines (0.5MB); the maximum size is
about 35M lines, or 200MB.) This function works perfectly most of
the time, but it deadlocks sporadically. (And it's a deadlock!
The script can be stuck for hours, until I kill it.)

I can't say for sure where the deadlock is happening (and I'd
appreciate suggestions on how to pinpoint this), but I *think* it
is at the in_f.readline() statement. So maybe the problem is with
the pipe. (But FWIW, I've used exactly the same pipe in another
script that processes the same set of files (but does not write a
temporary file when it does this), and this script terminates
without any problem. I.e. the input files are not too large for
the pipe.)

I suppose that I could use some timeout mechanism to unwedge the
script when it deadlocks and then repeat the call to write_tmpfile,
but I'd prefer to avoid the deadlock in the first place.

I'd appreciate suggestions on how to troubleshoot and debug this



NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.
Aug 1 '08 #1
0 742

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Byrom, R (Rob) | last post by:
hi, When I use: os.popen('ssh -l root@something tomcat4 stop') I find the command always hangs. The 'something' host contains the correct key in...
by: David H | last post by:
Background. I'm running on WinXP w/ MS Services for Unix installed (to give rsh/rlogin ability), both Python 2.3 and 2.4 version. In linux, I'm...
by: Dan | last post by:
I've created a pocketpc app which has a startup form containing a listview. The form creates an object which in turn creates a...
by: Maarten van Veen | last post by:
A long story made short, I've build a python/cgi website consisting of two pages. Page1 has a html form in which you can input a series of queries....
by: gregpinero | last post by:
Let's say I have this Python file called loop.py: import sys print 'hi' sys.stdout.flush() while 1: pass And I want to call it from another...
by: Marko Rauhamaa | last post by:
This tiny program hangs: ======================================================================== #!/usr/bin/env python import subprocess a =...
by: Jean-Paul Calderone | last post by:
On Mon, 12 May 2008 17:35:44 -0700 (PDT), schickb <schickb@gmail.comwrote: Alas, you haven't actually closed src's standard input. Though you did...
by: Kenneth McDonald | last post by:
When making calls of the form Popen(cmd, shell=True, stdout=subprocess.PIPE), we've been getting occasional, predictable hangs. Will Popen...
by: Christian Heimes | last post by:
Kenneth McDonald wrote: The subprocess module has already an API method for your use case. The communicate() method of a subprocess.Popen...
by: Kemmylinns12 | last post by:
Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...
by: jalbright99669 | last post by:
Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...
by: antdb | last post by:
Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...
by: Matthew3360 | last post by:
Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...
by: AndyPSV | last post by:
HOW CAN I CREATE AN AI with an .executable file that would suck all files in the folder and on my computerHOW CAN I CREATE AN AI with an .executable...
by: WisdomUfot | last post by:
It's an interesting question you've got about how Gmail hides the HTTP referrer when a link in an email is clicked. While I don't have the specific...
by: Matthew3360 | last post by:
Hi, I have been trying to connect to a local host using php curl. But I am finding it hard to do this. I am doing the curl get request from my web...
by: Oralloy | last post by:
Hello Folks, I am trying to hook up a CPU which I designed using SystemC to I/O pins on an FPGA. My problem (spelled failure) is with the...
by: BLUEPANDA | last post by:
At BluePanda Dev, we're passionate about building high-quality software and sharing our knowledge with the community. That's why we've created a SaaS...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.