469,955 Members | 1,412 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,955 developers. It's quick & easy.

popen hangs sporadically


I have a script that calls the function write_tmpfile, which looks
something like this:

def write_tmpfile(f, tmpfile):
# set-up code omitted
in_f = popen("""grep -v '^\\[eof\\]$' %s |\
grep '[^[:space:]]' |\
sort -u""" % f)
out_f = open(tmpfile, 'w')
while 1:
line = in_f.readline()
if not line: break
# i omit the code that munges line

The script calls this function several thousand times. (The average
size of the input file f is 70K lines (0.5MB); the maximum size is
about 35M lines, or 200MB.) This function works perfectly most of
the time, but it deadlocks sporadically. (And it's a deadlock!
The script can be stuck for hours, until I kill it.)

I can't say for sure where the deadlock is happening (and I'd
appreciate suggestions on how to pinpoint this), but I *think* it
is at the in_f.readline() statement. So maybe the problem is with
the pipe. (But FWIW, I've used exactly the same pipe in another
script that processes the same set of files (but does not write a
temporary file when it does this), and this script terminates
without any problem. I.e. the input files are not too large for
the pipe.)

I suppose that I could use some timeout mechanism to unwedge the
script when it deadlocks and then repeat the call to write_tmpfile,
but I'd prefer to avoid the deadlock in the first place.

I'd appreciate suggestions on how to troubleshoot and debug this



NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.
Aug 1 '08 #1
0 700

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

reply views Thread by Byrom, R (Rob) | last post: by
2 posts views Thread by Maarten van Veen | last post: by
1 post views Thread by Marko Rauhamaa | last post: by
1 post views Thread by Jean-Paul Calderone | last post: by
reply views Thread by Kenneth McDonald | last post: by
reply views Thread by Christian Heimes | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.