By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
464,833 Members | 1,188 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 464,833 IT Pros & Developers. It's quick & easy.

popen hangs sporadically

P: n/a

I have a script that calls the function write_tmpfile, which looks
something like this:

def write_tmpfile(f, tmpfile):
# set-up code omitted
in_f = popen("""grep -v '^\\[eof\\]$' %s |\
grep '[^[:space:]]' |\
sort -u""" % f)
out_f = open(tmpfile, 'w')
while 1:
line = in_f.readline()
if not line: break
# i omit the code that munges line

The script calls this function several thousand times. (The average
size of the input file f is 70K lines (0.5MB); the maximum size is
about 35M lines, or 200MB.) This function works perfectly most of
the time, but it deadlocks sporadically. (And it's a deadlock!
The script can be stuck for hours, until I kill it.)

I can't say for sure where the deadlock is happening (and I'd
appreciate suggestions on how to pinpoint this), but I *think* it
is at the in_f.readline() statement. So maybe the problem is with
the pipe. (But FWIW, I've used exactly the same pipe in another
script that processes the same set of files (but does not write a
temporary file when it does this), and this script terminates
without any problem. I.e. the input files are not too large for
the pipe.)

I suppose that I could use some timeout mechanism to unwedge the
script when it deadlocks and then repeat the call to write_tmpfile,
but I'd prefer to avoid the deadlock in the first place.

I'd appreciate suggestions on how to troubleshoot and debug this



NOTE: In my address everything before the first period is backwards;
and the last period, and everything after it, should be discarded.
Aug 1 '08 #1
Share this question for a faster answer!
Share on Google+

This discussion thread is closed

Replies have been disabled for this discussion.