problems with threaded socket app

[Gordon Messmer]

I've been working on a threaded daemon application to filter email. The
source for the program is here:

http://phantom.dragonsdawn.net/~gord...-pythonfilter/

The daemon loads individual filters as modules and hands the names of the
message and control files to each module in turn for processing. One of
the modules (filters/dialback.py) checks the address of the sender,
connects to the MX servers for the senders domain, and validates that the
sender address is valid. In order to implement a timeout on the dialback,
each message is processed by two threads. The first thread creates an
SMTP object and then starts a second thread to do the lookup using that
SMTP object. If the lookup takes too long, the first thread closes the
SMTP object's socket and collects the failure from the second thread.

During testing, that all works fine. However, in real world use, the
program eventually deadlocks. When it does so, there are several dialback
threads in process, and the first of each pair seems to be reading from
the status pipe. I cannot connect a debugger to the second of the pair to
see what state it's in.

I'm running this application on python2-2.2.2-11.7.3 under Red Hat Linux
7.3.

Does anyone have any suggestions for where I can start looking for the
problem?

[Anthony McDonald]

"Gordon Messmer" <go****@dragonsdawn.net> wrote in message
news:pa****************************@dragonsdawn.ne t...

I've been working on a threaded daemon application to filter email. The
source for the program is here:

http://phantom.dragonsdawn.net/~gord...-pythonfilter/
The daemon loads individual filters as modules and hands the names of the
message and control files to each module in turn for processing. One of
the modules (filters/dialback.py) checks the address of the sender,
connects to the MX servers for the senders domain, and validates that the
sender address is valid. In order to implement a timeout on the dialback,
each message is processed by two threads. The first thread creates an
SMTP object and then starts a second thread to do the lookup using that
SMTP object. If the lookup takes too long, the first thread closes the
SMTP object's socket and collects the failure from the second thread.

During testing, that all works fine. However, in real world use, the
program eventually deadlocks. When it does so, there are several dialback
threads in process, and the first of each pair seems to be reading from
the status pipe. I cannot connect a debugger to the second of the pair to
see what state it's in.

I'm running this application on python2-2.2.2-11.7.3 under Red Hat Linux
7.3.

Does anyone have any suggestions for where I can start looking for the
problem?

if rpipe not in ready_pipes[0]:
# Time to cancel this SMTP conversation
smtpi.close()
# The dialback thread will now write a failure message to
# its status pipe, and we'll need to clear that out.
os.read( rpipe, 1024 )
continue

The code creates a "race" condition. To work correctly it requires the
worker thread to raise and handle an exception, and to write that result
onto the pipe BEFORE your main thread attempts to read the pipe.

If the worker thread loses the race, the next MX result you process will
recieve the last MX's results 400 error code, and your left with 1 thread at
the end of the sequence which can't terminate as it stays active until what
its written to the pipe is read from the pipe.

Simple enough to fix, just add a select call between closing the SMTP
connection and reading the expected 400 error response.

Anthony McDonald

[Gordon Messmer]

On Mon, 15 Sep 2003 09:58:50 +0200, Anthony McDonald wrote:

if rpipe not in ready_pipes[0]:
# Time to cancel this SMTP conversation
smtpi.close()
# The dialback thread will now write a failure message to
# its status pipe, and we'll need to clear that out.
os.read( rpipe, 1024 )
continue

The code creates a "race" condition. To work correctly it requires the
worker thread to raise and handle an exception, and to write that result
onto the pipe BEFORE your main thread attempts to read the pipe.

If the worker thread loses the race, the next MX result you process will
recieve the last MX's results 400 error code, and your left with 1 thread at
the end of the sequence which can't terminate as it stays active until what
its written to the pipe is read from the pipe.

Simple enough to fix, just add a select call between closing the SMTP
connection and reading the expected 400 error response.

I can do that, but can you explain why, if the monitor thread loses the
race, it wouldn't block until the worker thread writes its status to the
pipe?

The code below should have the same race condition, but it works properly.

[root@deerhunter root]# python2
Python 2.2.2 (#1, Jan 30 2003, 21:26:22)
[GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import os
import thread
import time
def write_stat( wpipe ): .... time.sleep(20)
.... os.write( wpipe, 'my status' )
.... (rpipe, wpipe) = os.pipe()
thread.start_new_thread( write_stat, (wpipe,) ) 1026 test = os.read( rpipe, 1024)
print test my status