thread safe SMTP module

In article <ma***************************************@python. org>,
Gordon Messmer <gm******@u.washington.eduwrote:

>
I believe that I've seen this discussed previously, so maybe there's
some interest in it. I wrote a threaded mail filtering framework
a while ago, and one of the modules does address verification
via SMTP. Since smtplib.SMTP uses blocking IO, it can block the
whole interpreter. Sometimes the whole thing would stop working
indefinitely.

That doesn't make any sense. Blocking I/O generally releases the GIL,
which is the whole reason Python doesn't totally suck for threading.
There may be other thread problems, but I doubt that you have correctly
analyzed their source. You can prove this for yourself by trying out my
threaded spider at
http://www.pythoncraft.com/OSCON2001...dPoolSpider.py
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"I disrespectfully agree." --SJM

Aahz wrote:

>
That doesn't make any sense. Blocking I/O generally releases the GIL,
which is the whole reason Python doesn't totally suck for threading.

Nevertheless, among the caveats listed at
http://docs.python.org/lib/module-thread.html is:

"Not all built-in functions that may block waiting for I/O allow other
threads to run. (The most popular ones (time.sleep(), file.read(),
select.select()) work as expected.)"

There may be other thread problems, but I doubt that you have correctly
analyzed their source.

I subclassed smtplib.SMTP and replaced only the lines of code that had
to do with blocking IO (connect, send and receive operations).
Beforehand, python would occasionally lock up. Having made those
changes, python stopped locking up. I think the problem was pretty well
apparent. I can't pin it down to which one of those three operations
was at fault, and it may be that only one was. However, when I use
non-blocking IO, the application works. When I used smtplib.SMTP, it
didn't.

I'm open to other explanations.

In article <ma***************************************@python. org>,
Gordon Messmer <yi*****@eburg.comwrote:

>Aahz wrote:

>>
That doesn't make any sense. Blocking I/O generally releases the GIL,
which is the whole reason Python doesn't totally suck for threading.

Nevertheless, among the caveats listed at
http://docs.python.org/lib/module-thread.html is:

"Not all built-in functions that may block waiting for I/O allow other
threads to run. (The most popular ones (time.sleep(), file.read(),
select.select()) work as expected.)"

That's why I said "generally".

>There may be other thread problems, but I doubt that you have correctly
analyzed their source.

I subclassed smtplib.SMTP and replaced only the lines of code that had
to do with blocking IO (connect, send and receive operations).
Beforehand, python would occasionally lock up. Having made those
changes, python stopped locking up. I think the problem was pretty well
apparent. I can't pin it down to which one of those three operations
was at fault, and it may be that only one was. However, when I use
non-blocking IO, the application works. When I used smtplib.SMTP, it
didn't.

I'm open to other explanations.

Assuming you have correctly tracked down the problem area, I would call
that a thread bug in Python. But my experience is that you simply have
run into a problem with the socket. I would suggest that using
socket.setdefaulttimeout() would work just as well.
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"I disrespectfully agree." --SJM

Aahz wrote:

>
Assuming you have correctly tracked down the problem area, I would call
that a thread bug in Python. But my experience is that you simply have
run into a problem with the socket. I would suggest that using
socket.setdefaulttimeout() would work just as well.

I believe that solution, also would not work. This note is included in
the socket documentation, regarding "timeout mode":

http://docs.python.org/lib/socket-objects.html
"A consequence of this is that file objects returned by the makefile()
method must only be used when the socket is in blocking mode; in timeout
or non-blocking mode file operations that cannot be completed
immediately will fail."

smtplib.SMTP uses file objects when reading SMTP responses. If I used
setdefaulttimeout(), then the socket would be in timeout mode and the
above note would be applicable.

I am not at all above calling python's behavior a bug, except that it
seemed like a known behavior given the note in the thread documentation
regarding built-in functions that block on I/O.

In article <ma***************************************@python. org>,
Gordon Messmer <yi*****@eburg.comwrote:

>Aahz wrote:

>>
Assuming you have correctly tracked down the problem area, I would call
that a thread bug in Python. But my experience is that you simply have
run into a problem with the socket. I would suggest that using
socket.setdefaulttimeout() would work just as well.

I believe that solution, also would not work. This note is included in
the socket documentation, regarding "timeout mode":

http://docs.python.org/lib/socket-objects.html
"A consequence of this is that file objects returned by the makefile()
method must only be used when the socket is in blocking mode; in timeout
or non-blocking mode file operations that cannot be completed
immediately will fail."

smtplib.SMTP uses file objects when reading SMTP responses. If I used
setdefaulttimeout(), then the socket would be in timeout mode and the
above note would be applicable.

Hrm. At this point, I would suggest taking discussion to python-dev; it
has been too long since I looked closely at thread/socket behavior.

>I am not at all above calling python's behavior a bug, except that it
seemed like a known behavior given the note in the thread documentation
regarding built-in functions that block on I/O.

No, at this point I think this is neither bug nor about thread blocking
on I/O. I think it's about sockets dying and the inability of sockets in
blocking mode to recover. I have seen this kind of behavior in
single-threaded systems. But it really needs someone who knows more than
I do, and I think the first step here is to go ahead and file a bug
report for tracking purposes.
--
Aahz (aa**@pythoncraft.com) <* http://www.pythoncraft.com/

"I disrespectfully agree." --SJM