HowTo send emails reliably in a Python multi-threading application?

[Gernot Hillier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

I'm the developer of a Linux ISDN application which uses embedded Python for
controlling the communication. It starts several threads (i.e. one for each
incoming call and for outgoing faxes) which run Python scripts in embedded
Python interpreters which in turn do the real communication stuff. Any
incoming data or confirmations of done jobs are sent via Email.

Most real hard problems I had in the last year were either related to sending
emails or calling external tools via Pipes. I'll now try to roughly describe
the structure of CapiSuite and the problems I experienced. I'll try to do it
as shortly but understandably as possible. As all of it experiences
"sometimes" I can't provide small, clear test cases, sorry... :-((

All of the stuff coming now boils down to two questions:

1) How can one send mails reliable in a multi-threaded Python application?
Should the email package be used? Should smtplib be used or is it better to
call "sendmail" or "mail" directly? Should popen2 or os.system() be
preferred?

2) How to call an external tool (or many of them) and connect them via pipes
with each other and with the Python script in a multithreaded application? I
read the discussion http://www.python.org/doc/current/lib/
popen2-flow-control.html several times now and think I followed the
suggestions given there, but it seems still to cause trouble for me.

For those of you having some seconds, here's the rough description of my
problems...

I'm using normal linuxthreads on Linux 2.4.

Currently I have two mail send functions in use:

a) sendMIMEMail which first of all does some format conversion using external
commandline tools using popen2 and then constructs a mail using
email.MIMEText, email.MIMEBase or email.MIMEAudio. This mail is sent via the
"sendmail" commandline tool like this:

sendmail = popen2.Popen3("sendmail -t -f %s" % escape(mail_from))
if (sendmail.poll()!=-1):
...
sendmail.tochild.write(msg.as_string())
sendmail.tochild.close()
sendmail.fromchild.close()
ret=sendmail.wait()

b) sendSimpleMail which creates a simple mail using email.MIMEText and then
does the same as sendMIMEMail for sending it.

The problems I had so far:

- - when sendSimpleMail and sendMIMEMail are run nearly in parallel, thenthe
system sometimes gets stuck with one thread which is showed in "ps ax" but
does not exist according to gdb. It seems to be a try of one Python script to
run "sendmail" - but it gets stuck either before the new process was really
replaced by the started sendmail or after sendmail finished but before the
thread is destroyed. This can be worked around according to one user by not
using the mail package but sending the mail via
os.system("echo -e "+escape(text)+" | mail -s "+escape(mail_subject)+ " -r
"+escape(mail_from)+" "+escape(mail_to)) in sendSimpleMail

- - sometimes users (mainly Red Hat users) got the error message "cannot
unmarshal code objects in restricted execution mode" when the mail text was
encoded to ASCII. This one was reliably worked around by calling "import
encodings.ascii" at the begin of send*Mail.
(see http://lists.berlios.de/pipermail/ca...ly/000169.html
and http://lists.berlios.de/pipermail/ca...ly/000175.html
for details)

- - when a long document was converted by calling several commandline tools
(format1->format2->format3 via 2 tools), then popen2-pipes between process 1
and 2 were overflowed. This was worked around by using shell pipes between
process1 + 2 (popen2.Popen3("command1 | command2"))

- - very, very seldom (1 out of 5000 received faxes), one user of me saw the
following - at least with Python 2.2.2; I have no feedback about newer
versions yet:
File "/usr/lib/python2.2/site-packages/cs_helpers.py", line 233, in
sendMIMEMail
sendmail.tochild.write(msg.as_string())
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Message.py", line
107, in as_string
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
100, in flatten
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
135, in _write
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
167, in _write_headers
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
191, in _split_header
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
41, in _is8bitstring
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/encodings/__init__.py",
line 43, in search_function
AttributeError: 'NoneType' object has no attribute 'get'

So to summarize it: my current approach to do the format conversions and the
mail sending seems to be quite unstable, but I have no idea how to do it
better. But I'm quite sure there must be a reliable way to do these kinds of
things in Python. :-)) Perhaps there's also a general problem with my Python
embedding stuff - but as everything else works very nice, I doubt this is the
case...

I'll again have to excuse for the rough problem descriptions I can give yet
(and the complete miss of small test cases but it's really hard to debug...),
but I hope some of you could comment on my issues nevertheless. Any hint is
greatly appreciated!!!

TIA!!!

- --
Bye,

Gernot
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQFAN01Ok997/GGeSeIRAiALAJ9lGCv2RnbZpgklXx/mPYjpd5M+DACdHJsE
/RSiWpe7mU4u9Lp9qsKp2CQ=
=tnPN
-----END PGP SIGNATURE-----

[Josiah Carlson]

> 1) How can one send mails reliable in a multi-threaded Python application?

Should the email package be used? Should smtplib be used or is it better to
call "sendmail" or "mail" directly? Should popen2 or os.system() be
preferred?
I don't think this matters one way or another. It may be convenient to
use smtplib, if only because it wouldn't rely on sendmail or mail. The
problem with smtplib is that if you get rid of threads (which I suggest
later), smtplib could stall your program.

2) How to call an external tool (or many of them) and connect them via pipes
with each other and with the Python script in a multithreaded application? I
read the discussion http://www.python.org/doc/current/lib/
popen2-flow-control.html several times now and think I followed the
suggestions given there, but it seems still to cause trouble for me.
Connect them the same way you would in a shell (as you seem to be doing
already), with commands like:
command1 | command2 | command3

For getting the output from command3, I've always been a big fan of:
procs['unique_done'] = os.popen('command1 | command2 | command3 |'
' special unique_output unique_done')

Where special is a command that passes-though all information received
from command3, writes it to unique_output, and when done, creates a file
called unique_done, but doesn't write anything to stdout.

However often you want to, you can check for the existance of
unique_done (as many of them as you want) and execute the following:

toss = procs['unique_done'].read()
procs['unique_done'].close()
del procs['unique_done']
fil = open('unique_output', 'rb')
output = fil.read()
fil.close()
os.remove('unique_output')
os.remove('unique_done')
Personally, I would toss the multi-threaded portion and stick with
asynchronous IO. You don't mention how you are handling new information
coming into the system, whether this is by sockets or polling for files,
but basically everything can be wrapped up in an async loop:

while not quit:
asyncore.poll(0) #handles socket IO
poll_and_handle_done() #would check for all done
#outputs, and process them as above

By not using threads, you should find your software moves a few percent
faster when loaded heavily.

- - very, very seldom (1 out of 5000 received faxes), one user of me saw the
following - at least with Python 2.2.2; I have no feedback about newer
versions yet:
File "/usr/lib/python2.2/site-packages/cs_helpers.py", line 233, in
sendMIMEMail
sendmail.tochild.write(msg.as_string())
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Message.py", line
107, in as_string
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
100, in flatten
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
135, in _write
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
167, in _write_headers
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
191, in _split_header
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
41, in _is8bitstring
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/encodings/__init__.py",
line 43, in search_function
AttributeError: 'NoneType' object has no attribute 'get'

I don't know about this one, but it looks like somewhere your software
is passing a None when it should be passing a dictionary.

Good luck to you,
- Josiah