473,218 Members | 1,696 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,218 software developers and data experts.

HowTo send emails reliably in a Python multi-threading application?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi!

I'm the developer of a Linux ISDN application which uses embedded Python for
controlling the communication. It starts several threads (i.e. one for each
incoming call and for outgoing faxes) which run Python scripts in embedded
Python interpreters which in turn do the real communication stuff. Any
incoming data or confirmations of done jobs are sent via Email.

Most real hard problems I had in the last year were either related to sending
emails or calling external tools via Pipes. I'll now try to roughly describe
the structure of CapiSuite and the problems I experienced. I'll try to do it
as shortly but understandably as possible. As all of it experiences
"sometimes" I can't provide small, clear test cases, sorry... :-((

All of the stuff coming now boils down to two questions:

1) How can one send mails reliable in a multi-threaded Python application?
Should the email package be used? Should smtplib be used or is it better to
call "sendmail" or "mail" directly? Should popen2 or os.system() be
preferred?

2) How to call an external tool (or many of them) and connect them via pipes
with each other and with the Python script in a multithreaded application? I
read the discussion http://www.python.org/doc/current/lib/
popen2-flow-control.html several times now and think I followed the
suggestions given there, but it seems still to cause trouble for me.

For those of you having some seconds, here's the rough description of my
problems...

I'm using normal linuxthreads on Linux 2.4.

Currently I have two mail send functions in use:

a) sendMIMEMail which first of all does some format conversion using external
commandline tools using popen2 and then constructs a mail using
email.MIMEText, email.MIMEBase or email.MIMEAudio. This mail is sent via the
"sendmail" commandline tool like this:

sendmail = popen2.Popen3("sendmail -t -f %s" % escape(mail_from))
if (sendmail.poll()!=-1):
...
sendmail.tochild.write(msg.as_string())
sendmail.tochild.close()
sendmail.fromchild.close()
ret=sendmail.wait()

b) sendSimpleMail which creates a simple mail using email.MIMEText and then
does the same as sendMIMEMail for sending it.

The problems I had so far:

- - when sendSimpleMail and sendMIMEMail are run nearly in parallel, thenthe
system sometimes gets stuck with one thread which is showed in "ps ax" but
does not exist according to gdb. It seems to be a try of one Python script to
run "sendmail" - but it gets stuck either before the new process was really
replaced by the started sendmail or after sendmail finished but before the
thread is destroyed. This can be worked around according to one user by not
using the mail package but sending the mail via
os.system("echo -e "+escape(text)+" | mail -s "+escape(mail_subject)+ " -r
"+escape(mail_from)+" "+escape(mail_to)) in sendSimpleMail

- - sometimes users (mainly Red Hat users) got the error message "cannot
unmarshal code objects in restricted execution mode" when the mail text was
encoded to ASCII. This one was reliably worked around by calling "import
encodings.ascii" at the begin of send*Mail.
(see http://lists.berlios.de/pipermail/ca...ly/000169.html
and http://lists.berlios.de/pipermail/ca...ly/000175.html
for details)

- - when a long document was converted by calling several commandline tools
(format1->format2->format3 via 2 tools), then popen2-pipes between process 1
and 2 were overflowed. This was worked around by using shell pipes between
process1 + 2 (popen2.Popen3("command1 | command2"))

- - very, very seldom (1 out of 5000 received faxes), one user of me saw the
following - at least with Python 2.2.2; I have no feedback about newer
versions yet:
File "/usr/lib/python2.2/site-packages/cs_helpers.py", line 233, in
sendMIMEMail
sendmail.tochild.write(msg.as_string())
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Message.py", line
107, in as_string
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
100, in flatten
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
135, in _write
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
167, in _write_headers
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
191, in _split_header
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py",line
41, in _is8bitstring
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/encodings/__init__.py",
line 43, in search_function
AttributeError: 'NoneType' object has no attribute 'get'

So to summarize it: my current approach to do the format conversions and the
mail sending seems to be quite unstable, but I have no idea how to do it
better. But I'm quite sure there must be a reliable way to do these kinds of
things in Python. :-)) Perhaps there's also a general problem with my Python
embedding stuff - but as everything else works very nice, I doubt this is the
case...

I'll again have to excuse for the rough problem descriptions I can give yet
(and the complete miss of small test cases but it's really hard to debug...),
but I hope some of you could comment on my issues nevertheless. Any hint is
greatly appreciated!!!

TIA!!!

- --
Bye,

Gernot
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQFAN01Ok997/GGeSeIRAiALAJ9lGCv2RnbZpgklXx/mPYjpd5M+DACdHJsE
/RSiWpe7mU4u9Lp9qsKp2CQ=
=tnPN
-----END PGP SIGNATURE-----
Jul 18 '05 #1
1 5747
> 1) How can one send mails reliable in a multi-threaded Python application?
Should the email package be used? Should smtplib be used or is it better to
call "sendmail" or "mail" directly? Should popen2 or os.system() be
preferred?
I don't think this matters one way or another. It may be convenient to
use smtplib, if only because it wouldn't rely on sendmail or mail. The
problem with smtplib is that if you get rid of threads (which I suggest
later), smtplib could stall your program.

2) How to call an external tool (or many of them) and connect them via pipes
with each other and with the Python script in a multithreaded application? I
read the discussion http://www.python.org/doc/current/lib/
popen2-flow-control.html several times now and think I followed the
suggestions given there, but it seems still to cause trouble for me.
Connect them the same way you would in a shell (as you seem to be doing
already), with commands like:
command1 | command2 | command3

For getting the output from command3, I've always been a big fan of:
procs['unique_done'] = os.popen('command1 | command2 | command3 |'
' special unique_output unique_done')

Where special is a command that passes-though all information received
from command3, writes it to unique_output, and when done, creates a file
called unique_done, but doesn't write anything to stdout.

However often you want to, you can check for the existance of
unique_done (as many of them as you want) and execute the following:

toss = procs['unique_done'].read()
procs['unique_done'].close()
del procs['unique_done']
fil = open('unique_output', 'rb')
output = fil.read()
fil.close()
os.remove('unique_output')
os.remove('unique_done')
Personally, I would toss the multi-threaded portion and stick with
asynchronous IO. You don't mention how you are handling new information
coming into the system, whether this is by sockets or polling for files,
but basically everything can be wrapped up in an async loop:

while not quit:
asyncore.poll(0) #handles socket IO
poll_and_handle_done() #would check for all done
#outputs, and process them as above

By not using threads, you should find your software moves a few percent
faster when loaded heavily.

- - very, very seldom (1 out of 5000 received faxes), one user of me saw the
following - at least with Python 2.2.2; I have no feedback about newer
versions yet:
File "/usr/lib/python2.2/site-packages/cs_helpers.py", line 233, in
sendMIMEMail
sendmail.tochild.write(msg.as_string())
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Message.py", line
107, in as_string
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
100, in flatten
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
135, in _write
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
167, in _write_headers
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
191, in _split_header
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/email/Generator.py", line
41, in _is8bitstring
File "/var/tmp/python-2.2.2-build//usr/lib/python2.2/encodings/__init__.py",
line 43, in search_function
AttributeError: 'NoneType' object has no attribute 'get'


I don't know about this one, but it looks like somewhere your software
is passing a None when it should be passing a dictionary.

Good luck to you,
- Josiah
Jul 18 '05 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

4
by: Thom McGrath | last post by:
I'm writing a simple mailing list program, and I would like to know what the suggested method of sending a large number of emails to a list of addresses. (sounds like spam, no?) It's perfectly...
4
by: Logan | last post by:
Several people asked me for the following HOWTO, so I decided to post it here (though it is still very 'alpha' and might contain many (?) mistakes; didn't test what I wrote, but wrote it - more or...
4
by: Josef Sachs | last post by:
Is Andrew Kuchling's regex-to-re HOWTO available anywhere? I've found the following (dead) links on various Web pages: http://py-howto.sourceforge.net/regex-to-re/regex-to-re.html...
40
by: ian | last post by:
Hi, I'm a newbie (oh no I can here you say.... another one...) How can I get Python to send emails using the default windows email client (eg outlook express)? I thought I could just do the...
4
by: Philipp Ott | last post by:
Hello! I m looking for a solution to generate a digitally signed mime-email with linux/perl and to feed/pipe this then to sendmail. I found RFCs related to mime-signed etc. but somehow fail to...
0
by: alexmaster_2004 | last post by:
Hi There i'm using System.Web.Mail to send email message. but i have a problem with it. the emails that i send is in Turkish Language. so when i send the email the receipt can't be able to raed...
0
by: nauticalmac | last post by:
I'm using CDO to send mail to the site owner from ASP pages with forms. Recently one of my forms is occasionally sending email with what seems to be an insertion which is replacing the plain text...
4
by: Marco Meoni | last post by:
Hi. I read the Gordon McMillan's "Socket Programming Howto". I tried to use the example in this howto but this doesn't work. The code is class mysocket: '''classe solamente dimostrativa -...
5
by: Kun | last post by:
i have the following code: ---------------------------------- import smtplib from email.MIMEText import MIMEText fp = open('confirmation.txt', 'rb') msg = MIMEText(fp.read()) From =...
4
by: =?Utf-8?B?dHBhcmtzNjk=?= | last post by:
I have a web page that at the click of a button must send a bunch (1000+) emails. Each email is sent individually. I have the code working fine, using Mail Message classes and smtp and all that. ...
1
isladogs
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
0
by: veera ravala | last post by:
ServiceNow is a powerful cloud-based platform that offers a wide range of services to help organizations manage their workflows, operations, and IT services more efficiently. At its core, ServiceNow...
0
by: jianzs | last post by:
Introduction Cloud-native applications are conventionally identified as those designed and nurtured on cloud infrastructure. Such applications, rooted in cloud technologies, skillfully benefit from...
2
by: jimatqsi | last post by:
The boss wants the word "CONFIDENTIAL" overlaying certain reports. He wants it large, slanted across the page, on every page, very light gray, outlined letters, not block letters. I thought Word Art...
2
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 7 Feb 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:30 (7.30PM). In this month's session, the creator of the excellent VBE...
0
by: fareedcanada | last post by:
Hello I am trying to split number on their count. suppose i have 121314151617 (12cnt) then number should be split like 12,13,14,15,16,17 and if 11314151617 (11cnt) then should be split like...
0
by: stefan129 | last post by:
Hey forum members, I'm exploring options for SSL certificates for multiple domains. Has anyone had experience with multi-domain SSL certificates? Any recommendations on reliable providers or specific...
0
Git
by: egorbl4 | last post by:
Скачал я git, хотел начать настройку, а там вылезло вот это Что это? Что мне с этим делать? ...
1
by: davi5007 | last post by:
Hi, Basically, I am trying to automate a field named TraceabilityNo into a web page from an access form. I've got the serial held in the variable strSearchString. How can I get this into the...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.