Hello,
I am the author of OfflineIMAP[1], a program that currently is a heavy
user of threads. For various reasons (see below), thread groups are
something that is highly desirable for me in Python. Is anyone aware
of an implementation of thread groups for Python? Is anyone else
working on this? Is anyone else experiencing this sort of pain?
Here's why I need this:
OfflimeIMAP is a bi-directional IMAP/Maildir synchronization system.
It can synchronize multiple accounts at once. Within a given account,
it can synchronize multiple folders at once. Within a given folder,
it can synchronize multiple messages at once. All these simultaneous
actions are handled by threads. (In case you're wondering, there are
bounds in the system to prevent this all from running away.)
Right now, error handling is not very good. Let's say that one of the
5 connections to an IMAP server for Account A dies on us and we get a
network exception. Ideally, with this condition, I would shut down
all the other threads processing Account A, terminate its open
connections, and restart it. (An unexpected error can leave
OfflineIMAP with an incorrect idea of what is on the server, so the
safest thing to do is reset the algorithm and thus get updates from
the server on what exactly it knows about.)
However, I can't do that with the current system. The best thing I
can do is catch the exception and transmit it to an exception handling
thread. (By overriding methods of the Thread object.) That's close,
but...
The exception handling thread can do things like print out the error.
But the recourses available to it are: 1) terminate the entire
application, and 2) ignore the error. Option #2 is not satisfactory,
to the only remaining option is #1.
Here are some other things that won't work:
* Putting all network code within a try block.
Several problems with that: the exceptions thrown are different for
different types of connections (plain socket, ssl, etc), and may
not be known in advance.
Moreover, actions such as "download message" are done in individual
threads, and the parent thread -- which really needs to know about
this -- has no way to get the message until it does a join, which
it may or may not ever have cause to do. Even then, it would
usually be better for it to get the message immediately.
* Save some state on thread exit and retrieve it on join().
Possible, yes, and might be a usable workaround. However, there
are a number of places in the code that don't do a join(), and this
seems an error-prone way of doing things.
It basically involves manually passing data back up the "thread
chain", like passing error codes back up through the call chain in
C. Essentially, many of the benefits of exceptions in the first
place disappear.
Thoughts anyone?
[1] OfflineIMAP is at http://quux.org/devel/offlineimap
-- John Goerzen
