Hello there
question: I'm looking for information / patterns / recipe's for
implementing a multi threaded program (multiple producers / consumers)
that will manage transactions and maintain sequence across the different
threads.
background:
I'm busy writing an adapter taking messages from ibm websphere mq using
pymqi and inserting these messages into a database (after some
processing, the messages come in as xml). If I comment out the database
..execute bit of the code, i process hundreds of messages per second, the
database slows the whole process down to around 60 - 100 messages per
second. I'm hoping to speed this up some by having 2 threads reading the
messages from mq, and another populating the database - at least then
the database bit will never have to wait for me to fetch another
message. (i'm thinking of later adding a second thread for the databse
as well).
The challenge is that these messages need to be processed in exact
sequence and transactionally. (ie. the thread that fetched a message
will have to wait for confirmation that the message has successfully
been inserted into the database before committing the transaction, in mq
terms the message will only really be deleted then.)
I've started implementing a kind of state machine using dictionary's to
facilitate this, but I'm now concerned that
- there is a better way
- i might get bitten with dictionary processing updating / setting of
values not necessarily being thread safe.
I've looked at this recipe
http://aspn.activestate.com/ASPN/Coo.../Recipe/302997 as a
possible replacement for dictionary's but I'm not even sure if I'm using
it correctly.
thank you
jaco
