Asynchronous Messaging

[wink]

Hello,

I'm getting my feet wet in Python and thought I'd try to see how well
Python works for asynchronous messaging. I've been using asynchronous
messaging for 5 years and find it advantageous for many things. In the
scheme I use communication is not only asynchronous but it is also non-
blocking and inherently serialized via a queue.

This provides two benefits, a sender is never directly effected by the
receiver and since the receiver handles only one message at a time
it generally never has to use mutexes or semaphores. This allows for
the
programmer to use multiple threads without having to contend with the
associated issues mutexes have in the area of deadly embraces or race
conditions.

Anyway, below is a link to the first pass at an implementation
in Python, comments are welcome:

http://www.saville.com/python/mproc.tgz

Regards,

Wink Saville

[Fredrik Lundh]

wink wrote:

This provides two benefits, a sender is never directly effected by the
receiver and since the receiver handles only one message at a time
it generally never has to use mutexes or semaphores. This allows for
the programmer to use multiple threads without having to contend with the
associated issues mutexes have in the area of deadly embraces or race
conditions.

import Queue # ?

</F>

[A.T.Hofkamp]

On 2007-09-26, wink <wi**@saville.c omwrote:

Hello,

I'm getting my feet wet in Python and thought I'd try to see how well
Python works for asynchronous messaging. I've been using asynchronous

Have a look at Twisted (www.twistedmatrix.com)
Albert

[Fredrik Lundh]

wink wrote:

But its performance is poor if the number of items on a
Queue becomes large because it is implemented using a list.

One of the things I was thinking of was doing another implementation
using of Queue which was based on deque.

Updating from 2.3 to something newer will fix that, of course:

$ more Queue.py

....

from collections import deque

....

class Queue:

...

def _init(self, maxsize):
self.maxsize = maxsize
self.queue = deque()

</F>

[wink]

Fredrik,

You are most correct, but Queue is slow compared to deque but
not for the reason I guessed. Apparently it's because deque is
implemented in C while Queue is in python. Using the program below
it looks there is about a 35:1 speed difference.

100 d.append 0.000011s 0.1097us per
100 d.popL 0.000011s 0.1097us per
100 q.put 0.000429s 4.2892us per
100 q.get 0.000391s 3.9077us per

So someday it _might_ warrant adding Queue to collections.
#!/usr/bin/python

import timeit
import Queue
from collections import deque

setupD = """
from collections import deque
d=deque()
cnt = %d
for x in xrange(cnt):
d.append(x)
"""

setupQ = """
import Queue
q=Queue.Queue()
cnt = %d
for x in xrange(cnt):
q.put(x)
"""

def main():
cnt = 100
t = timeit.Timer(se tup=setupD % cnt, stmt="d.append( 0)")
time = min(t.repeat(10 000, cnt))
print " %9d d.append %fs %7.4fus per" % \
(cnt, time, (time/cnt) * 1000000.0)

t = timeit.Timer(se tup=setupD % cnt, stmt="d.popleft ()")
time = min(t.repeat(10 000, cnt))
print " %9d d.popL %fs %7.4fus per" % \
(cnt, time, (time/cnt) * 1000000.0)

t = timeit.Timer(se tup=setupQ % cnt, stmt="q.put(0)" )
time = min(t.repeat(10 000, cnt))
print " %9d q.put %fs %7.4fus per" % \
(cnt, time, (time/cnt) * 1000000.0)

t = timeit.Timer(se tup=setupQ % cnt, stmt="q.get()")
time = min(t.repeat(10 000, cnt))
print " %9d q.get %fs %7.4fus per" % \
(cnt, time, (time/cnt) * 1000000.0)
if __name__ == "__main__":
main()

[Gabriel Genellina]

En Thu, 27 Sep 2007 01:43:32 -0300, wink <wi**@saville.c omescribi�:

You are most correct, but Queue is slow compared to deque but
not for the reason I guessed. Apparently it's because deque is
implemented in C while Queue is in python. Using the program below
it looks there is about a 35:1 speed difference.

That't not the reason. A Queue is built around a container, and it happens
to be a deque in the default implementation. But the important thing is
that a Queue is a synchronized object - it performs the necesary
synchronization to ensure proper operation even from multiple threads
attempting to use it at the same time.
So your comparison is meaningless, apart from telling that using mutexes
is not cheap.

--
Gabriel Genellina