By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
440,035 Members | 1,995 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 440,035 IT Pros & Developers. It's quick & easy.

C replacement for Queue module

P: n/a
I'm working on an application that makes heavy use of Queue objects in
a multithreaded environment.

By "heavy" I mean "millions of calls to put and get, constituting ~20%
of the app's run time." The profiler thinks that a significant amount
of time is spent in this code -- not just a consumer waiting for a
producer, but actual _empty, notify calls, etc.

Would it be worth the time to write a CQueue module with pthread_cond
instead of Python Condition objects, etc? I don't really have a gut
feeling for how much bang-for-the-loc C optimization might give: twice
as fast? 5x as fast?

thanks,

-Jonathan

Oct 21 '05 #1
Share this Question
Share on Google+
6 Replies


P: n/a
How about Python 2.4's collections.deque class? Supposedly it's
thread-safe, and it's implemented in C.

- Jason

Oct 22 '05 #2

P: n/a
does collections.deque have a blocking popleft()? If not, it's not very
suitable to replace Queue.Queue.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDWbH1Jd01MZaTXX0RAm4yAJ4tQtff8/SjtnOkR0Zlrr5LNnSBFACaAyPr
L714IVqIIOBPy2UsTLz0krI=
=bCJe
-----END PGP SIGNATURE-----

Oct 22 '05 #3

P: n/a
je****@unpythonic.net wrote:
does collections.deque have a blocking popleft()? If not, it's not very
suitable to replace Queue.Queue.


It does not.

-Jonathan

Oct 22 '05 #4

P: n/a
Jonathan Ellis wrote:
I'm working on an application that makes heavy use of Queue objects in
a multithreaded environment.

By "heavy" I mean "millions of calls to put and get, constituting ~20%
of the app's run time." The profiler thinks that a significant amount
of time is spent in this code -- not just a consumer waiting for a
producer, but actual _empty, notify calls, etc.


I wonder if the use case would support hand-crafting an alternative.
Queues appear fairly heavy weight (when you look at the implementation),
and while they are very robust, if your own needs allow putting many
items in at the same time, or getting many items out, for example, then
perhaps you could come up with a much faster alternative based on the
primitives (e.g. Event, Condition, etc) and perhaps some wrapped data
structure other than the list that Queues use.

-Peter
Oct 22 '05 #5

P: n/a
As far as Queues go, the adding/popping is apparently done with deque
which are implemented in C. The Queue class pretty much just provides
blocking operations and is otherwise a very thin layer around deque. As
far as primitives go, only threading.Lock is written in C and the
others are pure Python, so they're not that fast, which might be a
reason for Queue's slowness.

As far as writing a custom C module, you could probably leave most of
the work to deque and just implement blocking. If you stick to a simple
lock primative, you can keep it portable and use Python's abstracted
thread interface with an amazing choice of two whole functions: acquire
and release.

Oct 22 '05 #6

P: n/a
Jason Lai wrote:
As far as Queues go, the adding/popping is apparently done with deque
which are implemented in C. The Queue class pretty much just provides
blocking operations and is otherwise a very thin layer around deque. As
far as primitives go, only threading.Lock is written in C and the
others are pure Python, so they're not that fast, which might be a
reason for Queue's slowness.


BTW, I didn't mean (or expect) that one could improve significantly on
Queue's performance in general just by re-writing it.

I meant that if the OP took into account _his own unique situation_, he
might see opportunities for improvement which Queue couldn't possibly
provide, given that it's a general purpose tool. If his own situation
simply involves a massive number of discrete put/get operations which
cannot be bundled in any fashion, rewriting the entire Queue in C just
to avoid the Python method calls (which have high setup overhead) might
be the only real option.

-Peter
Oct 23 '05 #7

This discussion thread is closed

Replies have been disabled for this discussion.