470,591 Members | 2,026 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 470,591 developers. It's quick & easy.

overhead of starting threads

Hi,

What is the overhead of starting threads in Python? Currently my app
starts around 20 threads per second, but I'm considering creating a pool
of a fixed number of threads and keeping them fed with data. But only if
it would give me a noticeable performance boost.

Has anyone got any any experience in this area that they could share? -
before I start investigating myself..

I'm running on Windows, if thats relevant.
Regards,

Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )
Jul 19 '05 #1
2 1939
How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

Jul 19 '05 #2
Simon Percivall wrote:
How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.
I wasn't asking about wether to use threads or not, I already am - for
several long running partialy I/O bound tasks.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.


I haven't had much luck with the profile module in the standard library.
It only shows me data from the main thread. Is there a better way of
profile heavily threaded code?
Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )
Jul 19 '05 #3

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

10 posts views Thread by Lenn | last post: by
8 posts views Thread by raghu | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.