473,324 Members | 2,356 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

overhead of starting threads

Hi,

What is the overhead of starting threads in Python? Currently my app
starts around 20 threads per second, but I'm considering creating a pool
of a fixed number of threads and keeping them fed with data. But only if
it would give me a noticeable performance boost.

Has anyone got any any experience in this area that they could share? -
before I start investigating myself..

I'm running on Windows, if thats relevant.
Regards,

Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )
Jul 19 '05 #1
2 2097
How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

Jul 19 '05 #2
Simon Percivall wrote:
How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.
I wasn't asking about wether to use threads or not, I already am - for
several long running partialy I/O bound tasks.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.


I haven't had much luck with the profile module in the standard library.
It only shows me data from the main thread. Is there a better way of
profile heavily threaded code?
Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )
Jul 19 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: karl | last post by:
I have a windows service that kicks off a 'monitor' thread which in turn kicks off 4 additional threads. These 4 threads basically are listen on a designated socket and report back any errors...
10
by: Lenn | last post by:
Hello, I have always used a certain design pattern for multithreaded Windows app; Start new worker thread from UI thread, use events to notify UI threads when something happens, update UI...
1
by: ErnieJ | last post by:
I a have a .net application which displays the results of a database query on a form in graphical format (i.e. charts and maps). This application is launched from another legacy Windows...
8
by: raghu | last post by:
Hello Everyone I am working on C platform for a project in that project I need to maintain 'n' number of states where n is a dynamic value simultaneously. But I should not use any kind of...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.