overhead of starting threads

Will McGugan

Hi,

What is the overhead of starting threads in Python? Currently my app
starts around 20 threads per second, but I'm considering creating a pool
of a fixed number of threads and keeping them fed with data. But only if
it would give me a noticeable performance boost.

Has anyone got any any experience in this area that they could share? -
before I start investigating myself..

I'm running on Windows, if thats relevant.
Regards,

Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )

Jul 19 '05 #1

Subscribe Post Reply

2016

Simon Percivall

How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

Jul 19 '05 #2

Will McGugan

Simon Percivall wrote:

How much you gain by starting threads is also determined by what you're
doing in those threads. Remember (or learn): In CPython only one thread
at a time can execute python code, so depending on your task threading
might gain you little. If you're doing I/O or calling functions written
in C (and if they release the Global Intepreter Lock [GIL]) you might
gain a lot by using threads.
I wasn't asking about wether to use threads or not, I already am - for
several long running partialy I/O bound tasks.

As for overhead: profile, profile, profile. You'll have to do sample
runs and find your sweet-spot. It all depends on what you're doing in
the threads.

I haven't had much luck with the profile module in the standard library.
It only shows me data from the main thread. Is there a better way of
profile heavily threaded code?
Will McGugan

--
http://www.willmcgugan.com
"".join( [ {'*':'@','^':'.'}.get(c,None) or chr(97+(ord(c)-84)%26) for c
in "jvyy*jvyyzpthtna^pbz" ] )

Jul 19 '05 #3

by: karl | last post by:

I have a windows service that kicks off a 'monitor' thread which in turn kicks off 4 additional threads. These 4 threads basically are listen on a...

C# / C Sharp

Starting a new Thread vs. ThreadPool

by: Lenn | last post by:

Hello, I have always used a certain design pattern for multithreaded Windows app; Start new worker thread from UI thread, use events to notify UI...

C# / C Sharp

Launching .net forms from non .net application - reducing overhead

by: ErnieJ | last post by:

I a have a .net application which displays the results of a database query on a form in graphical format (i.e. charts and maps). This application...

Visual Basic .NET

Starting Multitasking with out threads in C

by: raghu | last post by:

Hello Everyone I am working on C platform for a project in that project I need to maintain 'n' number of states where n is a dynamic value...

C / C++

Tips for Saving Money on Buying a Gaming Laptop

by: tammygombez | last post by:

Hey everyone! I've been researching gaming laptops lately, and I must say, they can get pretty expensive. However, I've come across some great...

Computer Hardware

Discuss your understanding of the Java platform.

by: better678 | last post by:

Question: Discuss your understanding of the Java platform. Is the statement "Java is interpreted" correct? Answer: Java is an object-oriented...

General

Benefits of Blockchain App Development for Enterprise Businesses

by: Kemmylinns12 | last post by:

Blockchain technology has emerged as a transformative force in the business world, offering unprecedented opportunities for innovation and...

General

How to minimize a tool bar that shows up when in reports

by: CD Tom | last post by:

This happens in runtime 2013 and 2016. When a report is run and then closed a toolbar shows up and the only way to get it to go away is to right...

General

how to collapse a custom toolbar in VBA

by: CD Tom | last post by:

This only shows up in access runtime. When a user select a report from my report menu when they close the report they get a menu I've called Add-ins...

General

https redirect with reverse proxy

by: jalbright99669 | last post by:

Am having a bit of a time with URL Rewrite. I need to incorporate http to https redirect with a reverse proxy. I have the URL Rewrite rules made...

IIS / Internet Information Server

Hyperconverged All-in-One Streaming Engine, ushering in new age for distributed database

by: antdb | last post by:

Ⅰ. Advantage of AntDB: hyper-convergence + streaming processing engine In the overall architecture, a new "hyper-convergence" concept was...

General

How to layout php header location with a variable as the location?

by: Matthew3360 | last post by:

Hi there. I have been struggling to find out how to use a variable as my location in my header redirect function. Here is my code. ...

PHP

How to use http requests (Get function) to get variable value?

by: Matthew3360 | last post by:

Hi, I have a python app that i want to be able to get variables from a php page on my webserver. My python app is on my computer. How would I make it...

Python

overhead of starting threads

Similar topics