468,161 Members | 1,956 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 468,161 developers. It's quick & easy.

Python threading, and processes

Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?

Cheers

-Rob
Feb 8 '06 #1
4 1844
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?


Both system-depend. The way threads are shwon depends on the systenm. And a
thread that runs, runs - so it consumes cpu-cycles. Try calling
time.sleep() to see how the consumption decreases.

And the limitation of allowed threads per process and/or system is also a
OS-dependend thing.

"""
Under Linux, threads are counted as processes, so any limits to the number
of processes also applies to threads. In a heavily threaded app like a
threaded TCP engine, or a java server, you can quickly run out of threads.
"""

from

http://people.redhat.com/alikins/sys...g.html#threads
Diez
Feb 8 '06 #2
On Wed, 08 Feb 2006 14:24:38 +0100, Diez B. Roggisch wrote:
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?
Both system-depend. The way threads are shwon depends on the systenm. And a
thread that runs, runs - so it consumes cpu-cycles. Try calling
time.sleep() to see how the consumption decreases.


Cheers for that info. The thread's main tasks are getting webpages
(spidering), the actual amount of processing done in each thread is
minimal - that's why I'm confused by the CPU usage.

Cheers

-Rob

Diez


Feb 8 '06 #3
Robin Haswell <ro*@digital-crocus.com> wrote:
...
Cheers for that info. The thread's main tasks are getting webpages
(spidering), the actual amount of processing done in each thread is
minimal - that's why I'm confused by the CPU usage.


BTW, spidering is a great use case for async (even-driven) programming;
use Twisted (or even build on top of asyncore) and you'll easily
saturate your network bandwidth with modest CPU demands.
Alex
Feb 8 '06 #4
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?
Though I can not answer your question, I have however a similar
situation here, on debian sarge.

I have a simple daemon that runs one thread, and I noticed that on our
sarges with kernel 2.4 my daemon creates 4 processes, on the ones with
kernel 2.6, only one process.

btw, I use the thread module.

best regards,
Yves
Cheers

-Rob

Feb 8 '06 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

65 posts views Thread by Anthony_Barker | last post: by
19 posts views Thread by Jane Austine | last post: by
5 posts views Thread by Carl J. Van Arsdall | last post: by
9 posts views Thread by corey.coughlin | last post: by
53 posts views Thread by Michael Tobis | last post: by
41 posts views Thread by km | last post: by
9 posts views Thread by cgwalters | last post: by
7 posts views Thread by Brendon Costa | last post: by
1 post views Thread by gcdp | last post: by
reply views Thread by kamranasdasdas | last post: by
reply views Thread by gcreed | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.