471,072 Members | 1,219 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,072 software developers and data experts.

Python threading, and processes

Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?

Cheers

-Rob
Feb 8 '06 #1
4 1900
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?


Both system-depend. The way threads are shwon depends on the systenm. And a
thread that runs, runs - so it consumes cpu-cycles. Try calling
time.sleep() to see how the consumption decreases.

And the limitation of allowed threads per process and/or system is also a
OS-dependend thing.

"""
Under Linux, threads are counted as processes, so any limits to the number
of processes also applies to threads. In a heavily threaded app like a
threaded TCP engine, or a java server, you can quickly run out of threads.
"""

from

http://people.redhat.com/alikins/sys...g.html#threads
Diez
Feb 8 '06 #2
On Wed, 08 Feb 2006 14:24:38 +0100, Diez B. Roggisch wrote:
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?
Both system-depend. The way threads are shwon depends on the systenm. And a
thread that runs, runs - so it consumes cpu-cycles. Try calling
time.sleep() to see how the consumption decreases.


Cheers for that info. The thread's main tasks are getting webpages
(spidering), the actual amount of processing done in each thread is
minimal - that's why I'm confused by the CPU usage.

Cheers

-Rob

Diez


Feb 8 '06 #3
Robin Haswell <ro*@digital-crocus.com> wrote:
...
Cheers for that info. The thread's main tasks are getting webpages
(spidering), the actual amount of processing done in each thread is
minimal - that's why I'm confused by the CPU usage.


BTW, spidering is a great use case for async (even-driven) programming;
use Twisted (or even build on top of asyncore) and you'll easily
saturate your network bandwidth with modest CPU demands.
Alex
Feb 8 '06 #4
Robin Haswell wrote:
Hey there

I'm doing some threading in python with Python 2.3 and 2.4 on Ubuntu and
Debian machines, and I've noticed that if I open a lot of threads (say,
50), I get lots of python processes with individual PIDs, which consume a
disproportionate amount of CPU. Does this mean that Python is using the
dummy_thread module by accident? And is there a realistic limitation to
the number of threads I can do?
Though I can not answer your question, I have however a similar
situation here, on debian sarge.

I have a simple daemon that runs one thread, and I noticed that on our
sarges with kernel 2.4 my daemon creates 4 processes, on the ones with
kernel 2.6, only one process.

btw, I use the thread module.

best regards,
Yves
Cheers

-Rob

Feb 8 '06 #5

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

65 posts views Thread by Anthony_Barker | last post: by
19 posts views Thread by Jane Austine | last post: by
5 posts views Thread by Carl J. Van Arsdall | last post: by
9 posts views Thread by corey.coughlin | last post: by
53 posts views Thread by Michael Tobis | last post: by
41 posts views Thread by km | last post: by
9 posts views Thread by cgwalters | last post: by
7 posts views Thread by Brendon Costa | last post: by
reply views Thread by leo001 | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.