469,646 Members | 1,407 Online
Bytes | Developer Community
New Post

Home Posts Topics Members FAQ

Post your question to a community of 469,646 developers. It's quick & easy.

maximum number of threads

I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
MercuryObject.__init__(self,mylink)
File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul
Jan 10 '07 #1
12 18572
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
>I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).
The server quickly runs out of threads.

File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread

Does anyone know what it going on here and how I can ensure that I have
all the threads I need?
Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
--
Gabriel Genellina
Softlab SRL


__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

Jan 10 '07 #2
On 1/10/07, Gabriel Genellina <ga******@yahoo.com.arwrote:
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
Maybe Stackless could help the OP?
http://www.stackless.com/

--
Felipe.
Jan 10 '07 #3
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
Jan 10 '07 #4
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #5
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul

Jan 10 '07 #6
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #7
Paul Sijben a écrit :
Gabriel Genellina wrote:
>Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
>Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
>
Paul
Jan 10 '07 #8
On 1/10/07, Laurent Pointal <la*************@limsi.frwrote:
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.

--
Felipe.

Jan 10 '07 #9
On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.
So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
Jan 11 '07 #10
Felipe Almeida Lessa wrote:
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread
I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
reached more as 1,25 million. ;-}

Jan 11 '07 #11
"William Heymann" <ko**@aesaeion.comwrote:

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
This almost looks as if the number of threads is a sixteen bit signed int...

- Hendrik

Jan 12 '07 #12
>>>>Felipe Almeida Lessa <fe**********@gmail.comwrites:
>>
to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.
It isn't that at all. The default Linux POSIX threads stack size is
8MB. Linux user space is 3GB (Kernel is mapped at upper 1GB).

382 * 8 = 3056MB.

Basically, you're running out of address space. I don't know if you have any
control at python level. In C you can call pthread_attr_setstacksize().

Ganesan

--
Ganesan Rajagopal

Jan 12 '07 #13

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

5 posts views Thread by Tzach | last post: by
2 posts views Thread by Kums | last post: by
3 posts views Thread by Raghu Rudra | last post: by
4 posts views Thread by Amir Shitrit | last post: by
5 posts views Thread by linda.chen | last post: by
6 posts views Thread by =?Utf-8?B?U2hhcm9u?= | last post: by
reply views Thread by gheharukoh7 | last post: by
By using this site, you agree to our Privacy Policy and Terms of Use.