By using this site, you agree to our updated Privacy Policy and our Terms of Use. Manage your Cookies Settings.
444,050 Members | 1,020 Online
Bytes IT Community
+ Ask a Question
Need help? Post your question and get tips & solutions from a community of 444,050 IT Pros & Developers. It's quick & easy.

maximum number of threads

P: n/a
I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
MercuryObject.__init__(self,mylink)
File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul
Jan 10 '07 #1
Share this Question
Share on Google+
12 Replies


P: n/a
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
>I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).
The server quickly runs out of threads.

File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread

Does anyone know what it going on here and how I can ensure that I have
all the threads I need?
Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
--
Gabriel Genellina
Softlab SRL


__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

Jan 10 '07 #2

P: n/a
On 1/10/07, Gabriel Genellina <ga******@yahoo.com.arwrote:
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
Maybe Stackless could help the OP?
http://www.stackless.com/

--
Felipe.
Jan 10 '07 #3

P: n/a
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
Jan 10 '07 #4

P: n/a
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #5

P: n/a
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul

Jan 10 '07 #6

P: n/a
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #7

P: n/a
Paul Sijben a écrit :
Gabriel Genellina wrote:
>Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
>Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
>
Paul
Jan 10 '07 #8

P: n/a
On 1/10/07, Laurent Pointal <la*************@limsi.frwrote:
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.

--
Felipe.

Jan 10 '07 #9

P: n/a
On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.
So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
Jan 11 '07 #10

P: n/a
Felipe Almeida Lessa wrote:
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread
I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
reached more as 1,25 million. ;-}

Jan 11 '07 #11

P: n/a
"William Heymann" <ko**@aesaeion.comwrote:

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
This almost looks as if the number of threads is a sixteen bit signed int...

- Hendrik

Jan 12 '07 #12

P: n/a
>>>>Felipe Almeida Lessa <fe**********@gmail.comwrites:
>>
to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.
It isn't that at all. The default Linux POSIX threads stack size is
8MB. Linux user space is 3GB (Kernel is mapped at upper 1GB).

382 * 8 = 3056MB.

Basically, you're running out of address space. I don't know if you have any
control at python level. In C you can call pthread_attr_setstacksize().

Ganesan

--
Ganesan Rajagopal

Jan 12 '07 #13

This discussion thread is closed

Replies have been disabled for this discussion.