473,324 Members | 2,268 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,324 software developers and data experts.

maximum number of threads

I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
MercuryObject.__init__(self,mylink)
File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul
Jan 10 '07 #1
12 18939
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
>I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).
The server quickly runs out of threads.

File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread

Does anyone know what it going on here and how I can ensure that I have
all the threads I need?
Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
--
Gabriel Genellina
Softlab SRL


__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

Jan 10 '07 #2
On 1/10/07, Gabriel Genellina <ga******@yahoo.com.arwrote:
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
Maybe Stackless could help the OP?
http://www.stackless.com/

--
Felipe.
Jan 10 '07 #3
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
Jan 10 '07 #4
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #5
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul

Jan 10 '07 #6
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #7
Paul Sijben a écrit :
Gabriel Genellina wrote:
>Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
>Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
>
Paul
Jan 10 '07 #8
On 1/10/07, Laurent Pointal <la*************@limsi.frwrote:
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.

--
Felipe.

Jan 10 '07 #9
On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.
So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
Jan 11 '07 #10
Felipe Almeida Lessa wrote:
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread
I tried your script on a PII 300 MHz and only 150 MB. I broke it of when it
reached more as 1,25 million. ;-}

Jan 11 '07 #11
"William Heymann" <ko**@aesaeion.comwrote:

So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
This almost looks as if the number of threads is a sixteen bit signed int...

- Hendrik

Jan 12 '07 #12
>>>>Felipe Almeida Lessa <fe**********@gmail.comwrites:
>>
to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.
It isn't that at all. The default Linux POSIX threads stack size is
8MB. Linux user space is 3GB (Kernel is mapped at upper 1GB).

382 * 8 = 3056MB.

Basically, you're running out of address space. I don't know if you have any
control at python level. In C you can call pthread_attr_setstacksize().

Ganesan

--
Ganesan Rajagopal

Jan 12 '07 #13

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
by: Tzach | last post by:
I'm developing a simple Java client that runs over a CORBA server. The main client thread is waiting for notification from this server. On each notification, The client creates a new thread...
0
by: David.Tymon | last post by:
>Description: MySQL v4.1.0-alpha only allows a client to prepare a maximum of 254 statements. On the 255th mysql_prepare() call, a failure is returned with no information returned by...
2
by: Kums | last post by:
What is the maximum permissible size of a database? Is there any limitation. What is the maximum # of tablespace's allowed in a database? Thanks for your response.
1
by: K. Davis | last post by:
I need to increment the maximum value of a field of a table by 1 when a form opens a blank record. (e.g. =max(!![trip_number}) so the logic and references are working at the form level. I've...
3
by: Raghu Rudra | last post by:
By default, the machine.config contains following information: <system.net> ....... <connectionManagement> <add address="*" maxconnection="2"/> </connectionManagement> </system.net> This...
1
by: Dominic | last post by:
I'd like to tune the performance of my application in a web garden. Our server has dual processors. Is there any guideline to set this "maximum number of worker processes" for web garden? In my...
4
by: Amir Shitrit | last post by:
What is the maximum number of threads Windows supports (for all processes), and what's the maximum number of threads available per process? Thanks.
5
by: linda.chen | last post by:
Hi all, We have a webservice (service1), which calls another webservice(service2) from another orginization. Our end users make requests throught service1. When service2 receives a quest, it...
6
by: =?Utf-8?B?U2hhcm9u?= | last post by:
I'm using the VScrollBar and set it as follow: m_vScrollBar.Minimum = -19602; m_vScrollBar.Maximum = 0; m_vScrollBar.SmallChange = 1; m_vScrollBar.LargeChange = 1089; m_vScrollBar.Value =...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: Vimpel783 | last post by:
Hello! Guys, I found this code on the Internet, but I need to modify it a little. It works well, the problem is this: Data is sent from only one cell, in this case B5, but it is necessary that data...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.