473,554 Members | 2,165 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

maximum number of threads

I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.p y", line 565, in __init__
MercuryObject._ _init__(self,my link)
File "/home/sijben/ORCA/src/libmercury_mt.p y", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thre ad(self.__boots trap, ())
error: can't start new thread
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul
Jan 10 '07 #1
12 18972
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
>I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).
The server quickly runs out of threads.

File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thre ad(self.__boots trap, ())
error: can't start new thread

Does anyone know what it going on here and how I can ensure that I have
all the threads I need?
Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
--
Gabriel Genellina
Softlab SRL


_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas

Jan 10 '07 #2
On 1/10/07, Gabriel Genellina <ga******@yahoo .com.arwrote:
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.
Maybe Stackless could help the OP?
http://www.stackless.com/

--
Felipe.
Jan 10 '07 #3
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul
Jan 10 '07 #4
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #5
Gabriel Genellina wrote:
>
Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.
sure but *how* fast they run out is the issue here
Do you really need so many threads?
I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul

Jan 10 '07 #6
Felipe Almeida Lessa wrote:
Maybe Stackless could help the OP?
http://www.stackless.com/
thanks I will look into it!
Jan 10 '07 #7
Paul Sijben a écrit :
Gabriel Genellina wrote:
>Simply you can't, as you can't have 10000 open files at once. Computer
resources are not infinite.

sure but *how* fast they run out is the issue here
>Do you really need so many threads?

I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
>
Paul
Jan 10 '07 #8
On 1/10/07, Laurent Pointal <la************ *@limsi.frwrote :
This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)
I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.

--
Felipe.

Jan 10 '07 #9
On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.
So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274
Jan 11 '07 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

5
30660
by: Tzach | last post by:
I'm developing a simple Java client that runs over a CORBA server. The main client thread is waiting for notification from this server. On each notification, The client creates a new thread executing some logic with the server and close. Each session spend most of the time waiting for the server. At most there can be ~1000 open session. My...
0
3001
by: David.Tymon | last post by:
>Description: MySQL v4.1.0-alpha only allows a client to prepare a maximum of 254 statements. On the 255th mysql_prepare() call, a failure is returned with no information returned by mysql_error(). This occurs even if the statements are closed after each use. >How-To-Repeat: The following program highlights the problem. Just change the...
2
28545
by: Kums | last post by:
What is the maximum permissible size of a database? Is there any limitation. What is the maximum # of tablespace's allowed in a database? Thanks for your response.
1
1915
by: K. Davis | last post by:
I need to increment the maximum value of a field of a table by 1 when a form opens a blank record. (e.g. =max(!![trip_number}) so the logic and references are working at the form level. I've tried a number of variants tied to the before update event, but always get the message "The field 'tripmain.trip_number' can't contain a Null value...
3
12211
by: Raghu Rudra | last post by:
By default, the machine.config contains following information: <system.net> ....... <connectionManagement> <add address="*" maxconnection="2"/> </connectionManagement> </system.net> This limits the number of connections to web server to 2. For example, if
1
9597
by: Dominic | last post by:
I'd like to tune the performance of my application in a web garden. Our server has dual processors. Is there any guideline to set this "maximum number of worker processes" for web garden? In my application, stress test shows that setting it to 2 gives a better performance result. In other words, web garden helps. My first question is if it...
4
6891
by: Amir Shitrit | last post by:
What is the maximum number of threads Windows supports (for all processes), and what's the maximum number of threads available per process? Thanks.
5
2556
by: linda.chen | last post by:
Hi all, We have a webservice (service1), which calls another webservice(service2) from another orginization. Our end users make requests throught service1. When service2 receives a quest, it puts all the request information into a database. The request will be picked up from a sheduler which is installed in 8 different servers. The...
6
10655
by: =?Utf-8?B?U2hhcm9u?= | last post by:
I'm using the VScrollBar and set it as follow: m_vScrollBar.Minimum = -19602; m_vScrollBar.Maximum = 0; m_vScrollBar.SmallChange = 1; m_vScrollBar.LargeChange = 1089; m_vScrollBar.Value = m_vScrollBar.Maximum; The scroll bar is set to start from the bottom.
0
7605
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
7530
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...
0
8047
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...
1
7570
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
7893
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
0
6156
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then...
0
5162
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...
0
3556
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2022
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.