threads and memory

Lee Leahu

Hi List,

I am trying to write a simple threaded application which will simulate 1000 connections to a remote service in order to "stress test" that the remote service can handle that many connections.

However, I have encountered the following error after I have started my 381st thread:

---------------------------------------------------------------------------
Traceback (most recent call last):
File "./test.py", line 39, in ?
current.start()
File "/usr/lib/python2.4/threading.py", line 416, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
---------------------------------------------------------------------------

The command 'ps -eo pid,%cpu,%mem,rss,sz,vsz' gives at the point of error, the following information:
---------------------
pid: 5448
%cpu: 33.5
%mem: 0.6
rss: 9492
sz: 782729
vsz: 3130916
---------------------

I have 1.5GB of total ram in my computer, as evident by the command 'free':
---------------------
total used free shared buffers cached
Mem: 1553128 1314152 238976 0 0 357392
-/+ buffers/cache: 956760 596368
Swap: 977248 7964 969284
---------------------

An 'strace' of the script reveals the following:
NOTE the line that reads 'ENOMEM (Cannot allocate memory).
--------------------------------------------------------
futex(0x80b2b08, FUTEX_WAKE, 1) = 1
select(0, NULL, NULL, NULL, {0, 0}) = 0 (Timeout)
write(1, "381\n", 4381
) = 4
futex(0x80b2b08, FUTEX_WAKE, 1) = 0
futex(0x80b2b08, FUTEX_WAKE, 1) = 0
futex(0x80581f8, FUTEX_WAKE, 1) = 0
mmap2(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35
open("./test.py", O_RDONLY|O_LARGEFILE) = 383
write(2, " File \"./test.py\", line 40, in "..., 34 File "./test.py", line 40, in ?
) = 34
fstat64(383, {st_mode=S_IFREG|0755, st_size=1037, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f65000
read(383, "#!/usr/bin/python\n\nimport time\nf"..., 4096) = 1037
write(2, " ", 4 ) = 4
write(2, "current.start()\n", 16current.start()
--------------------------------------------------------

Is there a way to accomplish what I am trying to do, perhaps in a more memory-friendly way?

My source code is attached.

Please advise.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
le*@ricis.com 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

Feb 6 '06 #1

Subscribe Post Reply

2480

Rene Pijlman

Lee Leahu:

I am trying to write a simple threaded application which will simulate 1000
connections to a remote service in order to "stress test" that the remote
service can handle that many connections. [...]Is there a way to accomplish what I am trying to do, perhaps in a more
memory-friendly way?

Yes. You don't need one thread per connection:
http://docs.python.org/lib/module-asyncore.html

--
René Pijlman

Feb 6 '06 #2

Lee Leahu

Hi List,

I am trying to write a simple threaded application which will simulate 1000
connections to a remote service in order to "stress test" that the remote
service can handle that many connections.

[...]
Is there a way to accomplish what I am trying to do, perhaps in a more
memory-friendly way?

Yes. You don't need one thread per connection:
http://docs.python.org/lib/module-asyncore.html

I am running into a problem where asyncore is through a filedescriptor error if I try to launch more that 1023 connections:
------------------------------------
Traceback (most recent call last):
File "./test.py", line 46, in ?
asyncore.loop()
File "/usr/lib/python2.4/asyncore.py", line 192, in loop
poll_fun(timeout, map)
File "/usr/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
------------------------------------

Is there a limitation on the number of simultaneous connects that can be made with asyncore ?

Attached is the updated source code.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
le*@ricis.com 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

Feb 6 '06 #3

Bryan Olson

Lee Leahu wrote:

I am trying to write a simple threaded application which will
simulate 1000 connections to a remote service in order to
stress test" that the remote service can handle that many
connections.
That shouldn't be a problem on a modern OS, but there are
still quite a few not-up-to-speed systems in the field that
choke on just a few hundred threads. There are also many
systems with modern threading that choke because they are
poorly configured by default.
However, I have encountered the following error after I
have started my 381st thread:
[...] I have 1.5GB of total ram in my computer, as evident by the
command 'free': [...] NOTE the line that reads 'ENOMEM (Cannot allocate memory).

The error message may be misleading. The limiting factor on
the number of threads is usually address space, not physical
memory. You can get ENOMEM when the system cannon allocate
stack space for a thread. The space it needs is not RAM,
nor even RAM+swap; it's pure virtual address space.

Look up how to set the stack-space for threads on your
system. A reasonable size on a 32-bit system is 2MB to
4MB. Then you should be able to create threads numbering
in the low thousands. (On 64-bit systems, the issue just
goes away.)
--
--Bryan

Feb 6 '06 #4

Fredrik Lundh

Lee Leahu wrote:

Yes. You don't need one thread per connection:
http://docs.python.org/lib/module-asyncore.html

I am running into a problem where asyncore is through a filedescriptor
error if I try to launch more that 1023 connections:
------------------------------------
Traceback (most recent call last):
File "./test.py", line 46, in ?
asyncore.loop()
File "/usr/lib/python2.4/asyncore.py", line 192, in loop
poll_fun(timeout, map)
File "/usr/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
------------------------------------

Is there a limitation on the number of simultaneous connects that can
be made with asyncore ?

what does "ulimit -n" (max number of file descriptors) say on your machine?

if it says 1024, that's what you're seeing. to increase this, you
have to reconfigure your OS.

if you get a larger value, you can try passing use_poll=True to the
asyncore.loop function.

if use_poll doesn't work, you may have to rebuild Python's select
module to support more handles. see the comments at the top of
Modules/selectmodule.c for details.

</F>

Feb 6 '06 #5

Lee Leahu

Hi List,

I am running into a problem where asyncore is through a filedescriptor
error if I try to launch more that 1023 connections:
------------------------------------
Traceback (most recent call last):
File "./test.py", line 46, in ?
asyncore.loop()
File "/usr/lib/python2.4/asyncore.py", line 192, in loop
poll_fun(timeout, map)
File "/usr/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
------------------------------------

Is there a limitation on the number of simultaneous connects that can
be made with asyncore ?
what does "ulimit -n" (max number of file descriptors) say on your machine?

As a normal user, I was limited to 1024. But su'ing to root allows me to set this higher, like 8196.

This error occurs after setting it higher (which resolves the socket.err(24, "too many open files") problem).

if you get a larger value, you can try passing use_poll=True to the
asyncore.loop function.
I will give this a try.

if use_poll doesn't work, you may have to rebuild Python's select
module to support more handles. see the comments at the top of
Modules/selectmodule.c for details.

Thanks!

--
Lee Leahu RICIS, Inc.
Internet Technology Specialist 866-RICIS-77 Toll Free Voice (US)
le*@ricis.com 708-444-2690 Voice (International)
http://www.ricis.com/ 866-99-RICIS Toll Free Fax (US)
708-444-2697 Fax (International)

RICIS, Inc. is a member of the Public Safety Alliance Group

This email and any attachments that are included in it have been scanned
for malicious or inappropriate content and are believed to be safe.

Feb 6 '06 #6

Ivan Voras

Lee Leahu wrote:

However, I have encountered the following error after I have started my 381st thread:

This number (actually, ~380) is suspicious when seen with threads
because it almost always means running out of process address space. As
threads are part of a single process, and that process has (usually
today) 32bit address space, and modern Linux systems allocate huge
amounts of memory per-thread (8 MB for threads' stack!), and the kernel
reserves ~1MB in the address space for itself, you can easily reach >
4GB of total allocated memory.

This *doesn't* mean you are actually using 4GB of memory (i.e. storing
values to it), only that the process gets "assigned" this memory. Python
doesn't support specifying thread attributes AFAIK, so you should find a
way to specify default thread stack size, possibly with an environment
variable (see manual page for "pthread" library calls). Try 64k - your
task seems simple enough :)

Feb 7 '06 #7

by: Michael Williams | last post by:

Hi, I am trying to understand the performance implications of running a number of separate ActiveXexe processes as opposed to a single ActiveXexe with multiple threads on a Windows 2000 server....

Visual Basic 4 / 5 / 6

C#, Threads, Events, and DataGrids/DataSets

by: Dennis Owens | last post by:

I am trying to run a thread off of a form, and every once in a while the thread will raise an event for the form to read. When the form gets the event, the form will place the event into a dataset...

.NET Framework

fork or threads

by: Andreas Müller | last post by:

i need two loops that run forever. one of it receives data and stores it to a vector. the other one writes the elements of the vector to the disk. this means the vector is a receiver buffer. How...

C / C++

Sharing DLL between threads

by: Ram P. Dash | last post by:

Hi: I've a third party DLL (not a .NET class library PE) which I use using DllImport attribute in my application running in multiple threads invoking different methods of the same DLL. The...

C# / C Sharp

How to transfer a memory block(not pointer) between two threads

by: jiing | last post by:

Now I am try to transfer a memory pointer between two threads. but there has a error mesage. _CrtIsValidHeapPointer(pUserData) in dbgheap.c I lookup google and find it's seems to be the local...

.NET Framework

Controlling Threads With MUTEX

by: Alex | last post by:

Dear colleagues I am programming an application which starts a lot of threads for some calculations. The problem that I have is that when I start up to 100 threads then everything works fine....

Visual Basic .NET

Threads vs Processes

by: Carl J. Van Arsdall | last post by:

Alright, based a on discussion on this mailing list, I've started to wonder, why use threads vs processes. So, If I have a system that has a large area of shared memory, which would be better? ...

Python

what are the memory caps for threads running as a CLR stored procedure executed by sql server 2005? is it limited by OS only or also by sql servers memory limits? e.g. lets say my clr stored procedure is executed by sql server 2005 then it creates 10

by: DR | last post by:

what are the memory caps for threads running as a CLR stored procedure executed by sql server 2005? is it limited by OS only or also by sql servers memory limits? e.g. lets say my clr stored...

C# / C Sharp

Python threads and memory usage

by: Mike | last post by:

Hi, I'm writing client-server application in Python. It's monitoring system, where server listen and waits for TCP connections, and every connection takes own thread. Every thread puts data from...

Python

167

What's the connection between objects and threads?

by: darren | last post by:

Hi I have to write a multi-threaded program. I decided to take an OO approach to it. I had the idea to wrap up all of the thread functions in a mix-in class called Threadable. Then when an...

C / C++

Easy Steps to Fix "Canon Printer Won't Connect to WiFi Network"

by: taylorcarr | last post by:

A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...

General

How to turn on java script in a villaon keypad mobile phone

by: Charles Arthur | last post by:

How do i turn on java script on a villaon, callus and itel keypad mobile phone

Java

Basic Javascript concepts

by: aa123db | last post by:

Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...

Javascript

Merging data from multiple Excel files

by: ryjfgjl | last post by:

In our work, we often receive Excel tables with data in the same format. If we want to analyze these data, it can be difficult to analyze them because the data is spread across multiple Excel files...

Data Management

Migrating Website to Cloud - Emmanuel Katto

by: emmanuelkatto | last post by:

Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel

General

Navigating the Data Structures and Algorithms (DSA)

by: BarryA | last post by:

What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...

Algorithms / Advanced Math

Looking to do Android software development, any suggestions? Is flutter better?

by: nemocccc | last post by:

hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?

General

How to build RAID in BIOS?

by: Hystou | last post by:

There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...

Computer Hardware

What is ONU?

by: marktang | last post by:

ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...

General

threads and memory

Similar topics