473,598 Members | 2,978 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

Threading problem when many sockets open

Hi all,

I have written a socket based service in python and under fairly heavy
traffic it performs really well. But i have encountered the following
problem: when the system runs out of file descriptors, it seems to
stop switching control between threads.

Here is some more detail:

The system has n+2 threads, where n is usually around 10. This was
implemented using the 'threading' and 'socket' modules in the python
2.5 standard library.

-- The "master" thread accepts new socket connections and then
enqueues the connection on a Queue (from the standard library).

-- There are n "handler threads" that pop a connection off of the
queue, read some number of bytes (~10), do some processing and then
send ~100 bytes back over the connection and close it.

-- The last thread is just a logging thread that has a queue of
messages that it writes to either stdout or a file.

Under pretty substantial load, the processing is quick enough that the
connections do not pile up very quickly. But, in some cases they do.
And what I found was that as soon as the size of the Queue of
connections reached a high enough number (and it was always the same),
all of the processing seemed to stay in the "master" thread. I
created a siege client that opened up 1000+ connections and the server
would go into a state where the master thread repeatedly polled the
socket and printed an error. The Queue size stayed fixed at 997 even
if i shut down all of the connectionso n the client side. Under
normal operating conditions, those connections would be detected as
broken in the handler thread and a message would be logged. So it
seems that the "handler" threads aren't doing anything. (And they are
still alive, i check that in the master thread).

OK. I hope all of this is clear. Currently, I've solved the problem
by putting a cap on the queue size and i haven't seen the problem
reoccur. But it would be nice to understand exactly what went wrong.

Thanks in advance.

Philip Zigoris I SPOCK I 650.366.1165
Spock is Hiring!
Aug 11 '07 #1
1 1162
In message <ma************ *************** ************@py thon.org>, Philip
Zigoris wrote:
... and the server
would go into a state where the master thread repeatedly polled the
socket and printed an error.
Is that because it cannot create a new socket to accept the connection on?
Aug 25 '07 #2

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: Gernot Hillier | last post by:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi! I'm the developer of a Linux ISDN application which uses embedded Python for controlling the communication. It starts several threads (i.e. one for each incoming call and for outgoing faxes) which run Python scripts in embedded Python interpreters which in turn do the real communication stuff. Any incoming data or confirmations of done jobs are sent via Email.
by: Eric | last post by:
I'm looking at this page in the MSDN right here: ms-help://MS.MSDNQTR.2003FEB.1033/cpref/html/frlrfsystemcollectionsarraylist classsynchronizedtopic2.htm (or online here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemcollectionsicollectionclasssyncroottopic.asp) And I'm interested in locking an ArrayList during the entire enumeration, as shown in the example code. My problem is that I'm STILL...
by: Matt | last post by:
Hello Everyone I am making an instant messenger program. I have used the MSDN sockets example to get started with this. I have transfered the code that is found within this project into a class and can access it fine The problem is, that the messenger sends a message to the server which relays the message to the correct user: great, but there is a thread running that is reading the incomming data and passes it to a process sub that checks...
by: richard | last post by:
OS: Winxp and Win2003 Visual Basic.NET 2003 MS-SQL Server 2000 hey all I am a newbie in vb.net but i have managed to build a simple chat server in vb.net using socket and a client connecting to it made in vb6. Some of the code in the server are based on the basic samples that i get from other authors. To give you an idea, the server uses sockets,
by: Remarkable | last post by:
Hello all I am trying to write a reliable web-crawler. I tried to write my own using recursion and found I quickly hit the "too many sockets" open problem. So I looked for a threaded version that I could easily extend. The simplest/most reliable I found was called Spider.py (see attached). At this stage I want a spider that I can point at a site, let it do it's thing, and reliable get a callback of sorts... including the html
by: Ryan Liu | last post by:
Hi, If I have many threads write to a variable(e.g. var++) and another thread read it on an interval base. For those writing thread, I know I need lock, or its value could be lower ( even I think it is mostly not going to happen for ++ operation since it is not something like read a value and wait sometime then write back in multiple threading environment, BTW, is this understanding right?).
by: Yehia A.Salam | last post by:
Hello, I am building a network application that make use of .Net Sockets, I created a class that works like a server and fires an event when anything arrives at the server, however I ran into some problems because of the WPF threading model, my code is like: //******************************************* public Window1(){ ... sConnection = new myserver();
by: RFleming | last post by:
I have a class that spawns a thread in a sub class to monitor data on a socket connection. Once data is found I read it as bytes and want to send it back to the parent class to a PopulateStatusRecord subroutine. I tried make the subroutine shared, but could not use the me keyword, so I don’t know how to pass the variable returndata outside of the thread and back to the parent class. The code below is psudeo code in areas to keep the...
by: Alexander Gnauck | last post by:
Hello, while using async sockets I ran into a strange problem which occurs only on some machines. I wrote a small demo application which can be used to reproduce the problem. You can download it from here: http://alex.ag-software.de/SocketTest.zip If you press the "Connect in new Thread" button from the test application you may be able to cause the System.IO.IOException: Unable to write data to the transport connection. While the...
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
by: muto222 | last post by:
How can i add a mobile payment intergratation into php mysql website.
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.