473,326 Members | 2,095 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,326 software developers and data experts.

Problem with blocking portably on sockets and Queue?

Hi,

I have several threads communicating with each other using events
stored in Queues. Threads block on Queue.get() until somebody
publishes an event in thread's event queue.

I need to add support for sockets to the system. Thread needs to
unblock when:

- there is socket ready to be read, or
- there is event waiting in the queue

My first tought was to replace blocking on Queue.get() with blocking
on poll or select and dedicating file descriptors (created with os.pipe())
as an semaphore. Event publisher would write something to the write
end of the pipe when it puts an event to the queue, effectively
unblocking the receiver.

BUT I noticed that select.poll() is not available on Windows and
secondly Windows version of select.select() will accept only socket
descriptors.

What options do I have that are still portable also to Windows
platform?

---
Tero

Jul 18 '05 #1
2 3864
<posted & mailed>

Tero Saarni wrote:
Hi,

I have several threads communicating with each other using events
stored in Queues. Threads block on Queue.get() until somebody
publishes an event in thread's event queue.
So far so wonderful -- excellent architecture.

I need to add support for sockets to the system. Thread needs to
unblock when:

- there is socket ready to be read, or
- there is event waiting in the queue

My first tought was to replace blocking on Queue.get() with blocking
on poll or select and dedicating file descriptors (created with os.pipe())
as an semaphore. Event publisher would write something to the write
end of the pipe when it puts an event to the queue, effectively
unblocking the receiver.

BUT I noticed that select.poll() is not available on Windows and
secondly Windows version of select.select() will accept only socket
descriptors.

What options do I have that are still portable also to Windows
platform?


I think you could devote a thread specifically to the task of handling
sockets, only. That special thread would not block on any queue but
just on select; when its select shows that a socket is ready, you can
have the thread itself do the reading and post the appropriate special
message including the data read to the appropriate thread, or you could
pass the ready-for-reading socket object to the appropriate thread, or
the like. This needs a bit more care if the set of sockets to be
select'ed on changes with time; in this case, I think the socket-handler
thread would have to use a timeout on its select so the set of sockets
can be updated once in a while (either by another thread, in which
case you need a lock to protect the "set of sockets" shared data, or
by the socket-handling thread (SHT) itself in response to a queued
message -- the SHT would read non-blockingly from that queue periodically
when the select times out) -- or you could perhaps use a special
identifiable socket for the SHT to receive from other threads requests
to change the set of ('true') sockets to be selected on.
Alex

Jul 18 '05 #2
"Alex Martelli" <al***@aleax.it> wrote in message
news:9P*********************@news2.tin.it...
<posted & mailed>

Tero Saarni wrote:
Hi,

I have several threads communicating with each other using events
stored in Queues. Threads block on Queue.get() until somebody
publishes an event in thread's event queue.


So far so wonderful -- excellent architecture.

I need to add support for sockets to the system. Thread needs to
unblock when:

- there is socket ready to be read, or
- there is event waiting in the queue

My first tought was to replace blocking on Queue.get() with blocking
on poll or select and dedicating file descriptors (created with os.pipe()) as an semaphore. Event publisher would write something to the write
end of the pipe when it puts an event to the queue, effectively
unblocking the receiver.

BUT I noticed that select.poll() is not available on Windows and
secondly Windows version of select.select() will accept only socket
descriptors.

What options do I have that are still portable also to Windows
platform?


I think you could devote a thread specifically to the task of handling
sockets, only. That special thread would not block on any queue but
just on select; when its select shows that a socket is ready, you can
have the thread itself do the reading and post the appropriate special
message including the data read to the appropriate thread, or you could
pass the ready-for-reading socket object to the appropriate thread, or
the like. This needs a bit more care if the set of sockets to be
select'ed on changes with time; in this case, I think the socket-handler
thread would have to use a timeout on its select so the set of sockets
can be updated once in a while (either by another thread, in which
case you need a lock to protect the "set of sockets" shared data, or
by the socket-handling thread (SHT) itself in response to a queued
message -- the SHT would read non-blockingly from that queue periodically
when the select times out) -- or you could perhaps use a special
identifiable socket for the SHT to receive from other threads requests
to change the set of ('true') sockets to be selected on.
Alex


Good idea, thanks! I will try dedicating a special thread for socket
handling, although breaking out of select() for modifying the set of
sockets gets a little tricky as you pointed out.

Momentarily I was even thinking that I could break out of select() by
sending a signal when I want to modify the set of sockets, but
as I was almost expecting, it appeared that also signal support is
quite limited on Windows platform.

I guess that fallback solution would be to learn Windows
sychronization primitives (wait functions etc) and write two separate
implementations of relevant parts of the system. With some work
that could probably be hidden behind matching interfaces and a
factory method choosing the implementation on the fly.

---
Tero

Jul 18 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
by: Gonçalo Rodrigues | last post by:
Hi, I have a problem with threads and sockets. I'll try to describe the problem in words with pseudo-code. I've been working on a few classes to make it easier to work with threads. This...
6
by: Tom | last post by:
I try to write a simple tcp based job queue server. It's purpose is to get commands from a client and store them in a queue (STL). The server has a thread which checks periodically the queue,...
7
by: David Sworder | last post by:
Hi, I'm developing an application that will support several thousand simultaneous connections on the server-side. I'm trying to maximize throughput. The client (WinForms) and server communicate...
3
by: Matthew King | last post by:
Hi all I've written a asynchronous socket client class, but i've found that in order to consume it I have to use events, and cannot simply for example SocketClient client = new...
15
by: mrpolitics | last post by:
So I'm working with PureIRCD (http://sourceforge.net/projects/pure-ircd) and everything was fine untill yesterday when the server crashed. So I did a cold restart and staretd the server back up...
0
by: Ben | last post by:
I modified the logmonitor sdk example so it would work over a network. It works great when the client and server are running on the same PC and have administrator privileges. So I have two...
1
by: opi | last post by:
My blocking TCP server hangs in the Accept method when the client software sometimes gets an error. Are there any ways to stop this blocking in the Accept method so I dont have to restart the...
1
by: Ryan Liu | last post by:
Hi, I have a 100 clients/ one server application, use ugly one thread pre client approach. And both side user sync I/O. I frequently see the error on server side(client side code is same, but...
11
by: Krzysztof Retel | last post by:
Hi guys, I am struggling writing fast UDP server. It has to handle around 10000 UDP packets per second. I started building that with non blocking socket and threads. Unfortunately my approach...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.