473,401 Members | 2,068 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,401 software developers and data experts.

Queue read by multiple threads

I have a system in which I have a single thread that places data on a Queue.
Then I have one worker thread that waits until data is put on the thread and
dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece
of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent.
That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?
Nov 16 '05 #1
2 2701
Just thinking off the top of my head, one way to do it would be to use a 2
stage queue: You would have one "main" queue, and multiple "secondary" or
work queues (one for each thread). When you need to post (enqueue) data,
you would always enqueue to the main queue. A single dispatcher thread
would dequeue items from the main queue and enqueue the data onto each work
queue:

work queue 1 -->
worker thread
--> main queue --> dispatch thread <
work queue2 -->
worker thread

There are some issues that could come up. If the data item in the queue is
a reference type, you might have to synchronize access to it (or deep-copy
it). If the data item is a value type, like a struct, you will end up with
copies of it.

I'm sure that there are probably more optimized patterns for handling this,
but off the top, I think that this would work.

If you want to extend the Manual Reset Event, you might have to write your
own implementation. I'm pretty sure that the MRE and ARE are just thin
wrappers over O/S level synchronization events. See CreateEvent() in the
Platform SDK (DLLs, Processes, and Threads)

"Hollywood" <ho************@thzero.com> wrote in message
news:uS**************@tk2msftngp13.phx.gbl...
I have a system in which I have a single thread that places data on a Queue. Then I have one worker thread that waits until data is put on the thread and dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent. That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?

Nov 16 '05 #2
Hollywood <ho************@thzero.com> wrote:
I have a system in which I have a single thread that places data on a Queue.
Then I have one worker thread that waits until data is put on the thread and
dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece
of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent.
That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?


I suggest you have as many queues as you have threads, and put the same
work items on all of them. Any other way is going to make it a lot
harder to find out which is the next job, etc.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Hollywood | last post by:
I have a system in which I have a single thread that places data on a Queue. Then I have one worker thread that waits until data is put on the thread and dequeues the Queue and processes that...
2
by: Chuck | last post by:
I have not been able to find a good answer about the System.Collections.Queue.Synchronized() method and how it actually works or is to be used. If I create my Queue in the following manner: ...
6
by: James Radke | last post by:
Hello, I have a multithreaded windows NT service application (vb.net 2003) that I am working on (my first one), which reads a message queue and creates multiple threads to perform the processing...
0
by: Dave Coate | last post by:
I am working on a generic way to launch multiple similar processes (threads) at once, but limit the number of threads running at any one time to a number I set. As I understand it the following...
6
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any...
7
by: Michael D. Ober | last post by:
When calling Enqueue, the internal array may need to be reallocated. My question is by how much? In the old MFC array classes, you could tell MFC how many additional elements to add to the array...
12
by: Paul Rubin | last post by:
I'd like to suggest adding a new operation Queue.finish() This puts a special sentinel object on the queue. The sentinel travels through the queue like any other object, however, when...
5
by: Craig Buchanan | last post by:
I would like to monitor a POP3 mailbox with multiple clients. However, I want to ensure that each message is processed by only one client. In essence, I would like to treat a POP3 mailbox like a...
3
by: scriptlearner | last post by:
I am trying to put up a queue (through a logging thread) so that all worker threads can ask it to log messages. However, the problem I am facing is that, well, the logging thread itself is running...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However,...
0
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven...
0
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.