473,385 Members | 1,562 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,385 software developers and data experts.

Queue read by multiple threads

I have a system in which I have a single thread that places data on a Queue.
Then I have one worker thread that waits until data is put on the thread and
dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece
of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent.
That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?
Nov 16 '05 #1
2 2699
Just thinking off the top of my head, one way to do it would be to use a 2
stage queue: You would have one "main" queue, and multiple "secondary" or
work queues (one for each thread). When you need to post (enqueue) data,
you would always enqueue to the main queue. A single dispatcher thread
would dequeue items from the main queue and enqueue the data onto each work
queue:

work queue 1 -->
worker thread
--> main queue --> dispatch thread <
work queue2 -->
worker thread

There are some issues that could come up. If the data item in the queue is
a reference type, you might have to synchronize access to it (or deep-copy
it). If the data item is a value type, like a struct, you will end up with
copies of it.

I'm sure that there are probably more optimized patterns for handling this,
but off the top, I think that this would work.

If you want to extend the Manual Reset Event, you might have to write your
own implementation. I'm pretty sure that the MRE and ARE are just thin
wrappers over O/S level synchronization events. See CreateEvent() in the
Platform SDK (DLLs, Processes, and Threads)

"Hollywood" <ho************@thzero.com> wrote in message
news:uS**************@tk2msftngp13.phx.gbl...
I have a system in which I have a single thread that places data on a Queue. Then I have one worker thread that waits until data is put on the thread and dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent. That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?

Nov 16 '05 #2
Hollywood <ho************@thzero.com> wrote:
I have a system in which I have a single thread that places data on a Queue.
Then I have one worker thread that waits until data is put on the thread and
dequeues the Queue and processes that information. Works just fine.

However, I possibly need 1..m worker threads that can act on the same piece
of data, i.e. when a dequeue happens on the Queue it can't be removed from
the Queue until all threads have used it. I've tried to come up with good
ways of doing this but haven't found anything I liked or has worked. I
think I'd really like to just get rid of the Queue and send the packet of
data when I trigger the waiting threads, i.e. an expanded ManualResetEvent.
That way each thread can get the same piece of data and process it
independently. However, ARE and MRE are sealed and I've not seen any good
information on replicating them to be able to extend them to send a packet
along with the set trigger.

Thoughts or comments?


I suggest you have as many queues as you have threads, and put the same
work items on all of them. Any other way is going to make it a lot
harder to find out which is the next job, etc.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Nov 16 '05 #3

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

2
by: Hollywood | last post by:
I have a system in which I have a single thread that places data on a Queue. Then I have one worker thread that waits until data is put on the thread and dequeues the Queue and processes that...
2
by: Chuck | last post by:
I have not been able to find a good answer about the System.Collections.Queue.Synchronized() method and how it actually works or is to be used. If I create my Queue in the following manner: ...
6
by: James Radke | last post by:
Hello, I have a multithreaded windows NT service application (vb.net 2003) that I am working on (my first one), which reads a message queue and creates multiple threads to perform the processing...
0
by: Dave Coate | last post by:
I am working on a generic way to launch multiple similar processes (threads) at once, but limit the number of threads running at any one time to a number I set. As I understand it the following...
6
by: les | last post by:
Here's a class which uses 2.0 generics to implement an inter-thread message queue in C#. Any number of threads can post and read from the queue simultaneously, and the message object can be any...
7
by: Michael D. Ober | last post by:
When calling Enqueue, the internal array may need to be reallocated. My question is by how much? In the old MFC array classes, you could tell MFC how many additional elements to add to the array...
12
by: Paul Rubin | last post by:
I'd like to suggest adding a new operation Queue.finish() This puts a special sentinel object on the queue. The sentinel travels through the queue like any other object, however, when...
5
by: Craig Buchanan | last post by:
I would like to monitor a POP3 mailbox with multiple clients. However, I want to ensure that each message is processed by only one client. In essence, I would like to treat a POP3 mailbox like a...
3
by: scriptlearner | last post by:
I am trying to put up a queue (through a logging thread) so that all worker threads can ask it to log messages. However, the problem I am facing is that, well, the logging thread itself is running...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...
0
by: ryjfgjl | last post by:
In our work, we often need to import Excel data into databases (such as MySQL, SQL Server, Oracle) for data analysis and processing. Usually, we use database tools like Navicat or the Excel import...
0
by: taylorcarr | last post by:
A Canon printer is a smart device known for being advanced, efficient, and reliable. It is designed for home, office, and hybrid workspace use and can also be used for a variety of purposes. However,...
0
by: Charles Arthur | last post by:
How do i turn on java script on a villaon, callus and itel keypad mobile phone
0
by: aa123db | last post by:
Variable and constants Use var or let for variables and const fror constants. Var foo ='bar'; Let foo ='bar';const baz ='bar'; Functions function $name$ ($parameters$) { } ...
0
by: emmanuelkatto | last post by:
Hi All, I am Emmanuel katto from Uganda. I want to ask what challenges you've faced while migrating a website to cloud. Please let me know. Thanks! Emmanuel
0
BarryA
by: BarryA | last post by:
What are the essential steps and strategies outlined in the Data Structures and Algorithms (DSA) roadmap for aspiring data scientists? How can individuals effectively utilize this roadmap to progress...
1
by: nemocccc | last post by:
hello, everyone, I want to develop a software for my android phone for daily needs, any suggestions?

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.