I am monitoring/controlling some realtime activities in a manufacturing
process.
When a part comes into my station, I have a bunch of processing to do.
There are 30-40 data acquisition and data processing steps that need to
be performed. Many of the steps rely on the results of other steps for
their processing.
I have each step coded as a separate instance. The way I'd like to
execute all the steps is to spawn a thread for each of the steps. Each
step's thread would:
1. Wait for all dependent steps to complete (WaitHandle.Wai tAll).
2. Perform the desired action
3. Set a ManualResetEven t to signify that this step is done.
Now my problem is that a) I have some serious realtime issues to
contend with; b) I'd like to use a ThreadPool but I think I can only
have 20 or so threads running in the .Net ThreadPool.
Should I write my own thread pool for this, or is a thread pool the
wrong way to go? By the way, I expect to have 5-8 threads running at
any given time with another 30-40 threads waiting for an event. 8 1462
JS <st********@ic. net> wrote: I am monitoring/controlling some realtime activities in a manufacturing process.
When a part comes into my station, I have a bunch of processing to do. There are 30-40 data acquisition and data processing steps that need to be performed. Many of the steps rely on the results of other steps for their processing.
I have each step coded as a separate instance. The way I'd like to execute all the steps is to spawn a thread for each of the steps. Each step's thread would:
1. Wait for all dependent steps to complete (WaitHandle.Wai tAll). 2. Perform the desired action 3. Set a ManualResetEven t to signify that this step is done.
Now my problem is that a) I have some serious realtime issues to contend with; b) I'd like to use a ThreadPool but I think I can only have 20 or so threads running in the .Net ThreadPool.
Should I write my own thread pool for this, or is a thread pool the wrong way to go? By the way, I expect to have 5-8 threads running at any given time with another 30-40 threads waiting for an event.
There are various other threadpools around, including my own: http://www.pobox.com/~skeet/csharp/miscutil
It doesn't sound like one thread per step is a good idea though. Can
you not work out which of the steps can actually be executed in
parallel, and which can't, and keep the number of threads created down
in that way?
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Yes I might be able to work out which steps can be executed at a given
time. However, a thread pool will do this for me just by having each
step wait on its own inputs.
Is there a reason to avoid a thread pool for this situation?
Here's some more background:
The user has complete control over the list of steps. There are
hundreds of possible steps to choose from. Each step may have a set of
inputs and/or a set of outputs. A step can't begin processing until
all its inputs are ready. Once a step is done with its processing, it
sets each of its outputs to the ready state.
A step is typically either a data acquisition step or a data processing
step.
Here's an example of a set of steps:
Acquire Motion Data : output={MotionD ata1}
Acquire Temperature Data : output={Temp1}
Check Temperature : input={Temp1}, output={TempOK1 }
Check Motion Data : input={MotionDa ta1}, output={MotionO K1}
Check for Emergency Stop : input={TempOK1, MotionOK1}, output={EStop1}
This is a very simple set of steps. In a real system, there would be
5-15 data acquisition steps and 20-50 data processing steps.
I think I'll put together a short but complete program that details how
the software is written -- perhaps that will be instructive.
Thanks.
JS <st********@ic. net> wrote: Yes I might be able to work out which steps can be executed at a given time. However, a thread pool will do this for me just by having each step wait on its own inputs.
Is there a reason to avoid a thread pool for this situation?
I can't see that a thread pool would actually help you. Thread pools
are good for lots of short-running tasks - not tasks which need to wait
until something else has happened first.
Here's some more background: The user has complete control over the list of steps. There are hundreds of possible steps to choose from. Each step may have a set of inputs and/or a set of outputs. A step can't begin processing until all its inputs are ready. Once a step is done with its processing, it sets each of its outputs to the ready state.
A step is typically either a data acquisition step or a data processing step.
Here's an example of a set of steps:
Acquire Motion Data : output={MotionD ata1} Acquire Temperature Data : output={Temp1} Check Temperature : input={Temp1}, output={TempOK1 } Check Motion Data : input={MotionDa ta1}, output={MotionO K1} Check for Emergency Stop : input={TempOK1, MotionOK1}, output={EStop1}
This is a very simple set of steps. In a real system, there would be 5-15 data acquisition steps and 20-50 data processing steps.
I think I'll put together a short but complete program that details how the software is written -- perhaps that will be instructive.
Possibly. Are you actually sure that multi-threading will be useful
here at all? Are these definitely steps which can occur properly in
parallel anyway, using different resources? For instance, there's no
point (at least on single processor machines) in running two CPU-
intensive tasks at once.
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
Yes, many of the data acquisition steps tell a PCI card (or other
resource) to begin data collection. Then the acquisition step waits
until the data is done. Data acquisition can take a variable amount of
time to complete, depending on how much data is acquired.
Here's a scenario for you:
1. Acquire dataset 1 (very large data set takes 50 ms to acquire)
2. Acquire dataset 2 (small data set takes 1 ms to acquire)
3. Process dataset 1 (takes 10ms)
4. Process dataset 2 (takes 30ms)
Time=0ms Start Acquiring dataset 1 and 2
Time=1ms Start processing dataset 2
Time=31ms <done processing dataset 2>
Time=50ms Start processing dataset 1
Time=60ms Done processing dataset 2
In this very simple example the CPU is really only running on one
thread at a time. What's nice about a thread pool to do this is that
it might optimize the use of the CPU. That's what I'm hoping anyway.
What is unknown to me is whether the threadpool overhead will be a
problem.
Additionally, I don't know what kind of steps I'll be executing.
Whereas the above example showed no more than 1 thread running (i.e.
not waiting) at a time, there are other possible cases. There may be 1
data acquisition step followed by 50 processing steps on this data.
This would result in 50 threads going at the same time. I guess I'm
trying to gage how much performance I might lose if I spawn 50 threads
vs. having a single thread make 50 method calls -- this is the
worst-case scenario. Any idea on this?
BTW, I started writing a short but complete program, and it's not
ending up very short. I'll keep working on it, but I may have to
e-mail the source instead of posting it.
Hi,
I suggest you write a simple scheduler, that maintains a list of all tasks
that haven't been started yet. Everytime it runs, it would start (and remove
from the list) the tasks that have all their inputs available (on the first
run, the tasks without inputs would be started). Then you would run the
scheduler again each time a new output becomes available, or when any of the
previous tasks is finished. This way, you could make use of the threadpool,
though I wouldn't do so. If we're talking about 20 - 50 tasks, each of them
taking some time, the thread creation overhead should be insignificant, so
you just might start a new thread everytime you need one and not worry about
draining the threadpool.
HTH,
Stefan
"JS" <st********@ic. net> wrote in message
news:11******** *************@f 14g2000cwb.goog legroups.com... Yes, many of the data acquisition steps tell a PCI card (or other resource) to begin data collection. Then the acquisition step waits until the data is done. Data acquisition can take a variable amount of time to complete, depending on how much data is acquired.
Here's a scenario for you:
1. Acquire dataset 1 (very large data set takes 50 ms to acquire) 2. Acquire dataset 2 (small data set takes 1 ms to acquire) 3. Process dataset 1 (takes 10ms) 4. Process dataset 2 (takes 30ms)
Time=0ms Start Acquiring dataset 1 and 2 Time=1ms Start processing dataset 2 Time=31ms <done processing dataset 2> Time=50ms Start processing dataset 1 Time=60ms Done processing dataset 2
In this very simple example the CPU is really only running on one thread at a time. What's nice about a thread pool to do this is that it might optimize the use of the CPU. That's what I'm hoping anyway. What is unknown to me is whether the threadpool overhead will be a problem.
Additionally, I don't know what kind of steps I'll be executing. Whereas the above example showed no more than 1 thread running (i.e. not waiting) at a time, there are other possible cases. There may be 1 data acquisition step followed by 50 processing steps on this data. This would result in 50 threads going at the same time. I guess I'm trying to gage how much performance I might lose if I spawn 50 threads vs. having a single thread make 50 method calls -- this is the worst-case scenario. Any idea on this?
BTW, I started writing a short but complete program, and it's not ending up very short. I'll keep working on it, but I may have to e-mail the source instead of posting it.
JS <st********@ic. net> wrote: Yes, many of the data acquisition steps tell a PCI card (or other resource) to begin data collection. Then the acquisition step waits until the data is done. Data acquisition can take a variable amount of time to complete, depending on how much data is acquired.
Here's a scenario for you:
1. Acquire dataset 1 (very large data set takes 50 ms to acquire) 2. Acquire dataset 2 (small data set takes 1 ms to acquire) 3. Process dataset 1 (takes 10ms) 4. Process dataset 2 (takes 30ms)
Time=0ms Start Acquiring dataset 1 and 2 Time=1ms Start processing dataset 2 Time=31ms <done processing dataset 2> Time=50ms Start processing dataset 1 Time=60ms Done processing dataset 2
In this very simple example the CPU is really only running on one thread at a time. What's nice about a thread pool to do this is that it might optimize the use of the CPU.
In what way? What are you expecting a thread pool (as opposed to just
starting that many threads) to do for you automatically? I don't think
thread pools work the way you expect them to, I'm afraid.
I'll look at your email when I can - I'm looking after my 1 year old
son for most of this week though, so it may be a little while.
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too
> > In this very simple example the CPU is really only running on one thread at a time. What's nice about a thread pool to do this is
that it might optimize the use of the CPU. In what way? What are you expecting a thread pool (as opposed to just
starting that many threads) to do for you automatically? I don't
think thread pools work the way you expect them to, I'm afraid.
The reason it would be a thread pool instead of just spawning threads
is because I was under the impression that starting a new thread took
some time, and that the thread pool had less overhead in starting the
thread.
Another reason I thought it might be optimized is that each thread
would basically be doing:
WaitHandle [] waitEvents = this.GetInputEv ents();
WaitHandle.Wait All(waitEvents) ;
// do some work using inputs, filling outputs with data
this.SetAllOutp utsReady();
this.ReadyEvent .Set();
return; // thread done
When my processing is triggered, I need to run all of the 30-50 threads
and have them all finish ASAP. So my view was that if our OS (XP) is
good, it ought to be better at scheduling thread processing than
anything I write. Perhaps I'm wrong.
Jon, no rush to look at that code I sent and have fun with your
1-year-old (I have been through it twice now!). You may find the
concept in the code mildly interesting.
All others: I could not post the code here because it was a little too
large. I can post to a website if anyone is interested. My e-mail is
jimnospam at lorusinc.com (change ' at ' to @ but don't remove the
'nospam').
JS <st********@ic. net> wrote: The reason it would be a thread pool instead of just spawning threads is because I was under the impression that starting a new thread took some time, and that the thread pool had less overhead in starting the thread.
But the thread pool has a limited number of threads available - 25 per
processor by default. Thread pools just aren't designed for your kind
of work, where the thread will wait around for a long time before
finishing (either because it's doing work or because it's waiting for
other things).
Another reason I thought it might be optimized is that each thread would basically be doing:
WaitHandle [] waitEvents = this.GetInputEv ents(); WaitHandle.Wait All(waitEvents) ; // do some work using inputs, filling outputs with data this.SetAllOutp utsReady(); this.ReadyEvent .Set(); return; // thread done
But the thread pool can't get the thread to switch to another context
during the WaitAll
When my processing is triggered, I need to run all of the 30-50 threads and have them all finish ASAP. So my view was that if our OS (XP) is good, it ought to be better at scheduling thread processing than anything I write. Perhaps I'm wrong.
I'm sure XP can handle 50 threads - although starting a new batch of 50
for every single job would be a bad idea, have you considered having
one thread per *type* of step, and have the same 50 threads running the
whole time, waiting for things to do?
However, it seems to me that you're unlikely to really *need* 50
threads. If one step *only* depends on the output of the previous step,
then they can effectively be combined into one step. Do this everywhere
you can, and you should be able to significantly reduce the number of
threads required. Similarly a thread which is needed for some parallel
work early on can then do some more parallel work later, when it's
available - there's no need to start a new thread for each of them.
--
Jon Skeet - <sk***@pobox.co m> http://www.pobox.com/~skeet
If replying to the group, please do not mail me too This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: dcrespo |
last post by:
Hi all,
How can I get a raised exception from other thread that is in an
imported module?
For example:
---------------
programA.py
---------------
|
by: benmorganpowell |
last post by:
I have a small windows service which connects to a POP3 server at
defined intervals, scans the available messages, extracts the required
information and inserts the data into a SQL database. I am assuming
that this is not an uncommon piece of software.
I want to get an architecture that conforms as closely as possible with
the recommendations from Microsoft on developing Windows Services, but
to be honest I have found difficultly in...
|
by: nd02tsk |
last post by:
Hello!
I have a couple of final ( I hope, for your sake ) questions regarding
PostgreSQL.
I understand PostgreSQL uses processes rather than threads. I found this
statement in the archives:
"The developers agree that multiple processes provide
more benefits (mostly in stability and robustness) than costs (more
|
by: NOtcarvinSPAM |
last post by:
Obviously wrapping a critical section around access to some set of
shared state variables flushes any cached data, etc so that the threads
involved don't see a stale copy. What I was wondering is *what*
exactly gets flushed. Does the compiler some how determine the data
that is accessible from that thread, and flush just that set? (Seems
unlikely to me). Is it all data cached in registers etc? Or am I
overthinking this and instead...
|
by: Jordan |
last post by:
All,
I have a UI form calling a class object that contains a timer that
routinely draws intensive information to the screen (~30 fps). The
drawing is invoked on the main UI thread. I need the user to be able
to interact with the UI while the drawing is happening (dynamically
change drawing properties). Since the drawing is happening on the main
UI thread, the UI form doesn't receive the user control events
consistently (if at all).
| |
by: Morpheus |
last post by:
Hi,
I have been coding in Windows for many years so have a mindset to it,
so forgive any stupid questions.
Is it possible to create a multithread application in C++ that is
portable (OS/compiler)? In my Windows applications, I use
synchronization objects like, mutex, semaphores etc. Is there anything
similar in the Standard ANSI C++ or would I be best off doing this
myself, possibly creating a new class to handle any critical...
|
by: JC |
last post by:
Hi People,
Please I need your help.
This code run a thread ok but Not close later.
thanks...
private void RunServer(int aPortNumber)
{
|
by: Creativ |
last post by:
Why does Thread class not support IDisposable? It's creating quite
some problem. Namely, it can exhaust the resource and you have not
control over it.
|
by: Hapa |
last post by:
Does only reading (never writing) of a variable need thread synchronisation?
Thanks for help?
PS.
Anybody knows a Visual C++ news group?
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
| |
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |