I do something similar but in more asynch way.
I use queue - or it could be list or collection like in your case.
Main thread, which is getting data to process (files), puts every new item
into the queue and signals using Mointor.PulseAll to all threads that there
is item to process.
All worker threads were created during app. startup; are locking on queue
and sit in Monitor.Wait. When Pulse arrives, first thread (which exactly I
never know - but from tests it looks like MS does this in round-robin
fashion) gets kicked up, locks the queue, takes out item, releases the queue
and starts processing. Sometime later it is interrupted (usually because of
asynch calls inside thread) and next waiting thread is being kicked up. It
locks the queue and gets next item. If there are no items yet - it goes into
Monitor.Wait immediately.
As you see - there is no polling, no loops or Thread.Sleep calls. Sure I sit
also at 100% nearly when there are several threads running together. However
this is split between my threads and OS (5%) and MS SQL (40-60%) which are
used from threads. Because threads are created during startup - there is no
overhead with repeated creation of threads and corresponding clogging of
memory and GC kicking in.
About callbacks - I use callbacks from threads only for updating UI visual
feedback elements. Never to start new threads. And I wouldn't recommend it
to you in your case - in essence, if your thread finished processing, it
should check for next work item immediately; there is no sense to get back
to some another thread to start new thread etc..
Btw, I started with scheme like yours - polling and attempts to create
additional threads for each work item. However with Monitor.Pulse/Wait and
my own fixed thread pool final implementation is much faster and stable. My
app can process around up to thousand items per hour on 1GHz PC - every item
is file (1-500K) and causes several MS SQL operations. App runs for several
hours usually.
The only change apart from Monitor.Pulse/Wait you might need to consider -
how to signal end of run to all threads. It could be special item or some
static flag field, which is less preferable. Of course, each work item
is/should be processed completely independently from any other. Put all
parameters controlling processing into item object.
Also, take a look if you really need 12 threads. E.g. on my machine there is
no sense to try to run more than 4-5 simultaneously. Higher than that there
is no gain at all and degradation starts even when all connections are
available. If you have multiprocessor one and low disk activity during
processing - maybe 12. But I would try to measure before making conclusions.
And last one - take a look how you process your files in threads.
Asynchronous IO can help to speed up processing too.
HTH
Alex
"m" <sk****@speakeasy.net> wrote in message
news:rD********************@speakeasy.net...
Hello,
I have an application that processes thousands of files each day. The
filenames and various related file information is retrieved, related
filenames are associate and placed in a linked list within a single
object, which is then placed on a stack(This cuts down thread creation and
deletions roughly by a factor of 4). I create up to 12 threads, which then process a
single object off of the stack. I use a loop with a boolean statement,
stack.Count > 0. Then I check each thread to see if it is alive, if it is
not, then I create a new thread with a new object off of the stack which
is passed as the constructor parameter for a new threaded object. If the
thread is alive, then it merely goes on to check the status of the next thread in
line. This is a big process and running the CPU at 100% is not an issue, I
would just like to optimize my threading code in order to make my
application faster and more efficient. The ThreadPool class does not seem
like a good option for my needs, as my threads will be constantly
processing throughout their lifetime. I think that my constant polling of threads
could definitely be replaced with something like a thread callback upon
completion of its processing. How can I further reduce the threading overhead? Would
it be better to just reset all the variables in a thread and pass a new stack
object, without creating a new thread to overwrite the dead thread? My
code, while reliable so far, could easily be simplified and improved upon.
Thanks for any and all input :)