Thread Pool versus Dedicated Threads

一首诗 wrote:

Hi all,

<snip>

While interesting, there isn't really a C++ question in there. You
would get more insight on comp.programmin g.threads.

--
Ian Collins.

On Aug 14, 8:20 am, ??? <newpt...@gmail .comwrote:

Recently I had a new coworker. There is some dispute between us.

The last company he worked for has a special networking
programming model. They split the business logic into
different modules, and have a dedicated thread for the each
module. Modules exchanged info through a in-memory message
queue.

In my opinion, such a model means very complicated
asynchronous interaction between module.

If there's common data, there's always a more or less
complicated asynchronous interaction between modules. The
dedicated thread model normally reduces the "common data" to
just the message queue, which makes things significantly
simpler.

A simple function call between modules would require a timer
to avoid waiting for answer forever. And if a module was
blocked by IO (such as db query), other modules depends on
would have to wait for it.

Yup. That's the downside. The single, dedicated thread is (or
can be) a bottleneck. Of course, such bottlenecks can occur
anyway; if your manipulating a shared resource, for example,
which needs locking.

For example, if module A want to query db, it would

1. save states in a list
2 .sending a message to db-adapter module (a thread dedicated for db
operation)
3. start a timer
4. if response message arrived on time, retrieve states from the
list, and go on
5. if timer fires, log an error message and cancel the operation ??
send an error notify to user??

I'd put the time-out in the DB adapter module. Other than this:
what's the difference between putting the request in a single
block and posting it to the message queue, and passing the
information as arguments to a function?

My new coworker had written 300,000 lines of code in this
model and claimed this is the most simple way to write a
network application. He said we could implement a message
queue in half-a day and message would make interface much more
clear.

It's typically easier to get the code right using the message
queue, but it's not a silver bullet. You can still end up with
deadlocks. But you're much less likely to have problems due to
two threads accessing the same data without sufficient
synchronization .

I think if module interact with each other through function
calls and a thread/process pool model would be more easier, in
which each thread/ process has no dedicated job but handle
whatever the master thread give it.

A "thread/process pool model" doesn't mean anything. I'm not
sure what real alternative you're suggesting. Most places I've
worked at use a thread per client connection; on receiving a
request, the thread either grabs whatever locks it needs and
does the work, or forwards it to the dedicated thread (which
then doesn't need any locks, because it is the only thread which
accesses the information). Both models work. Which one is
better depends on the application.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

"James Kanze" <ja*********@gm ail.comwrote:

A "thread/process pool model" doesn't mean anything. I'm not
sure what real alternative you're suggesting.

The real alternative is to create a similar message queue design, but completely break the relationship of client connections to threads. Client connections exist on as many or as few threads as needed by the scalibility of the comms library. Requests coming in are packaged and posted to a message queue.
A pool of worker threads, proportional to the number of virtual CPUs in the server (rather than the number of client connections) pull requests from the queue, process the request, and then go back to see if theres anything in the queue to process.

This sort of design can ultimately be far better tuned to keep the CPU cores as busy as possible, while minimising needless context switches from having an "active" thread count far in excess of CPU availability. Given a database server that can, likewise, process multiple requests at once using asynchronous file io, this design will keep the database busy, rather than continually bottlenecking in the single DB "object" thread.

On 2008-08-14 08:20, 一首诗 wrote:

Hi all,

Recently I had a new coworker. There is some dispute between us.

The last company he worked for has a special networking programming
model. They split the business logic into different modules, and have
a dedicated thread for the each module. Modules exchanged info
through a in-memory message queue.

In my opinion, such a model means very complicated asynchronous
interaction between module. A simple function call between modules
would require a timer to avoid waiting for answer forever.
And if a module was blocked by IO (such as db query), other modules
depends on would have to wait for it.

What do u think about it? Is there any successful projects that could
prove which model is **right**?

In general I think you might be right, but when dealing with networking
there is usually a very layered architecture with one-way communication
between the layers (i.e. a lower layer passing the processed data up to
a higher layer). In that case the message-passing model makes very much
sense since it models the actual workings very well and makes each layer
simple to implement (if there are any packages in the in-queue you
process it and put the result in the out-queue, if there are no packages
in the in-queue you wait 'till there are).

For other kinds of tasks it might be easier to let one thread handle the
work-package in all the steps (and modules). Of course there are other
models and combinations, and which one is the best for a given purpose
is not always clear until you have tried a few.

--
Erik Wikström

On Aug 14, 4:24 pm, "Chris Becke" <chris.be...@gm ail.comwrote:

"James Kanze" <james.ka...@gm ail.comwrote:

A "thread/process pool model" doesn't mean anything. I'm not
sure what real alternative you're suggesting.

The real alternative is to create a similar message queue
design, but completely break the relationship of client
connections to threads.

That's a valid solution if there is no client specific data.
That's not always the case, however.

Client connections exist on as many or as few threads as
needed by the scalibility of the comms library. Requests
coming in are packaged and posted to a message queue. A pool
of worker threads, proportional to the number of virtual CPUs
in the server (rather than the number of client connections)
pull requests from the queue, process the request, and then go
back to see if theres anything in the queue to process.

Again, it depends. If requests constantly use shared data,
there's no point in having more than one thread to handle them.
If requests never use shared data, there's no point in not
handling them immediately in the receiving thread.

This sort of design can ultimately be far better tuned to keep
the CPU cores as busy as possible, while minimising needless
context switches from having an "active" thread count far in
excess of CPU availability.

Do you have actual measurements from a real application to
support this claim. I doubt that it's true for most
applications.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

Hi all,

Thanks for all your help! After I read all your posts and reconsider
my coworker's arguments, I think some explanation may be needed.

1. Chris explained exactly my real alternative solutions in this post.

2. Also as James pointed out, the most valuable point of a 'dedicated
model' is that no lock is needed as only one thread would touch the
data.

3. As Erik wrote "there is usually a very layered architecture",
whether there should have a layered architecture, is a key
consideration whether to use a 'dedicated model'.

4. About shared data. Yes, of course there are shared data between
each client. Actually we are building an SIP server for VOIP and
Instant Message. But in the case of a web server, isn't there are
also shared data between each client? ...

(Sorry I have to attend a meeting, I will further explain my
consideration later.)

On Aug 14, 10:24*pm, "Chris Becke" <chris.be...@gm ail.comwrote:

"James Kanze" <james.ka...@gm ail.comwrote:

A "thread/process pool model" doesn't mean anything. *I'm not
sure what real alternative you're suggesting.

The real alternative is to create a similar message queue design, but completely break the relationship of client connections to threads. Client connections exist on as many or as few threads as needed by the scalibility ofthe comms library. Requests coming in are packaged and posted to a messagequeue.
A pool of worker threads, proportional to the number of virtual CPUs in the server (rather than the number of client connections) pull requests fromthe queue, process the request, and then go back to see if theres anythingin the queue to process.

This sort of design can ultimately be far better tuned to keep the CPU cores as busy as possible, while minimising needless context switches from having an "active" thread count far in excess of CPU availability. Given a database server that can, likewise, process multiple requests at once using asynchronous file io, this design will keep the database busy, rather than continually bottlenecking in the single DB "object" thread.

>This sort of design can ultimately be far better tuned to keep

>the CPU cores as busy as possible, while minimising needless
context switches from having an "active" thread count far in
excess of CPU availability.

>Do you have actual measurements from a real application to
support this claim. I doubt that it's true for most
applications .

Microsoft Windows needs to allocate stack space for each thread created. On the 32bit version of the OS then, this means an immediately scalibility problem :- with only 2Gb of address space per process, this implies a hard limit of 2048 connections (threads) per server. Even on a 64bit OS the working set added to the process for each thread means that phsyical hardware limits will be reached that much faster than a system that uses asynchronous IO to keep lots of connections on one thread.

On Aug 15, 10:03 am, "Chris Becke" <chris.be...@gm ail.comwrote:

This sort of design can ultimately be far better tuned to keep
the CPU cores as busy as possible, while minimising needless
context switches from having an "active" thread count far in
excess of CPU availability.

Do you have actual measurements from a real application to
support this claim. I doubt that it's true for most
applications.

Microsoft Windows needs to allocate stack space for each
thread created. On the 32bit version of the OS then, this
means an immediately scalibility problem :- with only 2Gb of
address space per process, this implies a hard limit of 2048
connections (threads) per server. Even on a 64bit OS the
working set added to the process for each thread means that
phsyical hardware limits will be reached that much faster than
a system that uses asynchronous IO to keep lots of connections
on one thread.

That's a different problem, but yes, it does have to be taken
into account. The cost of creating a thread can also be an
issue, if connections are short lived (e.g. as in an HTML
server).

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

On Aug 15, 4:07 am, 一首诗 <newpt...@gmail .comwrote:

Thanks for all your help! After I read all your posts and
reconsider my coworker's arguments, I think some explanation
may be needed.

1. Chris explained exactly my real alternative solutions in
this post.

2. Also as James pointed out, the most valuable point of a
'dedicated model' is that no lock is needed as only one thread
would touch the data.

3. As Erik wrote "there is usually a very layered
architecture", whether there should have a layered
architecture, is a key consideration whether to use a
'dedicated model'.

4. About shared data. Yes, of course there are shared data
between each client. Actually we are building an SIP server
for VOIP and Instant Message. But in the case of a web
server, isn't there are also shared data between each client?
...

Sometimes. Sometimes not. Of course, only mutable shared data
is a problem. And it depends on who's using it, when. As I
said, there's no silver bullet. It all depends on the
application.

--
James Kanze (GABI Software) email:ja******* **@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34