473,882 Members | 1,663 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

context switch

Hi,

I have a client/server application, using one thread/client approach.

I see very high context switch/sec. What are the ways to reduce it? Each
thread sleep longer in its endless loop if not reducing thread?

On the same machine, I also run a mysql server. I can see same amount thread
in mysqld, seems mysql is also using one thread/client. But its
context/swith is much lower. How can I get same performance?

Thanks a lot!
Ryan

Jun 27 '08
28 5570
Ryan,

Sleeping at all isn't a good idea. I'd eliminate that code immediately.

I don't know why you are using QueueUserWork item to send the response
back. I mean, if it takes a good amount of time, then I can understand
breaking it down into smaller parts, but generally, I would write the
response back after you have processed the request in the callback from
reading the request using asynchronous IO.

Your needs for reducing context switches and maintaining the
responsiveness of the server are at odds with each other. If you want to
reduce the context switches, then you have to reduce the amount of
concurrent processing occuring on the machine. In doing that, you are going
to reduce the throughput, which is going to ultimately affect
responsiveness.
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard. caspershouse.co m

"Ryan Liu" <rl**@powercati .comwrote in message
news:%2******** *******@TK2MSFT NGP06.phx.gbl.. .
Nicholas , Thanks for your quick response.

I have 120 client machines connect to one server for the C/S application.
MySql runs on the same server machine. Each client keep one connection to
the db server as well. Clients and the server exchange very small data,
but very frequently.

I am using sync I/O to read from client, and use
ThreadPool.Queu eUserWorkItem to send response back.

I haven't use ThreadPool to read data from client, because 1: my current
approach is easy to write code, and 2: I am afraid if I handle to
ThreadPool and since server work load is high, will my receiving job fail
to be queued or wait in queue for too long time? Is ThreadPool suitable
for C/S application that constantly scan and trying to receive data from
clients?

I think maybe I wait longer in each loop or use one thread for 10 clients.
In each loop, scan 10 clients one by one, and sleep less in each loop. So
for each client, interval maybe still same. Will that help a lot?

Now it sleep 100 ms in each loop. As said, I have 120 loops running.
Generally, is 100 ms a suitable amount?

Thanks!

"Nicholas Paldino [.NET/C# MVP]" <mv*@spam.guard .caspershouse.c om>
??????:uk****** ********@TK2MSF TNGP06.phx.gbl. ..
>Ryan,

Generally, you are going to have context switches when you have more
threads that are running than there are processors to handle the threads.

How are you allocating the threads? Generally, you should probably
use the ThreadPool to assign tasks to your threads, as the ThreadPool
will take into account the number of processors on the machine to
determine how many threads to keep in the pool (and minimize context
switches).

Also, are you sure the context switches are causing a performance
impact? If you switch to less threads, how do you kno you will get the
same concurrency that you have now?
--
- Nicholas Paldino [.NET/C# MVP]
- mv*@spam.guard. caspershouse.co m

"Ryan Liu" <rl**@powercati .comwrote in message
news:%2******* *********@TK2MS FTNGP06.phx.gbl ...
>>Hi,

I have a client/server application, using one thread/client approach.

I see very high context switch/sec. What are the ways to reduce it? Each
thread sleep longer in its endless loop if not reducing thread?

On the same machine, I also run a mysql server. I can see same amount
thread in mysqld, seems mysql is also using one thread/client. But its
context/swith is much lower. How can I get same performance?

Thanks a lot!
Ryan

Jun 27 '08 #11
On Sun, 01 Jun 2008 18:33:26 -0700, Ryan Liu <rl**@powercati .comwrote:
Nicholas , Thanks for your quick response.

I have 120 client machines connect to one server for the C/S application.
120 clients is definitely enough to justify moving to the async API. Can
you confirm that you are talking about network i/o here?
MySql runs on the same server machine. Each client keep one connection
to
the db server as well. Clients and the server exchange very small data,
but
very frequently.

I am using sync I/O to read from client, and use
ThreadPool.Queu eUserWorkItem to send response back.
How are you dividing the read i/o per-client? Is that "one thread/client"
also? Or do you have a single thread handling all clients? If the
latter, how do you implement that? The Select() method? Something else?
I haven't use ThreadPool to read data from client, because 1: my current
approach is easy to write code, and 2: I am afraid if I handle to
ThreadPool
and since server work load is high, will my receiving job fail to be
queued
or wait in queue for too long time?
It depends on how you use the ThreadPool. If you use it correctly, you'd
only get blocked from receiving as long as your server had something else
it needed to do, in which case that might be okay. If you use it
incorrectly, you could deadlock the thread pool. :)
Is ThreadPool suitable for C/S
application that constantly scan and trying to receive data from clients?
Use the async API.
I think maybe I wait longer in each loop or use one thread for 10
clients.
In each loop, scan 10 clients one by one, and sleep less in each loop. So
for each client, interval maybe still same. Will that help a lot?
What loop? Are you saying that you are _polling_ in your read i/o?
Now it sleep 100 ms in each loop. As said, I have 120 loops running.
Generally, is 100 ms a suitable amount?
There should be no reason for any thread involved in i/o to sleep. If
there seems to be a reason, then it's almost certain that there's a more
fundamental problem with the implementation (like, it's polling).

Pete
Jun 27 '08 #12
On Sun, 01 Jun 2008 18:41:41 -0700, Nicholas Paldino [.NET/C# MVP]
<mv*@spam.guard .caspershouse.c omwrote:
Peter,

Well, the statement about the thread pool is true, in the sense that
it takes the number of threads into account when figuring out how many
threads it should keep in the pool. I didn't say that it kept a number
of threads equal to the number of processors =)
Right. I wasn't questioning that part of your statement. The "this" I
meant was the part about "and minimize context switches". Given the large
number of threads in the thread pool, I don't see anything about its
design that actually does minimize context switches. If you've got 25
busy thread pool threads, never mind 250, you're going to have lots of
context switches, possibly more than were really necessary.
The new parallel task library is better at this, I believe (the new
System.Threadin g.dll).
Can you elaborate? I wasn't aware there's new System.Threadin g stuff.
What did they add that helps?

Thanks!
Pete
Jun 27 '08 #13
More info:

I use performance counter fount that:

Threads used by mysql server on the same machine, context/switches is abut
5-15 per second.
Mine thread, very silly, is about 43,384,322 - 4,295,039,822

And as for I/O data bytes/sec , mysql is 350,000 to 439,951. My server
application is 0.(Suprise, I need check this again.)

And my application % process time is lower than mysql as well at the moment
I watch.

And whole syste ave disk queue length is vary greatly, from 1.5 to 15.845.

And system-threads average is 881,049, max is 29,934,599. So high! This is
read from Control Panel - Administrator Tool - Performance . I think in task
manager, total threads shows there should be less than 1000.

Thanks!
"Ryan Liu" <rl**@powercati .comдϢ :%2*********** *****@TK2MSFTNG P06.phx.gbl...
Hi,

I have a client/server application, using one thread/client approach.

I see very high context switch/sec. What are the ways to reduce it? Each
thread sleep longer in its endless loop if not reducing thread?

On the same machine, I also run a mysql server. I can see same amount
thread in mysqld, seems mysql is also using one thread/client. But its
context/swith is much lower. How can I get same performance?

Thanks a lot!
Ryan

Jun 27 '08 #14

"Peter Duniho" <Np*********@nn owslpianmk.com Ϣ:op* **************@ petes-computer.local. ..
On Sun, 01 Jun 2008 18:42:44 -0700, Ryan Liu <rl**@powercati .comwrote:
>[...]
I used asynchronous API before, but I was told by my customer, the
server
stops itself after run for 15 minutes.

I don't know that that means. Were you instructed by your customer to
make the server stop after 15 minutes of execution? Or did your customer
complain to you that the server would only run after 15 minutes of
execution? In either case, how is that relevant to the current topic.
The customer complains to me. So I changed back to one thread per client
with sync read/write approach. And have't dare try async again.
Oh, virus were found before on the customer's network and maybe still there.
The customer says that virus has no harm and they have it for years. They
will kill it later but now my application must live with it for months.
:-( My customer is a sub sub sidrary of a forturn 500.

>Instead write a while(!Stop) in each thread, when I use async approach,
I
call BeginRead() again in each call back method of EndRead, to create an
endless loop.

Correct. You can abort a BeginRead() that hasn't completed yet by closing
the i/o object (e.g. call Stream.Close()) . Then the i/o will complete
with an exception thrown when you call EndRead(). In that way you can
detect the closure of the object and do whatever cleanup you need to,
rather than calling BeginRead() again.
>Maybe wrong idea: I also worry will asynchronous read/write make the
server
not response in timely manner?

It should have the exact opposite effect. Using IOCP is the best way to
reduce CPU overhead and make the server run efficiently.
>I have 120 client machines connect to one server for the C/S application.
MySql runs on the same server machine. Each client keep one connection
to
the db server as well. Clients and the server exchange very small data,
but
very frequently.

I use mysql command "show full processlist", and I can see about 120
threads. But about 119 are sleeping.

Well, the fact is that there are other things about your own
implementation that, at least without further clarification from you, seem
particularly troubling. It's certainly theoretically possible to manage
120 clients with a "one thread/client" implementation, but only if you do
it correctly.

If you are in fact not dedicating a single thread to a particular client's
read and write both, but rather (as it seems from the vague information
you've provided so far) polling for reads and queuing each write to the
thread pool individually, then that seems like that would in fact be one
of the worst ways to try to implement a "one thread/client" design.
Are you sure this is the worest way? Then I made a big mistake. I am doing
just that.
If you are pretty sure, then I will change the code -- either dedicated
thread or try async I/O again.

>The applications runs well for most of time, but once a while, server
gets
very busy and slow.

From what we've heard so far, that doesn't seem surprising to me. You
should at a minimum fix your design so that you have a dedicated thread
for each client (don't use the thread pool), where that dedicated thread
handles both reads and writes with the client. Preferably, you should
just go ahead and use the async API, to take advantage of the efficiency
that IOCP will get you.

Pete
Thank you very much, Pete!
Jun 27 '08 #15

"Peter Duniho" <Np*********@nn owslpianmk.com Ϣ:op* **************@ petes-computer.local. ..
On Sun, 01 Jun 2008 18:57:54 -0700, Ryan Liu <rl**@powercati .comwrote:
>More info:

I use performance counter fount that:

Threads used by mysql server on the same machine, context/switches is
abut
5-15 per second.
Mine thread, very silly, is about 43,384,322 - 4,295,039,822

How CPUs do you have? I have a hard time imagining how you could possibly
have 4 billion context switches in a second. Even 43 million seems
implausible if your polling thread (assuming you have one...you haven't
confirmed that yet) sleeps for 10ms each loop.
I have one dure core CPU, 1G memory. (Total 1.2G in use with virtual
memory). I need check customer what extract CPU it is. It runs on server
2003.
I thread 100 ms for each read loop. But use ThreadPool to write back to
clients.

>>
And as for I/O data bytes/sec , mysql is 350,000 to 439,951. My server
application is 0.(Suprise, I need check this again.)

And my application % process time is lower than mysql as well at the
moment
I watch.

And whole syste ave disk queue length is vary greatly, from 1.5 to
15.845.

And system-threads average is 881,049, max is 29,934,599. So high!

What OS? On 32-bit Windows, I think the maximum number of threads,
assuming nothing else competing for resources, is something like 1000 per
process. Without an enormous number of processes, it's hard to see how
you could get up to 30 million threads (especially assuming that most
processes aren't themselves hosting an exorbitant number of threads).

On 64-bit Windows, my recollection is that the max is much higher of
course. In any case, I'd agree that even nearing 1 million threads seems
wrong. But again, how many CPUs is there on the hardware?

I reiterate: you really should move to the async API.

Pete
It is Windows 2003 server. And the process count in task manager is less
than 1000, as I remember. The unbeliever number is shown in Admin
Tool -Performance. Maybe numbers in performance counter are different.

Thanks!
Jun 27 '08 #16
On Sun, 01 Jun 2008 20:32:58 -0700, Ryan Liu <rl**@powercati .comwrote:
The customer complains to me. So I changed back to one thread per client
with sync read/write approach. And have't dare try async again.
Oh, virus were found before on the customer's network and maybe still
there.
Viruses on computers are a bad idea, no matter what. :) On the other
hand, it's possible you did something wrong in the async implementation
that caused the server to terminate prematurely. Or the virus could have
done it. One reason it's so bad to have a virus on the computer is that
when weird things happen, it's harder to know for sure it's your own code
that's causing the problem.

That said, if the virus were killing the server after 15 minutes with the
async implementation, I'd think it would with the current implementation
too. There's no way from here to know for sure, but it sounds like the
implementation itself might have had bugs.
[...]
>If you are in fact not dedicating a single thread to a particular
client's
read and write both, but rather (as it seems from the vague information
you've provided so far) polling for reads and queuing each write to the
thread pool individually, then that seems like that would in fact be one
of the worst ways to try to implement a "one thread/client" design.

Are you sure this is the worest way? Then I made a big mistake.
I am doing just that.
Sorry to say, yes. Think about what you're doing. On the one hand, it's
nice that you're sleeping. On the other hand, if you are sleeping between
each read, then you are forcing a context switch between each read. Even
if you're only sleeping when you run out of reads, you're still assigning
a different thread to each write operation. Again, this forces a context
switch for each write operation.

Basically, with each i/o operation, you force a context switch.
If you are pretty sure, then I will change the code -- either dedicated
thread or try async I/O again.
Please do. At the very minimum, take out the sleeps, and dedicate a
thread to each client. That way, the Windows thread scheduler can make
sure that any given thread will run as long as possible before a context
switch happens. Either the thread will run out of things to do, or it
will use up its timeslice when there's another thread ready to run.

Using the async API would be much better even still, but even switching to
a dedicated thread for each client would improve things a _lot_ as
compared to your current implementation.
[...]
Thank you very much, Pete!
You're welcome. Happy to help if I can. :)

Pete
Jun 27 '08 #17
On Jun 2, 2:57 am, "Peter Duniho" <NpOeStPe...@nn owslpianmk.com>
wrote:
Can you elaborate? I wasn't aware there's new System.Threadin g stuff.
What did they add that helps?
It's not actually released yet, but there's a CTP available. I've been
trying it out with a few mini-projects which are embarrassingly
parallel - plotting the Mandelbrot set and Conway's Game of Life.
Parallel Extensions makes this really, really easy - it was pretty
much just a case of turning "for" into "Parallel.F or" and rewriting
the body as a lambda expression. See my blog (last couple of posts)
for details - and search for "parallel extensions" to find a load more
stuff and the CTP.

From what I've seen, it's going to be very, very good. In terms of
context switch avoidance, it uses work stealing to allocate tasks, and
doesn't create more threads than cores (at least in the normal case -
it may do more if it can detect that cores are idle due to IO-
bottlenecked tasks).

Jon
Jun 27 '08 #18
On Sun, 01 Jun 2008 23:10:38 -0700, Jon Skeet [C# MVP] <sk***@pobox.co m>
wrote:
On Jun 2, 2:57 am, "Peter Duniho" <NpOeStPe...@nn owslpianmk.com>
wrote:
>Can you elaborate? I wasn't aware there's new System.Threadin g stuff.
What did they add that helps?

It's not actually released yet, but there's a CTP available. I've been
trying it out with a few mini-projects which are embarrassingly
parallel - plotting the Mandelbrot set and Conway's Game of Life.
Parallel Extensions makes this really, really easy - it was pretty
much just a case of turning "for" into "Parallel.F or" and rewriting
the body as a lambda expression. See my blog (last couple of posts)
for details - and search for "parallel extensions" to find a load more
stuff and the CTP.
Figures. The one post where you neglect to include your .sig (with the
blog link), and it's the one that has me looking for your blog. :)

Anyway, thanks...sounds very cool. I'll check it out when I get a
chance. (Don't worry about the blog link...I can grab it from another
post).

Pete
Jun 27 '08 #19
On Jun 2, 7:36 am, "Peter Duniho" <NpOeStPe...@nn owslpianmk.com>
wrote:

<snip>
Figures. The one post where you neglect to include your .sig (with the
blog link), and it's the one that has me looking for your blog. :)
Oops. That's the difference between posting with Google Groups and
posting with Gravity. (Must investigate whether Google Groups has sig
functionality lurking somewhere.)
Anyway, thanks...sounds very cool. I'll check it out when I get a
chance. (Don't worry about the blog link...I can grab it from another
post).
Might as well include it now though :)

http://msmvps.com/jon.skeet

Jon
Jun 27 '08 #20

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

0
3152
by: Johannes B. Ullrich | last post by:
--=-WKgoK98ejo9BZyGYc3N/ Content-Type: text/plain Content-Transfer-Encoding: quoted-printable I am having problems with MySQL 4.0.12 on RedHat Advanced Server 2.1 using a dual Xeon with 8 GByte of RAM. I have a database collecting logs. Each day, a new table is created. In order to allow for queries across more than one day, I use 'MERGE'
13
7478
by: William Stacey | last post by:
Using the following code sample: public byte Get() { // <= Possible to switch Here?? lock(syncLock) { //Do something in Get(). } }
5
3694
by: Nadav | last post by:
Hi, I am using FileStream's Async API: BeginRead/EndRead, upon completion callback execution I use the read data and call EndRead, Taking that in mind, I Wonder... does calling EndRead will cause a context switch? What is the kernel object used for blocking EndRead calls? Event and mutex cause a context switch even when the object is signaled and no wait is needed, usage of critical section prevent this switch from happening... what is the...
13
2115
by: Michael M. | last post by:
In Perl, it was: ## Example: "Abc | def | ghi | jkl" ## -"Abc ghi jkl" ## Take only the text betewwn the 2nd pipe (=cut the text in the 1st pipe). $na =~ s/\ \|(.*?)\ \|(.*?)\ \|/$2/g; ## -- remove in text
20
310
by: Ryan Liu | last post by:
Hi, I have a client/server application, using one thread/client approach. I see very high context switch/sec. What are the ways to reduce it? Each thread sleep longer in its endless loop if not reducing thread? On the same machine, I also run a mysql server. I can see same amount thread in mysqld, seems mysql is also using one thread/client. But its context/swith is much lower. How can I get same performance?
0
9931
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, well explore What is ONU, What Is Router, ONU & Routers main usage, and What is the difference between ONU and Router. Lets take a closer look ! Part I. Meaning of...
0
11108
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10725
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
10830
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
10403
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
1
7956
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupr who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms. Adolph will...
0
7113
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5781
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
3
3226
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.