Context Switches, Web Gardens and InProc Session State

[=?Utf-8?B?QnJlbmRhbiBLYXk=?=]

Hi there,

We have an ASP.Net application that is generating huge numbers of Context
Switches when a certain number of active users are present. We tried changing
the configuration of the Application Pool to use a Web Garden and the number
of Context Switches decreased by more than 95% (that's the good news).

Our problem is that we have something in our session state that cannot be
serialized and therefore we cannot use StateServer for the Session mode.

We have been trying to find a way to get all requests associated with a
specific session to be dispatched to the one worker process. The Microsoft KB
article at http://support.microsoft.com/default.aspx/kb/822171 says:

"If an application keeps a connection alive and uses the same source port,
the requests are routed back to the same worker process."

We cannot seem to get this to occur for us. Does this apply to http requests
or is it only referring to ftp and other protocols served by IIS?

We have found plenty of places that state categorically that InProc session
state cannot be used with Web Gardens but we are hoping that the KB article
above knows something that they don't. Can anyone help us to tie particular
sessions to particular worker processes?

Thanks,

Brendan

[Anthony Jones]

"Brendan Kay" <Br********@dis cussions.micros oft.comwrote in message
news:FA******** *************** ***********@mic rosoft.com...

Hi there,

We have an ASP.Net application that is generating huge numbers of Context
Switches when a certain number of active users are present. We tried

changing

the configuration of the Application Pool to use a Web Garden and the

number

of Context Switches decreased by more than 95% (that's the good news).

Define 'Huge'. Are you sure that it was a problem? How many CPU cores do
you have and can you add more? I'm not sure how using a Web Garden actually
resolves the problem. What is the CPU usage rate like?

Our problem is that we have something in our session state that cannot be
serialized and therefore we cannot use StateServer for the Session mode.

We have been trying to find a way to get all requests associated with a
specific session to be dispatched to the one worker process. The Microsoft

KB

article at http://support.microsoft.com/default.aspx/kb/822171 says:

"If an application keeps a connection alive and uses the same source port,
the requests are routed back to the same worker process."

We cannot seem to get this to occur for us. Does this apply to http

requests

or is it only referring to ftp and other protocols served by IIS?

Are your clients connecting via a proxy or firewall, (even if they are
internal clients I've often seen configurations which unintentionaly route
HTTP through a firewall/proxy to an internal server)?

That said its not possible to guarantee a fixed connection. Additionally
browser will use more than one connection, the likely hood is then that the
browser will have one connection on one process and another to the other
process.

>
We have found plenty of places that state categorically that InProc

session

state cannot be used with Web Gardens but we are hoping that the KB

article

above knows something that they don't. Can anyone help us to tie

particular

sessions to particular worker processes?

Web Gardens are not compatible with inproc session state. There is no good
mechanism in place that can affiliate a session with a process.
Load balancing and a Web Farm is more intelligent its able to affiliate a
client with a server.
Other options would be:-

Divide the site into separate applications what each run in their own pool

Allow commonly request chunks of dynamically generated HTML that don't often
differ from one request to the next to be cached.

--
Anthony Jones - MVP ASP/ASP.NET

[Mark Rae [MVP]]

"Anthony Jones" <An*@yadayadaya da.comwrote in message
news:%2******** ********@TK2MSF TNGP06.phx.gbl. ..

Web Gardens are not compatible with inproc session state. There is no
good
mechanism in place that can affiliate a session with a process.

Correct.

Load balancing and a Web Farm is more intelligent its able to affiliate a
client with a server.

I would suggest that this is probably the best solution for the OP,
budgetary constraints notwithstanding ...
--
Mark Rae
ASP.NET MVP
http://www.markrae.net

[=?Utf-8?B?QnJlbmRhbiBLYXk=?=]

"Anthony Jones" wrote:

Define 'Huge'. Are you sure that it was a problem? How many CPU cores do
you have and can you add more? I'm not sure how using a Web Garden actually
resolves the problem. What is the CPU usage rate like?

Hi Anthony, thanks for your reply.

You are quite right that I should have provided more details - sorry about
the vagueness of the first post.

When running on a single worker process the Context Switches per second gets
up to 300K+. This is on a dual quad core machine so it a lot of context
switches per core.

When the application pool is configured to use 8 worker processes (one per
core) the context switches fall to less than 10K.

In both cases the CPU utilization stays around 30-50%. It seems to be pretty
clear that the context switches are the problem for us (unless I am missing
something else ;-).

I am not really sure why there would be such a big difference in context
switches because of the number of worker process (for exactly the same load)
but it is dramatic. The average response time per page goes from about 1.3s
to 12s.

Are your clients connecting via a proxy or firewall, (even if they are
internal clients I've often seen configurations which unintentionaly route
HTTP through a firewall/proxy to an internal server)?

No, the clients are connecting directly (we have done tests that go through
a firewall and the results are the same).

That said its not possible to guarantee a fixed connection. Additionally
browser will use more than one connection, the likely hood is then that the
browser will have one connection on one process and another to the other
process.

Web Gardens are not compatible with inproc session state. There is no good
mechanism in place that can affiliate a session with a process.

That is a shame (but not a huge surprise). We were hoping the knowledge base
article 822171 might be referring to some technique for keeping the
connection alive and having all the requests for a particular session routed
to the same worker process.

Load balancing and a Web Farm is more intelligent its able to affiliate a
client with a server.

We are looking into that now.

Other options would be:-

Divide the site into separate applications what each run in their own pool

This is another option we are looking at. We are seeing whether the
application can be run as several IIS applications and we may put our own
dispatcher in front of it.

Allow commonly request chunks of dynamically generated HTML that don't often
differ from one request to the next to be cached.

Unfortunately all our pages are generated dynamically so caching is not an
option but we do believe we will find an alternate solution.

Thanks again for your thoughts.

>
--
Anthony Jones - MVP ASP/ASP.NET

[=?Utf-8?B?QnJlbmRhbiBLYXk=?=]

Hi Mark,

Thanks for the reply. We will look into the web farm option as well as
options for inserting our own dispatcher.

Brendan

"Mark Rae [MVP]" wrote:

"Anthony Jones" <An*@yadayadaya da.comwrote in message
news:%2******** ********@TK2MSF TNGP06.phx.gbl. ..

Web Gardens are not compatible with inproc session state. There is no
good
mechanism in place that can affiliate a session with a process.

Correct.

Load balancing and a Web Farm is more intelligent its able to affiliate a
client with a server.

I would suggest that this is probably the best solution for the OP,
budgetary constraints notwithstanding ...
--
Mark Rae
ASP.NET MVP
http://www.markrae.net

[Anthony Jones]

"Brendan Kay" <Br********@dis cussions.micros oft.comwrote in message
news:17******** *************** ***********@mic rosoft.com...

"Anthony Jones" wrote:

Define 'Huge'. Are you sure that it was a problem? How many CPU cores

do

you have and can you add more? I'm not sure how using a Web Garden

actually

resolves the problem. What is the CPU usage rate like?

Hi Anthony, thanks for your reply.

You are quite right that I should have provided more details - sorry about
the vagueness of the first post.

When running on a single worker process the Context Switches per second

gets

up to 300K+. This is on a dual quad core machine so it a lot of context
switches per core.

When the application pool is configured to use 8 worker processes (one per
core) the context switches fall to less than 10K.

In both cases the CPU utilization stays around 30-50%. It seems to be

pretty

clear that the context switches are the problem for us (unless I am

missing

something else ;-).

I am not really sure why there would be such a big difference in context
switches because of the number of worker process (for exactly the same

load)

but it is dramatic. The average response time per page goes from about

1.3s

to 12s.

Did you mean from 12s prior to using a web garden to 1.3s using a web
garden?
Do the worker processes run in 64bit?
How much memory do you have on the box?

Are your clients connecting via a proxy or firewall, (even if they are
internal clients I've often seen configurations which unintentionaly

route

HTTP through a firewall/proxy to an internal server)?

No, the clients are connecting directly (we have done tests that go

through

a firewall and the results are the same).

That said its not possible to guarantee a fixed connection.

Additionally

browser will use more than one connection, the likely hood is then that

the

browser will have one connection on one process and another to the other
process.

Web Gardens are not compatible with inproc session state. There is no

good

mechanism in place that can affiliate a session with a process.

That is a shame (but not a huge surprise). We were hoping the knowledge

base

article 822171 might be referring to some technique for keeping the
connection alive and having all the requests for a particular session

routed

to the same worker process.

Load balancing and a Web Farm is more intelligent its able to affiliate

a

client with a server.

We are looking into that now.

Other options would be:-

Divide the site into separate applications what each run in their own

pool

>
This is another option we are looking at. We are seeing whether the
application can be run as several IIS applications and we may put our own
dispatcher in front of it.

I'm not sure what you mean by dispatcher or why you would need one?

The problem would be if there is some state in one sub application or
session thereof, that is needed in another sub application. Typically this
would be user authentication data. You would need to handle this yourself
and would probably need to form an umbrella session cookie that has a
path="/" (the sub application session cookies will be rooted in their
directories). It can get a bit hairy, and ultimately more costly that
installing a Web Farm. OTH, if most access is anonymous and you have
significant independant parts to your site then its fairly straightforward .

>

Allow commonly request chunks of dynamically generated HTML that don't

often

differ from one request to the next to be cached.

Unfortunately all our pages are generated dynamically so caching is not an
option but we do believe we will find an alternate solution.

They may be generated dynamically but is the underlying data so volatile
that the same URL is unlikely to generate the same markup even when hit
several times a second. Just because something is generated dynamically
does not mean it can't be cached. Even for data that some would consider
quite volatile a 2 minute cache period for a high frequency URL could have a
significant impact.

--
Anthony Jones - MVP ASP/ASP.NET

[=?Utf-8?B?QnJlbmRhbiBLYXk=?=]

Hi Anthony,

Thanks again for your continuing help.

Did you mean from 12s prior to using a web garden to 1.3s using a web
garden?
Do the worker processes run in 64bit?
How much memory do you have on the box?

Yep, avarage page response time is about 1.3s when configured as a web
garden and about 12s when configured as a single worker thread.

The server has 4Gb of memory and the actual memory used by each worker
process is only about 100Mb. The server is running 32bit (and we can't change
that right now).

I'm not sure what you mean by dispatcher or why you would need one?

The problem would be if there is some state in one sub application or
session thereof, that is needed in another sub application. Typically this
would be user authentication data. You would need to handle this yourself
and would probably need to form an umbrella session cookie that has a
path="/" (the sub application session cookies will be rooted in their
directories). It can get a bit hairy, and ultimately more costly that
installing a Web Farm. OTH, if most access is anonymous and you have
significant independant parts to your site then its fairly straightforward .

If we do go with the dispatcher it will be a front end application that
receives the request and creates an outer session for it. This session would
simply remember which URL it is passed on to and store the appropriate
session id/cookie to pass on. For any requests that come in from the same
outer session it would pass them on to the inner session with the appropriate
cookie (or URL).

In theory we believe it will work and we think the performance impact will
be much less than the context switches we are currently getting but we have
not actually coded it yet so there may be something that stops it being used.

They may be generated dynamically but is the underlying data so volatile
that the same URL is unlikely to generate the same markup even when hit
several times a second. Just because something is generated dynamically
does not mean it can't be cached. Even for data that some would consider
quite volatile a 2 minute cache period for a high frequency URL could have a
significant impact.

In our case the dynamically generated pages use data from another source
that cannot be cached and must always display the current version (this is
ultimately the data and state that actually stops us from serializing the
session state). We can't see a way to use caching but we hope the web farm or
dispatcher will give us what we need.

--
Anthony Jones - MVP ASP/ASP.NET

Thanks again for your help.

Brendan