Bandwidth constraint causing IIS/ASP.NET problem

[Jack Fox]

While stress-testing our ASP.NET app, we discovered a flaw in our test
set-up, which also reveals a flaw in our ASP.NET configuration that we do
not know how to address.

We simulated too many clients, making requests faster than the bandwidth on
that part of the network could consume the responses. It seems that the
inability of the requestors to receive their responses causes IIS, and/or
aspnet_wp.exe, to eventually max-out its threads and come to its knees.

We changed our unrealistic virtual client distribution and distributed our
load generation to other parts of the network. Our app then did fine.

But.it seems that configuring more response load on the network than the
bandwidth can handle is a type of denial of service attack against
IIS/ASP.NET. How can we configure IIS and/or ASP.NET to prevent this?

[JIMCO Software]

Jack Fox wrote:

We simulated too many clients, making requests faster than the
bandwidth on that part of the network could consume the responses. It
seems that the inability of the requestors to receive their responses
causes IIS, and/or aspnet_wp.exe, to eventually max-out its threads
and come to its knees.
But.it seems that configuring more response load on the network than
the bandwidth can handle is a type of denial of service attack against
IIS/ASP.NET. How can we configure IIS and/or ASP.NET to prevent this?

It's hard to believe that you could have exceeded bandwidth limitations in
this scenario. It seems more likely that you stressed the box. However,
without some perfmon data, it's hard to say where you hosed yourself.

There are so many places where a performance problem can happen. If you're
in GC too much, you will see this kind of behavior. When we're in GC (or
when CPU reaches a certain threshold), new worker threads are not spun up to
handle requests. It's also possible that you might not have enough
completion port threads available. You also might not have enough worker
threads or too many worker threads.

Without details, it's impossible to say. However, there is a test you can
run. When you're bogged down, do HTML/ASP pages serve okay? If so, it's
likely something from paragraph 2. If not, something unrelated to ASP.NET
is going on.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003

[Jack Fox]

I should have mentioned the reason I know we exceeded the bandwidth is
because we were simulating 1,200 clients with 3 - 6 seconds between each
client request, requesting pages that are up to 100kb. If you do some back
of the envelope arithmetic you'll probably agree that all of that request
and response traffic to one machine connected through one 100mbs connection
will exceed the bandwidth.

When we distributed the client load over several machines so that we didn't
have the 100mbs bandwidth constraint, everything worked fine.

The point I'm trying to make is that by making requests that will generate
too much response traffic for the client network node, you are in effect
performing a denial of service attack on the IIS/ASP.NET server. We've
boosted our thread count before, and your suggestion of looking at the
completion port threads is probably worthwhile, but it seems that a
malicious client could still overwhelm the server because there is alway a
thread limit. Is there any way to short-circuit this kind of attack?

"JIMCO Software" <co*******@jimc osoftware.com> wrote in message
news:%2******** *******@TK2MSFT NGP15.phx.gbl.. .

Jack Fox wrote:

We simulated too many clients, making requests faster than the
bandwidth on that part of the network could consume the responses. It
seems that the inability of the requestors to receive their responses
causes IIS, and/or aspnet_wp.exe, to eventually max-out its threads
and come to its knees.
But.it seems that configuring more response load on the network than
the bandwidth can handle is a type of denial of service attack against
IIS/ASP.NET. How can we configure IIS and/or ASP.NET to prevent this?

It's hard to believe that you could have exceeded bandwidth limitations in
this scenario. It seems more likely that you stressed the box. However,
without some perfmon data, it's hard to say where you hosed yourself.

There are so many places where a performance problem can happen. If
you're in GC too much, you will see this kind of behavior. When we're in
GC (or when CPU reaches a certain threshold), new worker threads are not
spun up to handle requests. It's also possible that you might not have
enough completion port threads available. You also might not have enough
worker threads or too many worker threads.

Without details, it's impossible to say. However, there is a test you can
run. When you're bogged down, do HTML/ASP pages serve okay? If so, it's
likely something from paragraph 2. If not, something unrelated to ASP.NET
is going on.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003

[JIMCO Software]

Jack Fox wrote:

I should have mentioned the reason I know we exceeded the bandwidth is
because we were simulating 1,200 clients with 3 - 6 seconds between
each client request, requesting pages that are up to 100kb. If you do
some back of the envelope arithmetic you'll probably agree that all
of that request and response traffic to one machine connected through
one 100mbs connection will exceed the bandwidth.

When we distributed the client load over several machines so that we
didn't have the 100mbs bandwidth constraint, everything worked fine.

Jack,

What you are describing here would not cause a DoS.

As for increasing worker threads, I didn't suggest that you do that. You
would only do that if you have available CPU to process them and currently
have no requests queued. What I was suggesting is that you need to use
performance data (via Perfmon logs) to determine where your perf problem is.
It doesn't seem to me that you've done that.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003