Slow rendering and CPU bound application

[Jeff Morgan]

I have an application that runs on ASP.NET. We have been working through
performance testing and it appears that at about 40-50 users, I start to get
CPU bound.

The application is a three form, ASP.NET 2.0 application. A login page, and
a main page, that is a frame with a header page, and the main page. On the
main page, there are two datagrids that pull data from a database. The
problem is that on one of the grids, the amount of rows may be high (I don't
think so, but 100-250). One the two grids, I bind a checkbox so that the
user can check each row, and then it is processed once the page is posted
back.

The problem that I have when there are 35 + users is that it apppears that
the rendering of the html for the web page is killing the processor. We have
moved around to different boxes in the test lab and find that using a bigger
processor helps, but in the end, it is still not quite what I would like to
see. To me, and the tests that we have done, it appears that the html
rendering of the datagrid is slowing down the processor. As the rows
increase, so does the render time. If the users are low, the processor seems
to keep up and kick the processes out with trouble. When it is a single
user, the page will return within a second max, no matter what is selected on
the page. However when we get out to the 40 or so, the page times can go
well over 100 seconds, which seems to not make a whole lot of sense, seeing
as each user should be in about a second each, so 40-60 would be more of what
I would expect. I would like to see that it is much less (less than 8-10),
but realize that might not be possible.

We have traced the applicaition and ran a bunch of perf mon stuff on it,
network traces, etc. Hard part is that the worker process is using most of
the cpu, so it is hard to determine what the real issue is. I have profiled
the code and am not sure what to do differently. Sure that there are things
that can be done, but not sure.

My initial problem is trying to figure out what the bottleneck is. How can I
tell if the page rendering is the issue (how asp.net is pumping out the html)
or if it is something that I can look to improve in the code. Utimately I am
certain it will be coding, right, but how to ID what is a limit of code
versus ASP.NET. In other words, how can I tell if no matter what I do, it
won't make the big return.

The load on the db server is nothing. Hardly moves over 4%. Memory usage on
the server is less than 150 MB and I have cached about all that I can. The
whole thing is less than 1500 lines, and have been through it a bunch of
times.

Is it realistic to think that ASP.NET should be able to support more than 75
users for a two page application with checkboxes? Help me, as I would think
it should without too much extra effort, but maybe I am off base?

Thanks
Jeff