Our system is:
IIS Server: dual Intel Xeon 2.80 GHz, 4 GB Ram
Windows Server 2003 SP2
IIS 6.0
SQL Server: dual Intel Xeon 2.80 GHz, 4 GB Ram (separate server)
Windows Server 2003 SP2
SQL Server 2000
We are having some problems with a website we are developing, and had some
questions.
Background:
The application consists of ASP.Net 2.0 pages calling VB.Net 2.0 dlls. The
application retrieves a large dataset from SQL Server and creates a large
object structure, which is then stored in the Application “state” area with a
line like
Current.Application.Add(Current.Session.SessionID + keyName, keyValue)
As the user navigates to the various pages of the application, the stored
object is retrieved from state as follows
strWhatIsCheckedOut = SessionCache.Project.ItemCheckedOutValue
The SessionCache.Project object is actually a function, that when unrolled
makes this line:
strWhatIsCheckedOut =
DirectCast(Current.Application.Get(Current.Session .SessionID +
"CurrentProject"), Project).ItemCheckedOutValue
There are times in the code when SessionCache.Project is called repeatedly
to fetch multiple items. Sometimes the object that is returned is walked in
order to retrieve the information needed to present to the user for this
particular page. After a couple of minutes when multiple people, and
possibly multiple browsers per person, are accessing the site one of the
pages will get an ‘Object reference not set to an instance of an object”. We
have tracked this down to the line of code where we get the object from the
application state. Once the error occurs, the other browsers for the same
user will also throw similar errors when the user takes some action on the
page. The other users also have similar errors at that point in time. Even
before the errors happen, there are sometimes substantial delays (~10
seconds) in bringing a page up with no database hits, just retrieving data
from the application state.
We added code to our Application_OnStart event to let us know when the
application recycles. Very seldom did it appear that the Application_OnStart
event fired when we were losing sessions. Thus, is it possible, that the site
spawns a new application pool when the existing one becomes heavily loaded
which might cause a inproc session problem without triggering an
Application_OnStart event.
We have been using various tools to troubleshoot these problems. We have
been watching CPU utilization with Performance Monitor, and Memory
utilization with Task Manager. We have also put in some debugging
information into our pages. Looking at the size of the garbage collector
memory {System.GC.GetTotalMemory(True) }, it appears that one of the larger
“Project” objects can take up around 70 MB of memory when it is created. As
we load one of these Project objects from the database and into the
application state, we observed in TaskMgr the memory footprint of the
w3wp.exe process increases by around 120MB (fairly evenly split between real
memory and virtual memory). As additional browsers were launched and loaded
this “project” object, the w3wp.exe process kept increasing by varying
amount, often around 100MB.
When the user initially loads a “project”, we see a short (1 sec) ~80% CPU
hit on the SQL box as the data is retrieved. On the IIS box, we see a longer
(4 – 20 sec) ~40% CPU hit when we think the object is being put into the
application state. As the user navigates around the site, and each page
fetches the object from the application state, we see similar 4-20 sec 40%
CPU hits on IIS.
We have 3 main concerns:
1. The errors which not only bring down the first user, but tends to bring
down all the users.
2. The performance of the site is poor. Pages take a substantial amount of
time to load. (CPU utilization is high for an extended period of time)
3. It appears that there is a large memory footprint, that greatly limits
the number of users that the system can handle.
Questions:
1. Are we guaranteed that every unique instance of an IE browser on a single
computer will have its own unique session and unique session id, even if the
browsers are all open concurrently?
2. How is a complex object, such as an arbitrary object structure with
several thousand child objects, stored in the application state?
3. What kind of overhead is associated with storing an object in application
state?
4. How are lookups by key for objects in the application state performed?
a. Do lookups happen in O(1), O(#keys), O(size of application state)?
5. What has to happen (deserialize, transfer data from one process to
another, etc.) to get an object out of application state? E.g. obj =
Application(key)
6. What kinds of things, besides Application(key) = SomethingElse, could
cause the object associated to a given key to “disappear” from the
application state?
7. Will the system need to refetch the object from state when there are
multiple calls to Current.Application.Get() occurring on the same key?
8. What is the w3wp.exe process?
9. Is its growth in memory footprint, both real and virtual, indicative of
the size of object stored in the application state?
10. What will happen as this footprint approaches and exceeds 2GB?
11. Should there be concern having the memory footprint of w3wp.exe get
large.
12. Is it to be expected to have multiple second elapsed time in putting
objects of this size into and getting it out of the application state?
