OutOfMemoryException

[Adam Weigert]

Occassionally, our ASP.NET applications (about 24 of them) running on Windows
2000 Advanced Server with ASP.NET 1.1.4322, start generating
OutOfMemoryException exceptions.

The server has 3GB of ram and this usually occurs between 600MB - 800MB of
used memory in the ASPNET_WP.EXE process.

I have a full dump but have not been able to find out how to figure out
which AppDomain is consuming the resources (or the most resources).

Are there any commands I can use to find this out?

[JIMCO Software]

Adam Weigert wrote:

Occassionally, our ASP.NET applications (about 24 of them) running on
Windows 2000 Advanced Server with ASP.NET 1.1.4322, start generating
OutOfMemoryException exceptions.

The server has 3GB of ram and this usually occurs between 600MB -
800MB of used memory in the ASPNET_WP.EXE process.

I have a full dump but have not been able to find out how to figure
out which AppDomain is consuming the resources (or the most
resources).

Are there any commands I can use to find this out?

You'll first want to load SOS:

..load clr10/sos

Then do this:

!eeheap -gc

Look at the total heap size. If it's the majority of your memory, you're
looking at a managed memory problem. If the managed heap is small, it's
going to be a native leak.

Assuming managed:

Do this:

!dumpheap -stat

That will give you a list of all objects on the heap sorted by size. If you
can post the bottom portion of that, I'll take you from there.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003

[Adam Weigert]

Thanks, I already did this from there on I get lost lol ... total size of the
dump is 697MB ...

Number of GC Heaps: 2
------------------------------
Heap 0 (0x00150460)
generation 0 starts at 0x34eaf59c
generation 1 starts at 0x34eaf590
generation 2 starts at 0x00f40030
ephemeral segment allocation context: none
segment begin allocated size
0x00f40000 0x00f40030 0x0402ce54 0x030ece24(51,301,924)
0x34950000 0x34950030 0x34eb15a8 0x00561578(5,641,592)
Large object heap starts at 0x08f40030
segment begin allocated size
0x08f40000 0x08f40030 0x09de58e0 0x00ea58b0(15,358,128)
0x1a5c0000 0x1a5c0030 0x1b3623c8 0x00da2398(14,295,960)
Heap Size 0x5295fe4(86,597,604)
------------------------------
Heap 1 (0x00150b68)
generation 0 starts at 0x5c24ffe8
generation 1 starts at 0x5c24ffdc
generation 2 starts at 0x04f40030
ephemeral segment allocation context: none
segment begin allocated size
0x04f40000 0x04f40030 0x08e7d400 0x03f3d3d0(66,311,120)
0x30950000 0x30950030 0x3494ffc4 0x03ffff94(67,108,756)
0x58250000 0x58250030 0x5c24fff4 0x03ffffc4(67,108,804)
Large object heap starts at 0x21ce0030
segment begin allocated size
0x21ce0000 0x21ce0030 0x221ef3b8 0x0050f388(5,305,224)
0x3e7d0000 0x3e7d0030 0x3e9e22d8 0x002122a8(2,171,560)
Heap Size 0xc65e958(208,005,464)
------------------------------
GC Heap Size 0x118f493c(294,603,068)
0x79ba0d74 29,645 711,480 System.Collections.ArrayList
0x0bf54384 46,512 744,192 System.Web.UI.AttributeCollection
0x0bf58d8c 47,691 763,056
System.Web.Configuration.CapabilitiesPattern
0x79ba968c 14,920 775,840 System.Collections.Hashtable
0x0c01dd9c 12,662 810,368 System.Data.DataColumnPropertyDescriptor
0x0bf58e44 41,230 824,600
System.Web.Configuration.CapabilitiesAssignment
0x0bf5ac50 47,293 1,135,032 System.Web.UI.WebControls.Unit
0x0c015d48 12,663 1,620,864 System.Data.DataColumn
0x00e36d9c 939 1,921,040 System.DateTime[]
0x0bf5388c 69,631 1,949,668 System.Web.UI.ControlCollection
0x0bf562a4 49,684 1,987,360 System.Web.UI.WebControls.Style
0x0e3863fc 22,978 2,205,888 System.Web.UI.WebControls.HyperLink
0x0c01bc7c 66,122 2,644,880 System.Data.DataRow
0x0d165dd0 22,967 2,664,172 System.Web.UI.WebControls.DataGridItem
0x00e3236c 1,332 3,488,596 System.Char[]
0x00e35fd4 1,578 3,781,116 System.Decimal[]
0x00e334a8 15,387 4,390,968 System.Collections.Hashtable/bucket[]
0x0bf540c4 46,377 4,452,192 System.Web.UI.WebControls.Label
0x0bc74374 390,931 4,691,172 System.ComponentModel.EventHandlerList
0x00e331e8 69,247 5,204,408 System.Int32[]
0x0bc76c2c 391,779 7,835,580
System.ComponentModel.EventHandlerList/ListEntry
0x0bf54868 509,860 8,157,760 System.Web.UI.StateBag
0x0bf54bcc 611,517 9,784,272 System.Web.UI.StateItem
0x0ba9b868 510,255 10,205,100
System.Collections.Specialized.HybridDictionary
0x0bf52c3c 138,759 10,545,684 System.Web.UI.LiteralControl
0x79bc3564 393,788 11,026,064 System.EventHandler
0x0ba9c03c 612,080 12,241,600
System.Collections.Specialized.ListDictionary/DictionaryNode
0x0ba9bea8 510,125 12,243,000
System.Collections.Specialized.ListDictionary
0x00e3209c 198,014 15,620,264 System.Object[]
0x00e3292c 2,675 20,743,940 System.Byte[]
0x0014fca8 198 21,305,128 Free
0x0d166454 390,378 37,476,288 System.Web.UI.WebControls.TableCell
0x79b925c8 1,040,230 57,232,640 System.String
Total 6,750,901 objects, Total size: 294,595,116

[Bruce Barker]

the easiest approach is to load the dump file into a sql database and run
queries on it. looking at the class summaries does no good, as you have to
look at each invidual allocation, determine its size and appdomain (look at
the root allocation), then sum by appdomain.

if you switched to iis6.0 (win2003), you can run each app in its own
process, and then its easy.

-- bruce (sqlwork.com)

"Adam Weigert" <adam dot weigert at wegmans dot com> wrote in message
news:71**********************************@microsof t.com...

Thanks, I already did this from there on I get lost lol ... total size of
the
dump is 697MB ...

Number of GC Heaps: 2
------------------------------
Heap 0 (0x00150460)
generation 0 starts at 0x34eaf59c
generation 1 starts at 0x34eaf590
generation 2 starts at 0x00f40030
ephemeral segment allocation context: none
segment begin allocated size
0x00f40000 0x00f40030 0x0402ce54 0x030ece24(51,301,924)
0x34950000 0x34950030 0x34eb15a8 0x00561578(5,641,592)
Large object heap starts at 0x08f40030
segment begin allocated size
0x08f40000 0x08f40030 0x09de58e0 0x00ea58b0(15,358,128)
0x1a5c0000 0x1a5c0030 0x1b3623c8 0x00da2398(14,295,960)
Heap Size 0x5295fe4(86,597,604)
------------------------------
Heap 1 (0x00150b68)
generation 0 starts at 0x5c24ffe8
generation 1 starts at 0x5c24ffdc
generation 2 starts at 0x04f40030
ephemeral segment allocation context: none
segment begin allocated size
0x04f40000 0x04f40030 0x08e7d400 0x03f3d3d0(66,311,120)
0x30950000 0x30950030 0x3494ffc4 0x03ffff94(67,108,756)
0x58250000 0x58250030 0x5c24fff4 0x03ffffc4(67,108,804)
Large object heap starts at 0x21ce0030
segment begin allocated size
0x21ce0000 0x21ce0030 0x221ef3b8 0x0050f388(5,305,224)
0x3e7d0000 0x3e7d0030 0x3e9e22d8 0x002122a8(2,171,560)
Heap Size 0xc65e958(208,005,464)
------------------------------
GC Heap Size 0x118f493c(294,603,068)
0x79ba0d74 29,645 711,480 System.Collections.ArrayList
0x0bf54384 46,512 744,192 System.Web.UI.AttributeCollection
0x0bf58d8c 47,691 763,056
System.Web.Configuration.CapabilitiesPattern
0x79ba968c 14,920 775,840 System.Collections.Hashtable
0x0c01dd9c 12,662 810,368
System.Data.DataColumnPropertyDescriptor
0x0bf58e44 41,230 824,600
System.Web.Configuration.CapabilitiesAssignment
0x0bf5ac50 47,293 1,135,032 System.Web.UI.WebControls.Unit
0x0c015d48 12,663 1,620,864 System.Data.DataColumn
0x00e36d9c 939 1,921,040 System.DateTime[]
0x0bf5388c 69,631 1,949,668 System.Web.UI.ControlCollection
0x0bf562a4 49,684 1,987,360 System.Web.UI.WebControls.Style
0x0e3863fc 22,978 2,205,888 System.Web.UI.WebControls.HyperLink
0x0c01bc7c 66,122 2,644,880 System.Data.DataRow
0x0d165dd0 22,967 2,664,172 System.Web.UI.WebControls.DataGridItem
0x00e3236c 1,332 3,488,596 System.Char[]
0x00e35fd4 1,578 3,781,116 System.Decimal[]
0x00e334a8 15,387 4,390,968 System.Collections.Hashtable/bucket[]
0x0bf540c4 46,377 4,452,192 System.Web.UI.WebControls.Label
0x0bc74374 390,931 4,691,172 System.ComponentModel.EventHandlerList
0x00e331e8 69,247 5,204,408 System.Int32[]
0x0bc76c2c 391,779 7,835,580
System.ComponentModel.EventHandlerList/ListEntry
0x0bf54868 509,860 8,157,760 System.Web.UI.StateBag
0x0bf54bcc 611,517 9,784,272 System.Web.UI.StateItem
0x0ba9b868 510,255 10,205,100
System.Collections.Specialized.HybridDictionary
0x0bf52c3c 138,759 10,545,684 System.Web.UI.LiteralControl
0x79bc3564 393,788 11,026,064 System.EventHandler
0x0ba9c03c 612,080 12,241,600
System.Collections.Specialized.ListDictionary/DictionaryNode
0x0ba9bea8 510,125 12,243,000
System.Collections.Specialized.ListDictionary
0x00e3209c 198,014 15,620,264 System.Object[]
0x00e3292c 2,675 20,743,940 System.Byte[]
0x0014fca8 198 21,305,128 Free
0x0d166454 390,378 37,476,288 System.Web.UI.WebControls.TableCell
0x79b925c8 1,040,230 57,232,640 System.String
Total 6,750,901 objects, Total size: 294,595,116

[Adam Weigert]

How would I put it into a SQL database?

We won't be moving to IIS 6.0 until after the new year and this is a problem
that keeps coming up at least once if not twice a week.

[JIMCO Software]

Adam Weigert wrote:

Thanks, I already did this from there on I get lost lol ... total
size of the dump is 697MB ...

------------------------------
GC Heap Size 0x118f493c(294,603,068)

You are experiencing a native leak. The managed heap is only ~290MB which
is not large. Now it gets a little harder because you probably don't the
extensions that would really aid you here. You can use !heap to dig into
the heaps and see where allocations are, but unless you are tracking
allocations when the process is running, you probably won't be able to find
out who made the allocations.

I have an extension that will let me list out all of the heaps and the
allocations in each by size or by count. After you get that, you can do a
symbol search on *!*g_Heap and look for the heap handle for the heap that's
leaking to see who owns it. Even then, you won't have a stack unless
something was tracking allocations.

Your best bet would be to open a case with Microsoft.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003

[JIMCO Software]

Bruce Barker wrote:

the easiest approach is to load the dump file into a sql database and
run queries on it. looking at the class summaries does no good, as
you have to look at each invidual allocation, determine its size and
appdomain (look at the root allocation), then sum by appdomain.

if you switched to iis6.0 (win2003), you can run each app in its own
process, and then its easy.

-- bruce (sqlwork.com)

A dump is a binary file. You can't load it into a database.

If this were a managed leak, looking at the heap using !dumpheap -stat is
exactly what you would want to do. In a managed leak, tracking allocations
is worthless because allocations are made to grow GC heaps in known segment
sizes.

--
Jim Cheshire
JIMCO Software
http://www.jimcosoftware.com

FrontPage add-ins for FrontPage 2000 - 2003