Random crashes in mscorwks.dll...

[Gary Joy]

I really am banging my head with this one...

I have a regular unmanaged C++ application that is using mixed DLLs to
(amongst other things) call a C# back-end (which is using ADO .NET). I am
getting random crashes, typically after 15-60 minutes of inactivity. The
error message and call stack are listed below.

Unhandled exception at 0x792483e0 (mscorwks.dll) in
debug_build_wks _crash.dmp: 0xC0000005: Access violation reading location
0xfeef00da.

mscorwks.dll!Co rExitProcess() + 0x5e2c
mscorwks.dll!ND _CopyObjDst() + 0x2fa9
mscorwks.dll!Co rExitProcess() + 0x4a3c
mscoree.dll!__C orDllMain@12() + 0x1c
ntdll.dll!_Ldrp CallInitRoutine @16() + 0x14
ntdll.dll!_LdrS hutdownThread@0 () + 0xdb
kernel32.dll!_E xitThread@4() + 0x3e
mscorwks.dll!Re leaseFusionInte rfaces() + 0x44a7f
mscorwks.dll!Re leaseFusionInte rfaces() + 0x441b0
kernel32.dll!_B aseThreadStart@ 8() + 0x37

In an attempt to fix or identify the problem I have:

1. Searched the code for places where unmanaged code could be corrupting
managed memory.
2. Tracked (using Trace.WriteLine ()) garbage collection. This seemed to
indicate that garbage collection (of my objects) had taken place long before
the crash.
3. Implemented the workaround for the mixed DLL loading problem
(http://support.microsoft.com/?id=814472).

I can't make much sense of the call stack, even this
(http://msdn.microsoft.com/library/de...xitprocess.asp) document doesn't really help me.

I would really appreciate either suggestions as to what might be causing the
problem or ideas that might help me track it down.

Thanks

[Oleg Starodumov]

It looks like there is no symbols for mscorwks.dll on your system
(exports are loaded instead), and therefore the call stack is incorrect.
You should obtain the good symbols (e.g. download them from
the symbol server) to get the correct call stack.

Regards,
Oleg

[Gary Joy]

Sorry... The call stack was created from a dump file which for some reason
didn't pick up the symbols when I opened it in VS. Here is the actual call
stack (I have typed it from a screen-shot and triple checked for typos):

mscorwks.dll!Th read::UnhijackT hread() + 0x70d9f
mscorwks.dll!Th read::RareEnabl ePreemptiveGC() + 0x36
mscorwks.dll!Th read::RareDisab lePreemptiveGC( ) + 0x49cb6
mscorwks.dll!Sy stemDomain::Run DllMain() + 0x7d
mscorwks.dll!Ex ecuteDLL() + 0x7fd12
mscoree.dll!__C orDllMain@12() + 0x1c
ntdll.dll!_Ldrp CallInitRoutine @16() + 0x14
ntdll.dll!_LdrS hutdownThread@0 () + 0xdb
kernel32.dll!_E xitThread@4() + 0x3e
mscorwks.dll!Th readpoolMgr::Wo rkerThreadStart () + 0x123
mscorwks.dll!Th readpoolMgr::in termediateThrea dProc() + 0x44
kernel32.dll!_B aseThreadStart@ 8() + 0x37

I have since tried using pageheap
(http://support.microsoft.com/default...b;en-us;286470 and
http://support.microsoft.com/default...b;en-us;264471) to detect
memory corruption in the application. Despite identifying and fixing a memory
problem the application still crashes in the same way.

I will be very gratefull for any help you can provide... :-)

[Vladimir Nesterovsky]

Hello,

I believe some problem araises during thread (from thread pool)
finalization. May be this is somehow related to an unmanaged code that was
executed in that thread.
Can you replace (for testing purposes) asynchronous calls to explicit
threads. The earlier you will finalize these threads, the better.
--
Vladimir Nesterovsky
e-mail: vl******@nester ovsky-bros.com
home: http://www.nesterovsky-bros.com

Sorry... The call stack was created from a dump file which for some reason
didn't pick up the symbols when I opened it in VS. Here is the actual call
stack (I have typed it from a screen-shot and triple checked for typos):

mscorwks.dll!Th read::UnhijackT hread() + 0x70d9f
mscorwks.dll!Th read::RareEnabl ePreemptiveGC() + 0x36
mscorwks.dll!Th read::RareDisab lePreemptiveGC( ) + 0x49cb6
mscorwks.dll!Sy stemDomain::Run DllMain() + 0x7d
mscorwks.dll!Ex ecuteDLL() + 0x7fd12
mscoree.dll!__C orDllMain@12() + 0x1c
ntdll.dll!_Ldrp CallInitRoutine @16() + 0x14
ntdll.dll!_LdrS hutdownThread@0 () + 0xdb
kernel32.dll!_E xitThread@4() + 0x3e
mscorwks.dll!Th readpoolMgr::Wo rkerThreadStart () + 0x123
mscorwks.dll!Th readpoolMgr::in termediateThrea dProc() + 0x44
kernel32.dll!_B aseThreadStart@ 8() + 0x37

I have since tried using pageheap
(http://support.microsoft.com/default...b;en-us;286470 and
http://support.microsoft.com/default...b;en-us;264471) to detect
memory corruption in the application. Despite identifying and fixing a memory problem the application still crashes in the same way.

I will be very gratefull for any help you can provide... :-)

[Oleg Starodumov]

In addition, try to run these commands in Command Window
after you have opened the dump:

..load sos
!DumpStack

What call stack will be shown?

Regards,
Oleg

[Gary Joy]

The ouput is:

..load sos

!DumpStack
succeeded
Loaded Son of Strike data table version 5 from
"C:\WINDOWS\Mic rosoft.NET\Fram ework\v1.1.4322 \mscorwks.dll"
Current frame: ?ReturnToContex t@Thread@@QAEXP AVFrame@@H@Z+0x 2d
ChildEBP RetAddr Caller,Callee
0428f6d8 792a5ea4 ?RunDllMain@Sys temDomain@@SGJP AUHINSTANCE__@@ KPAX@Z+0x288,
calling ?ReturnToContex t@Thread@@QAEXP AVFrame@@H@Z
0428f78c 79246ff0 ?ExecuteDLL@@YG HPAUHINSTANCE__ @@KPAX@Z+0x3c0, calling
?RunDllMain@Sys temDomain@@SGJP AUHINSTANCE__@@ KPAX@Z
0428f7b0 792438ac _DllMain@12+0x3 3, calling
?EEDllMain@@YGH PAUHINSTANCE__@ @KPAX@Z
0428f7c4 791c049c _DllMain@12+0x1 17, calling @__security_che ck_cookie@4
0428f7c8 79b91ba8 (stub for mdToken: 0600002a (), calling 79b7a188
0428f7cc 79a2bbe7 79a2bbe7, calling 79b91ba3 (MethodDesc 0x79b91ba8
mdToken: 0600002a ()
0428f7dc 791b1b3a ?JIT_MonExit@@Y IXPAVObject@@@Z +0xf, calling
?DummyGetThread @@YGPAVThread@@ XZ
0428f81c 791ed194
?CallDescr@Meth odDesc@@QAE_JPB EPAVModule@@PAV MetaSig@@HPB_J@ Z+0x1b8, calling
_CallDescrWorke r@16
0428f850 791daddf ??0ArgIterator@ @QAE@PAEPAVMeta Sig@@H@Z+0x55, calling
?IsArgumentInRe gister@@YGHPAHE IHE0@Z
0428f864 791d94bc _CallDescrWorke r@16+0x30
0428f86c 791ed194
?CallDescr@Meth odDesc@@QAE_JPB EPAVModule@@PAV MetaSig@@HPB_J@ Z+0x1b8, calling
_CallDescrWorke r@16
0428f874 791ed1cd
?CallDescr@Meth odDesc@@QAE_JPB EPAVModule@@PAV MetaSig@@HPB_J@ Z+0x1f1, calling
?getFPReturn@@Y GXHAA_J@Z
0428f890 77f944cb _RtlpFreeToHeap Lookaside@8+0x1 f, calling
@RtlpInterlocke dPushEntrySList @8
0428f898 77f58bcd _RtlFreeHeap@12 +0x18f, calling _RtlpFreeToHeap Lookaside@8
0428f8a0 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428f8c0 77f944cb _RtlpFreeToHeap Lookaside@8+0x1 f, calling
@RtlpInterlocke dPushEntrySList @8
0428f8c8 77f58bcd _RtlFreeHeap@12 +0x18f, calling _RtlpFreeToHeap Lookaside@8
0428f8d0 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428f8e0 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428f8ec 79513737 _DllMain@12+0x1 19, calling @__security_che ck_cookie@4
0428f910 77f944cb _RtlpFreeToHeap Lookaside@8+0x1 f, calling
@RtlpInterlocke dPushEntrySList @8
0428f918 77f58bcd _RtlFreeHeap@12 +0x18f, calling _RtlpFreeToHeap Lookaside@8
0428f920 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428f938 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428f960 77f59037 _RtlFreeHeap@12 +0x5f9, calling __SEH_epilog
0428fa14 7917cff3 __CorDllMain@12 +0x40
0428fa2c 77f5b42c _LdrpCallInitRo utine@16+0x14
0428fa4c 77f629df _LdrShutdownThr ead@0+0xdb, calling _LdrpCallInitRo utine@16
0428fa74 7929b73f
?SetDelayedInhe ritedSecuritySt ack@Thread@@QAE XPAVCompressedS tack@@@Z+0x1f,
calling ?Release@Compre ssedStack@@QAEJ XZ
0428fac8 77e74af8 _ExitThread@4+0 x3e, calling _LdrShutdownThr ead@0
0428fb20 792ec273 ?intermediateTh readProc@Thread poolMgr@@CGKPAX @Z+0x44
0428fb2c 771256e2 _DllMain@12+0x2 7, calling
?_typesDllMain@ @YGHPAUHINSTANC E__@@KPAX@Z
0428fb74 77f58a3a _RtlAllocateHea p@12+0xe8c, calling __SEH_epilog
0428fb88 762fa952 ?I_CryptTlsDllM ain@@YGHPAUHINS TANCE__@@KPAX@Z +0x19f,
calling __SEH_epilog
0428fbac 76317375 ?CryptPFXDllMai n@@YGHPAUHINSTA NCE__@@KPAX@Z+0 x24, calling
?EncodeDecodeDl lMain@@YGHPAUHI NSTANCE__@@KPAX @Z
0428fbd8 77f58a3a _RtlAllocateHea p@12+0xe8c, calling __SEH_epilog
0428fbdc 76f611cd _ldapMalloc@8+0 x17, calling _LdrpDllTagProc edures+0x1
0428fbf0 76f61234 _AddPerThreadEn try@4+0x4d, calling
_RtlComputeImpo rtTableHash@12+ 0x18
0428fc00 76f610e6 _LdapDllInit@12 +0x67, calling _AddPerThreadEn try@4
0428fc08 77f5b42c _LdrpCallInitRo utine@16+0x14
0428fc30 77f62b67 _LdrpInitialize Thread@4+0xe3, calling
_RtlDeactivateA ctivationContex tUnsafeFast@4
0428fc38 77f62bc6 _LdrpInitialize Thread@4+0x142, calling __SEH_epilog
0428fc9c 77f55432 _LdrpInitialize @12+0x26f, calling _RtlLeaveCritic alSection@4
0428fca0 77f5c474 _ZwTestAlert@0+ 0xc
0428fca4 77f5541d _LdrpInitialize @12+0x25a, calling __SEH_epilog
0428fd18 77f5541d _LdrpInitialize @12+0x25a, calling __SEH_epilog
0428fd1c 77f5b644 _ZwContinue@8+0 xc
0428fd20 77f75d8f _KiUserApcDispa tcher@20+0xf, calling _ZwContinue@8
0428fdd4 77f944a8 _RtlpAllocateFr omHeapLookaside @4+0x42, calling __SEH_epilog
0428fdd8 77f57d70 _RtlAllocateHea p@12+0x1c2, calling
_RtlpAllocateFr omHeapLookaside @4
0428ffa4 792ec261 ?intermediateTh readProc@Thread poolMgr@@CGKPAX @Z+0x32,
calling __alloca_probe
0428ffb4 77e7d28e _BaseThreadStar t@8+0x37

I have managed to "solve" the problem by doing a complete "rebuild" i.e.
clearing my development folder and getting files from scratch from source
control. I believe that the problem may be related to the fact that I had 23
different versions of a managed DLL in various locations with five different
timestamps. VS copies managed DLLs when it is building and I don't think the
copies were being correctly updated.

I have accepted your answer because you have provided me with an additional
mechanism to try and identify the problem. I can't make much sense of the
managed call stack but if there are any hints in there please let me know.

Thanks for taking the time to post... :-)

[Gary Joy]

Update - we are still working to resolve this. The issue seems to be related
to (i) the use of connection pooling and (ii) running the application on dual
processor machines.

[Shy]

Gary Joy wrote:

*Update - we are still working to resolve this. The issue seems to be
related
to (i) the use of connection pooling and (ii) running the application
on dual
processor machines. *

I'm having a similar problem. In my case, whenever it happens I stopped
the application and tried to find the renegade thread in the visual
studio.
It appears that it's not even one of mine...
Does anyone know what can be done?

--
Shy
------------------------------------------------------------------------
Posted via http://www.codecomments.com
------------------------------------------------------------------------

[Gary Joy]

My apologies for not updating this when we finally found a "solution". With
the help of MCS we solved the problem by disabling thread attach and detach
notification for the troublesome DLLs.

This was some time ago and I have since changed jobs so cannot provide much
more information than that. I do remember that Microsoft were very helpful in
getting to the bottom of the problem.

P.S. I hope you don't *need* thread attach and detach notifications.. . ;-)