Debbugging help! (.NET 1.1 Framework Garbage Collection Problems)

[Cheung, Jeffrey Jing-Yen]

Hi all,

I need some help with debugging a "deadlock" in a production
environment. I've managed to get a crash dump and have tried to
analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
in debugging to pinpoint the exact problem; however, I have narrowed it
down to the garbage collection process. All threads seem to be waiting
for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
up 100% cpu.

Can someone suggest what methods I could use to fully show and pinpoint
where the GC is locking up? I have briefly skimmed the SOS docs and a
lot of the exposed functions seem useful; however, I can't really make
heads or tails out of them. I've examined the heap via !dumpheap -stat,
and the largest object that comes back is a whopping 22 MB System.String
object. In the MSDN docs, what is supposed to follow is a listing of
Large Objects; however, I figure that maybe because I am using the 1.1
framework, large objects aren't really differentiated any differently.
I managed to get the address by issuing a dumpheap -mt <mt of the
System.String> and I got a huge listing of addresses that really don't
make much sense to me. I certainly don't want to issue an !objsize
<addr> on each of these (there are over 30k addresses).

I think my biggest concern is that I want to be able to load the symbols
of the .NET assemblies that are being used so that I can analyze the
managed stack thoroughly, yet I'm not too certain how to do this. If
that's not necessary to find out the core problem, then I guess I need
to be directed towards the right place.

Thanks in advance everyone,

Jeff Cheung

[Pavel Lebedinsky]

Are you sure it's a single 22 MB string, and not 22 MB allocated
for all strings?

You can use !dumpheap (without parameters) to dump all objects.
If you see any huge strings in the list you can then do !dumpobj on
them to find out the contents.

You could also try examining the way your app allocates objects
using something like the CLR Allocation Profiler:

[url]http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/highperfmanagedapps.asp[/url]

Basically, it looks like you need to either reduce the load on GC
by optimizing your code or use a bigger machine. I noticed from your
post to the windbg newsgroup that your process is using the
workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
Does the app run on a single proc machine? As far as I know
ASP.NET should load mscorsrv.dll on MP systems.


"Cheung, Jeffrey Jing-Yen" wrote:

> Hi all,
>
> I need some help with debugging a "deadlock" in a production
> environment. I've managed to get a crash dump and have tried to
> analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
> in debugging to pinpoint the exact problem; however, I have narrowed it
> down to the garbage collection process. All threads seem to be waiting
> for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
> up 100% cpu.
>
> Can someone suggest what methods I could use to fully show and pinpoint
> where the GC is locking up? I have briefly skimmed the SOS docs and a
> lot of the exposed functions seem useful; however, I can't really make
> heads or tails out of them. I've examined the heap via !dumpheap -stat,
> and the largest object that comes back is a whopping 22 MB System.String
> object. In the MSDN docs, what is supposed to follow is a listing of
> Large Objects; however, I figure that maybe because I am using the 1.1
> framework, large objects aren't really differentiated any differently.
> I managed to get the address by issuing a dumpheap -mt <mt of the
> System.String> and I got a huge listing of addresses that really don't
> make much sense to me. I certainly don't want to issue an !objsize
> <addr> on each of these (there are over 30k addresses).
>
> I think my biggest concern is that I want to be able to load the symbols
> of the .NET assemblies that are being used so that I can analyze the
> managed stack thoroughly, yet I'm not too certain how to do this. If
> that's not necessary to find out the core problem, then I guess I need
> to be directed towards the right place.
>
> Thanks in advance everyone,
>
> Jeff Cheung

[Cheung, Jeffrey Jing-Yen]

Pavel,

Thanks again for your replies; you've been a great help. While I am
waiting for !dumpheap to finish (it is taking awhile), I have a few
questions.

Will I have to build the CLR Allocation Profiler with the 1.1 framework
in order for it to profile my application? I guess this is kind of a
silly question. I assume that the profiler would just attach on to the
ASPNET worker process, so the framework would be independent, but I
thought I'd check first since I'm known to be wrong on many occasions.
Through a Visual Studio.NET command prompt, I ran nmake and built the
CLRProfiler objects, ran it, and tried to Profile the ASP.NET
application. It kept sitting there waiting for a response even though I
kept hitting the page. Quite interesting...

I'll post my findings once I get dumpheap's results.

Thanks again,

Jeff Cheung

Pavel Lebedinsky wrote:

> Are you sure it's a single 22 MB string, and not 22 MB allocated
> for all strings?
>
> You can use !dumpheap (without parameters) to dump all objects.
> If you see any huge strings in the list you can then do !dumpobj on
> them to find out the contents.
>
> You could also try examining the way your app allocates objects
> using something like the CLR Allocation Profiler:
>
> [url]http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/highperfmanagedapps.asp[/url]
>
> Basically, it looks like you need to either reduce the load on GC
> by optimizing your code or use a bigger machine. I noticed from your
> post to the windbg newsgroup that your process is using the
> workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
> Does the app run on a single proc machine? As far as I know
> ASP.NET should load mscorsrv.dll on MP systems.
>
>
> "Cheung, Jeffrey Jing-Yen" wrote:
>
>

>>Hi all,
>>
>>I need some help with debugging a "deadlock" in a production
>>environment. I've managed to get a crash dump and have tried to
>>analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
>>in debugging to pinpoint the exact problem; however, I have narrowed it
>>down to the garbage collection process. All threads seem to be waiting
>>for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
>>up 100% cpu.
>>
>>Can someone suggest what methods I could use to fully show and pinpoint
>>where the GC is locking up? I have briefly skimmed the SOS docs and a
>>lot of the exposed functions seem useful; however, I can't really make
>>heads or tails out of them. I've examined the heap via !dumpheap -stat,
>>and the largest object that comes back is a whopping 22 MB System.String
>>object. In the MSDN docs, what is supposed to follow is a listing of
>>Large Objects; however, I figure that maybe because I am using the 1.1
>>framework, large objects aren't really differentiated any differently.
>>I managed to get the address by issuing a dumpheap -mt <mt of the
>>System.String> and I got a huge listing of addresses that really don't
>>make much sense to me. I certainly don't want to issue an !objsize
>><addr> on each of these (there are over 30k addresses).
>>
>>I think my biggest concern is that I want to be able to load the symbols
>>of the .NET assemblies that are being used so that I can analyze the
>>managed stack thoroughly, yet I'm not too certain how to do this. If
>>that's not necessary to find out the core problem, then I guess I need
>>to be directed towards the right place.
>>
>>Thanks in advance everyone,
>>
>>Jeff Cheung

>
>
>

[Cheung, Jeffrey Jing-Yen]

As of right now, !dumpheap is still going. I had to restart windbg
because it locked up on me yesterday, so I restarted the debug session
and it's still going. I understand that it is dumping all the objects
in the heap, and this is a 222 mb crash dump, but it has been running
for quite some time... I wonder if maybe this is the problem. Does too
many objects in the dump heap equate to too many objects for the GC to
collect?

Regards,

Jeff Cheung

Pavel Lebedinsky wrote:

> Are you sure it's a single 22 MB string, and not 22 MB allocated
> for all strings?
>
> You can use !dumpheap (without parameters) to dump all objects.
> If you see any huge strings in the list you can then do !dumpobj on
> them to find out the contents.
>
> You could also try examining the way your app allocates objects
> using something like the CLR Allocation Profiler:
>
> [url]http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndotnet/html/highperfmanagedapps.asp[/url]
>
> Basically, it looks like you need to either reduce the load on GC
> by optimizing your code or use a bigger machine. I noticed from your
> post to the windbg newsgroup that your process is using the
> workstation version of GC (mscorwks.dll) instead of mscorsvr.dll.
> Does the app run on a single proc machine? As far as I know
> ASP.NET should load mscorsrv.dll on MP systems.
>
>
> "Cheung, Jeffrey Jing-Yen" wrote:
>
>

>>Hi all,
>>
>>I need some help with debugging a "deadlock" in a production
>>environment. I've managed to get a crash dump and have tried to
>>analyzed it via WinDbg and SOS. Unfortunately, I am not skilled enough
>>in debugging to pinpoint the exact problem; however, I have narrowed it
>>down to the garbage collection process. All threads seem to be waiting
>>for the GC to complete; when the process hangs, ASPNET_WP.EXE is taking
>>up 100% cpu.
>>
>>Can someone suggest what methods I could use to fully show and pinpoint
>>where the GC is locking up? I have briefly skimmed the SOS docs and a
>>lot of the exposed functions seem useful; however, I can't really make
>>heads or tails out of them. I've examined the heap via !dumpheap -stat,
>>and the largest object that comes back is a whopping 22 MB System.String
>>object. In the MSDN docs, what is supposed to follow is a listing of
>>Large Objects; however, I figure that maybe because I am using the 1.1
>>framework, large objects aren't really differentiated any differently.
>>I managed to get the address by issuing a dumpheap -mt <mt of the
>>System.String> and I got a huge listing of addresses that really don't
>>make much sense to me. I certainly don't want to issue an !objsize
>><addr> on each of these (there are over 30k addresses).
>>
>>I think my biggest concern is that I want to be able to load the symbols
>>of the .NET assemblies that are being used so that I can analyze the
>>managed stack thoroughly, yet I'm not too certain how to do this. If
>>that's not necessary to find out the core problem, then I guess I need
>>to be directed towards the right place.
>>
>>Thanks in advance everyone,
>>
>>Jeff Cheung

>
>
>