Runtime: Add features to help diagnose unloading issues

[janvorli]

There are many things that can prevent successful unload of AssemblyLoadContext, like threads running in the context, GC handles holding references to objects of types loaded into the context, other external references to objects of types loaded into the context etc. It is inherently non-trivial to debug such issues and so we need to figure out what we can do to help diagnose such issues. This can be done on the runtime side or the debugger side.

[tommcdon]

@janvorli I believe that SOS support was added for this. Is there any other work outstanding for .net core 3.0?

[tommcdon]

@janvorli ping

[janvorli]

I was originally thinking we could do something more than just the SOS support we have. For example, when a collectible AssemblyLoadContext is being held alive by a thread local or static variable in another collectible context, the gcroot command would show you a reference chain that goes through a managed LoaderAllocator, but it is not trivial to find which AssemblyLoadContext the LoaderAllocator belongs to. You have to go from that LoaderAllocator to the associated LoaderAllocatorScout, in that one get the m_nativeLoaderAllocator member, dump the native LoaderAllocator at that address, get its m_hLoaderAllocatorObjectHandle field value and extract the managed reference from the handle. It would be great to have a new SOS command that would find AssemblyLoadContext for a LoaderAllocator.
So I'd like to keep this open for a little longer to see if I can find some time to add something like that.

[leculver]

@janvorli I'm ramping up a bit in this feature. I need to add support to ClrMD, which @NextTurn helpfully contributed most of the implementation in https://github.com/microsoft/clrmd/pull/309.

Are GCDescs for collectable methodtables different from the regular implementation? Off the top of my head ClrMD makes these assumptions:

1. All types use GCDescs to walk GC object references.
2. MethodTables for objects on the heap will be valid as long as the object lives on the heap (even if it's garbage and will be collected next GC).
3. GCDescs live in the negative space just before a MethodTable. For example: https://github.com/microsoft/clrmd/blob/be1059dcccbed6047f6ddcb7b210d3c6ecfbd67b/src/Microsoft.Diagnostics.Runtime/src/Desktop/BaseDesktopHeapType.cs#L39

Alternatively, is there a better changeset/spec for me to read and ramp up?

[janvorli]

@leculver the collectible types have only one difference from the non-collectible ones. The MethodTable of the type contains a reference to the underlying managed LoaderAllocator that the GC walks when walking object references. That way, the managed LoaderAllocator object is kept alive as long as there are objects of types from assemblies loaded into the unloadable context.
The collectible types can be identified using MethodTable::Collectible. The MethodTable::GetLoaderAllocator can be used to get pointer to the native AssemblyLoaderAllocator. which in turn contains weak handle to the managed LoaderAllocator and also a pointer to CLRPrivBinderAssemblyLoadContext. That one in turn contains a handle to the managed AssemblyLoadContext that loaded the assembly containing the type represented by the MethodTable.

Please note that assemblies and types can also be loaded into non-collectible AssemblyLoadContext. For those, the MethodTable::GetLoaderAllocator would work the same way as for the collectible ones. The only difference is in the return value of the MethodTable::Collectible.

I think that my design doc https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/unloadability.md can give you all the additional details. But I'll be happy to help to fill in more details if needed.

[raffaeler]

@janvorli do you have any update on this?
I am currently trying to follow your suggestion to find the AssemblyLoadContext ...

[raffaeler]

@janvorli gently pinging :)

[janvorli]

@raffaeler I am sorry for missing your previous comment. There were no additions to the diagnostic tools since the SOS commands were fixed to properly handle dependencies for unloadable objects.
Have you seen my doc https://docs.microsoft.com/en-us/dotnet/standard/assembly/unloadability? There is a lot of detail on how to debug unloadability issues.
If that's not helpful for your case, please share more details on your problem and I'll be happy to help you to debug it.

[raffaeler]

@janvorli Thanks for your answer.
I know that document and it is not always straightforward understanding which is the ALC for each object.
I am currently using the new CLRMD 2.0 release and searching for System.Reflection.LoaderAllocator instances, it is difficult to understand which objects belong to each ALC.
Do you have any hints? (I am trying to do that programmatically)

[janvorli]

Unfortunately, this is the case when you currently need a native debugger (WinDbg on Windows, LLDB on Unix) and follow the steps I've suggested. It is true though that I didn't write these steps in an easy to follow detailed way that one could easily follow, but rather as a brief explanation of the issue.
If using the native debugger is an option for you, I'd be happy to put down an exact step by step way to get that.
Obviously, the ideal way would be to add that to the SOS extension so that you can just run a SOS command to get an AssemblyLoadContext for a given object. And having the same ability in clrmd.

[raffaeler]

@janvorli thanks again.
Yes, it is a feasable option. I would be very interested in having the detailed information using a native debugger.
BTW I wrote a debugger for a customer and I could integrate that in my code if they will, therefore, it would be really precious.

[janvorli]

Ok, let me put it together for you.

[janvorli]

@raffaeler here are example steps to get from an object to the related AssemblyLoadContext using WinDbg. The address used in each step is taken from the output of the previous step.

```
0:000> !DumpObj /d 000002520000e108
Name: Plugin.PluginClass
MethodTable: 00007ffc1d96ce08
EEClass: 00007ffc1d96ccf8
Size: 24(0x18) bytes
File: D:\git\samples\core\tutorials\Unloading\Hostbin\Debug\netcoreapp3.1........\Pluginbin\Debug\netcoreapp3.0\Plugin.dll
Fields:
None

0:000> !DumpMT /d 00007ffc1d96ce08
EEClass: 00007ffc1d96ccf8
Module: 00007ffc1d96c080
Name: Plugin.PluginClass
mdToken: 0000000002000002
File: D:\git\samples\core\tutorials\Unloading\Hostbin\Debug\netcoreapp3.1........\Pluginbin\Debug\netcoreapp3.0\Plugin.dll
LoaderAllocator: 000002520000b268
BaseSize: 0x18
ComponentSize: 0x0
DynamicStatics: false
ContainsPointers false
Slots in VTable: 9
Number of IFaces in IFaceMap: 1

0:000> !DumpObj /d 000002520000b268
Name: System.Reflection.LoaderAllocator
MethodTable: 00007ffc1d94a378
EEClass: 00007ffc1d95fce8
Size: 48(0x30) bytes
File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.4\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffc1d94a4a0 4000eca 8 ...derAllocatorScout 0 instance 000002520000b2d8 m_scout
00007ffc1d846610 4000ecb 10 System.Object[] 0 instance 000002520000b298 m_slots
0000000000000000 4000ecc 20 1 instance 000002520000b288 m_methodInstantiations
00007ffc1d84b1e8 4000ecd 18 System.Int32 1 instance 3 m_slotsUsed

0:000> !DumpObj /d 000002520000b2d8
Name: System.Reflection.LoaderAllocatorScout
MethodTable: 00007ffc1d94a4a0
EEClass: 00007ffc1d95ff88
Size: 24(0x18) bytes
File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.4\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffc1d84ecc0 4000ec9 8 System.IntPtr 1 instance 0000025270BED360 m_nativeLoaderAllocator

0:000> dt (AssemblyLoaderAllocator)0000025270BED360 m_binderToRelease
coreclr!AssemblyLoaderAllocator
+0x4e0 m_binderToRelease : 0x00000252`72a51d70 CLRPrivBinderAssemblyLoadContext

0:000> dt (CLRPrivBinderAssemblyLoadContext)0x00000252`72a51d70 m_ptrManagedAssemblyLoadContext
coreclr!CLRPrivBinderAssemblyLoadContext
+0x160 m_ptrManagedAssemblyLoadContext : 0n2553101555672

0:000> dq 0n2553101555672
0000025270b617d8 000002520000b1f8 000002520000abd8 0000025270b617e8 000002520000ab40 000002520000aa08
0000025270b617f8 000002520000a6f8 0000000000000000 0000025270b61808 0000000000000000 0000000000000000
0000025270b61818 0000000000000000 0000000000000000 0000025270b61828 0000000000000000 0000000000000000
0000025270b61838 0000000000000000 0000000000000000 0000025270b61848 0000000000000000 0000000000000000

0:000> !dumpobj 00000252`0000b1f8
Name: Host.HostAssemblyLoadContext
MethodTable: 00007ffc1d948188
EEClass: 00007ffc1d95ab20
Size: 88(0x58) bytes
File: D:\git\samples\core\tutorials\Unloading\Hostbin\Debug\netcoreapp3.1\Host.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffc1d840ae8 4000ae2 8 System.Object 0 instance 000002520000b250 _unloadLock
0000000000000000 4000ae3 10 0 instance 0000000000000000 _resolvingUnmanagedDll
0000000000000000 4000ae4 18 0 instance 0000000000000000 _resolving
00007ffc1d98e0c8 4000ae5 20 ...Private.CoreLib]] 0 instance 000002520000e120 _unloading
00007ffc1d901e18 4000ae6 28 System.String 0 instance 0000000000000000 _name
00007ffc1d84ecc0 4000ae7 30 System.IntPtr 1 instance 0000025272A51D70 _nativeAssemblyLoadContext
00007ffc1d84c568 4000ae8 38 System.Int64 1 instance 0 _id
00007ffc1d920ae0 4000ae9 40 System.Int32 1 instance 0 _state
00007ffc1d8471c8 4000aea 44 System.Boolean 1 instance 1 _isCollectible
00007ffc1d94b018 4000ae0 828 ...Private.CoreLib]] 0 static 000002520000b2f0 s_allContexts
00007ffc1d84c568 4000ae1 920 System.Int64 1 static 1 s_nextId
0000000000000000 4000aeb 830 ...yLoadEventHandler 0 static 0000000000000000 AssemblyLoad
0000000000000000 4000aec 838 ...solveEventHandler 0 static 0000000000000000 TypeResolve
0000000000000000 4000aed 840 ...solveEventHandler 0 static 0000000000000000 ResourceResolve
0000000000000000 4000aee 848 ...solveEventHandler 0 static 0000000000000000 AssemblyResolve
0000000000000000 4000aef 850 0 static 0000000000000000 s_asyncLocalCurrent
00007ffc1d949a48 4000001 48 ...ependencyResolver 0 instance 000002520000b4f8 _resolver```

[raffaeler]

I am reading from my phone and I can't get where did you take the addresses ending as 17f0 and 3672. But tomorrow morning I will try to understand with windbg and replicate it with clrmd which has the access to the while process memory.
Thanks

[janvorli]

Oops, I've done it twice and maybe combined them by the end. Let me fix it.

[janvorli]

It should be correct now.

[raffaeler]

Bingo, I was able to get the name of the ALC for the object using CLRMD.
The obvious questions is:
how likely the vtable offsets 0x4e0 and 0x160 could change in future releases? (minor, major, ...)

Disclaimer: I absolutely understand and agree that I cannot rely on them in production and that you could change it at any time.
But since this is just a programmatic way to repeat the same actions you do in WinDbg, it's an acceptable risk if they don't change very often (as I would expect).
In any case, I can at least get the offsets from the PDBs, but in this case I would ask you how likely the field names could change in the future.

Thank you

[weltkante]

I assume you're asking for name consistency in LTS/service updates? For major releases you already have to test for compatibility anyways, they may have added new features or changed the way you reach the ALC (e.g. another indirection). I don't think its possible to "guarantee" indefinite upwards compatibility without abstracting the process away into the DAC.

[raffaeler]

@weltkante as I wrote I am not expecting any assurance for leaving that unchanged.
Anyway, reading many core parts of the runtime, I can see comments where the vtable is considered as a published contract and very unlikely to change in the future.
I just wanted to have an idea from @janvorli, take that as an estimation of probability if you will.

[janvorli]

@raffaeler these are not vtable offsets but rather field offsets in the types. I think that they can change any time. Also note that they are different between 32 and 64 bit platforms and may be different between windows / unix or even architectures.

I would rather enable clrmd to be able to access those fields. If I recall it correctly, clrmd uses DAC to access stuff from the managed process. I want to add a SOS command to extract the ALC for object too, so I'd need to expose all the necessary information via DAC and then it might be simple to let clrmd take advantage of that.

[weltkante]

clrmd is only doing managed analysis and removed the native PDB reader

[raffaeler]

@janvorli That sounds a very good plan.
Any estimated timeframe?

[janvorli]

I am not sure how much time it would take. I can see that the whole CLRPrivBinderAssemblyLoadContext is not exposed via DAC and neither is the m_binderToRelease field of AssemblyLoaderAllocator. Exposing some stuff via DAC might have a domino effect (you'd need to expose more types that these depend on).
I'll take an initial look in the next couple of days to get better idea on this.

[janvorli]

@raffaeler I have merged in a change that exposes a new method on ISOSDacInterface8, named GetAssemblyLoadContext that gets address of the managed AssemblyLoadContext for a given MethodTable. So I hope it should be easy to consume it in the clrmd. It works for both collectible and non-collectible AssemblyLoadContexts, including the default one. As for the default one, the managed part is created on demand, so the method will return null if it was not created yet.

[raffaeler]

Thank you @janvorli!!
I guess the integration should be done/modeled by @leculver now.
Great work :)

[leculver]

Yep, I will start on that work this week.

[tommcdon]

Closing as the appropriate DAC/CLRMD API's have been added. Please feel free to open an issue in dotnet/diagnostics to track any needed SOS work.

[raffaeler]

Thank you all.
BTW @leculver do you have an issue on this in the CLRMD repo? Maybe it is already done and I didn't notice!

[leculver]

The underlying issue with adding this to ClrMD is that I really don't understand this .Net Core feature or what it means to debug it. There's some code that exposes the raw concepts, here's the test that covers the basics of it:

https://github.com/leculver/clrmd/blob/232a21f5474ae34eb91f15a52e0e329fa3e8247f/src/Microsoft.Diagnostics.Runtime.Tests/src/TypeTests.cs#L589

And the properties that expose things in ISOSDacInterface8 are the properties around here:

https://github.com/leculver/clrmd/blob/232a21f5474ae34eb91f15a52e0e329fa3e8247f/src/Microsoft.Diagnostics.Runtime/src/Common/ClrType.cs#L59

I'm not really sure if that's sufficient to diagnose problems in this area or not though.

[raffaeler]

@leculver
I am already using ClrObject.Type.LoaderAllocatorHandle but you don't go far.
The goal is having the ability to retrieve the AssemblyLoadContext for each managed object.
The procedure requires jumping into native and coming back into managed memory as described by @janvorli here

[janvorli]

@raffaeler the description you have mentioned was needed before I've added the the support for getting AssemblyLoadContext for MethodTable to DAC interface. No such jumps should be necessary anymore.

[raffaeler]

@janvorli Are you saying I should take clrObject.Type.AssemblyLoadContextAddress and get the matching object from there?
I tried on a dump with 2 ALCs but I always see zeros there.
What am I missing?

[janvorli]

Are you sure that specific object's type you were trying it with is actually loaded into a non-default ALC? Getting zero means that the object's type is loaded into the default assembly load context.

[raffaeler]

Yes, because I can see the other ALC by manually poking the memory as you detailed above

On Oct 15, 2020 00:40, Jan Vorlicek [e-mail] wrote:

Are you sure that specific object's type you were trying it with is actually loaded into a non-default ALC? Getting zero means that the object's type is loaded into the default assembly load context.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dotnet/runtime/issues/11157#issuecomment-708697977, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHRNMAPW77JKLACLGWNQFDSKYSEZANCNFSM4M77HS7Q.

[janvorli]

I can see that the AssemblyLoadContextAddress would return 0 also when the DAC was too old and didn't have the version 8 of the interface. Can you check that the _sos8.GetAssemblyLoadContext is actually being called?

[raffaeler]

I am not sure to fully understand the issue.
I can see _sos8 field with the debugger, but it is null. Do I need a preview release of CLRMD or whatelse? (I am working with the nuget package that may be too old for this).

[janvorli]

I can see _sos8 field with the debugger, but it is null

That explains why you are getting always zero from the GetAssemblyLoadContext. Unless there is a bug in ClrMD in getting the _sos8 interface, it means that the mscordaccore.dll ClrMD was using is old and doesn't support the interface to get the AssemblyLoadContext. Since mscordaccore.dll is part of the runtime and matches 1:1 the coreclr.dll, that would mean that you are trying to debug version of runtime that doesn't support that interface. Please note that it is supported in 5.0 only (since preview 7), it was not back ported to 3.1.

[raffaeler]

Now I understand!
As you can see below, after recompiling the target app with .NET, it is working now :)

Thank you very much

[janvorli]

It is great to hear it works now