Aspnetcore: Memory Leak in new ASPNET 2.2 routing?
Describe the bug
I was editing some Razor Pages (the Project\Pages kind) then refreshing the changes in Chrome (Version 71.0.3578.98). My dev loop is tight, so I make a small edit then refresh the page, and the change is visible in Chrome.
Eventually, I noticed my edits were not making their way to the browser. On a few occasions, I noticed waiting a longer time delay before refreshing the browser sometimes helped the situation. Eventually, though, edits would no longer make their way to the browser.
Originally, I was using InProcess but decided to switch to OutOfProcess, rebooted PC, and the same problem occurred. Somewhere between 15 minutes to 1 hour of constantly editing Razor Pages in Visual Studio, I begin to notice my edits not showing up in the browser.
With out of OutOfProcess enabled, I noticed, a massive 9.7 GB memory usage in dotnet.exe
(later, trying to debug dotnet.exe, we're up at 27GB)
dotnet.exe:
As dotnet.exe consumes memory, there appears to be a few threads that pop off coreclr_shutdown. Not much to discern, but worth noting.
Next, I attached a VS debugger to dotnet.exe, broke into the debugger and landed in Program.Main(). None of my user code was running. Continue & Pausing the debugger multiple times did show one thread standing out. Specifically, Thread 3952 playing around in the new ASPNET Core 22 routing code? Microsoft.AspNetCore.Routing.DecisionTree.
Specifically, Array.Copy, DecisionTree.Resize and Capacity.set seem to be popular destinations with Thread 3952. Thread 3952 is the only thread that appears to be active during this memory leak.
To Reproduce
Steps to reproduce the behavior:
- Using this version of ASP.NET Core '2.2'
- Edit somewhat complex Razor Pages for about 15 minutes to 1 hour in a tight dev loop.
Expected behavior
I should be able to make edits in Razor Pages and see my changes in the browser without having to restart IISExpress or kill dotnet.exe manually or experience a memory leak during the editing experience.
Additional context
Operating System
- Windows 7
IDE
- Visual Studio 15.9.4
C:\>dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.2.101
Commit: 236713b0b7
Runtime Environment:
OS Name: Windows
OS Version: 6.1.7601
OS Platform: Windows
RID: win7-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.101\
Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed
.NET Core SDKs installed:
1.0.0 [C:\Program Files\dotnet\sdk]
1.0.1 [C:\Program Files\dotnet\sdk]
1.0.2 [C:\Program Files\dotnet\sdk]
1.0.3 [C:\Program Files\dotnet\sdk]
1.0.4 [C:\Program Files\dotnet\sdk]
1.1.0 [C:\Program Files\dotnet\sdk]
2.0.0 [C:\Program Files\dotnet\sdk]
2.0.3 [C:\Program Files\dotnet\sdk]
2.1.4 [C:\Program Files\dotnet\sdk]
2.1.101 [C:\Program Files\dotnet\sdk]
2.1.103 [C:\Program Files\dotnet\sdk]
2.1.104 [C:\Program Files\dotnet\sdk]
2.1.200 [C:\Program Files\dotnet\sdk]
2.1.201 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.300 [C:\Program Files\dotnet\sdk]
2.1.302 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
2.1.402 [C:\Program Files\dotnet\sdk]
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.500 [C:\Program Files\dotnet\sdk]
2.1.502 [C:\Program Files\dotnet\sdk]
2.2.100 [C:\Program Files\dotnet\sdk]
2.2.101 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
Also, here is Process Monitor monitoring file access during the memory leak event:
I will continue to do more testing to see if I can narrow down the issue. Thanks for your help in advance.
Thanks,
Brian Chavez
:chocolate_bar: :cookie: :lollipop: Ronald Jenkees - Stay Crunchy
bchavez
👍2
All 20 comments
As a workaround - could you try setting EnableEndpointRouting = false and let us know if that solves the problem? We'll investigate this but that will help us narrow it down if it's related to new routing features or not.
services.AddMvc(options =>
{
options.EnableEndpointRouting = false;
});
rynowak
on 22 Dec 2018
Yes sir, thank you for the help! Will do, I'll report back what I find.
bchavez
on 22 Dec 2018
If you have a dump I鈥檇 gladly take a look (david.fowler at microsoft)
davidfowl
on 22 Dec 2018
I did more testing and found a reliable repro on my machine:
Straight up: dotnet new webapp; then edit Index.cshtml in the following way:
- Press enter several times at the end of the
Index.cshtmlfile. Maybe add a few characters likefffandaa. - Save the file.
- Refresh the browser. (Notice memory increase when browser reloads.)
- Delete a few characters you added.
- Save the file.
- Refresh the browser.
- Repeat; go to 1.
Executing the repro above in a fairly quick way you can ramp up the memory usage to 3 GB in almost no time flat. In my case, the new app almost consumes 1 GB / minute. (I have an AutoHotkey shortcut that refreshes the last browser tab used).
Here is the project I used to create the following diag chart: fff.zip
Check it out:
dotnet new webapp -o fff
I tried taking a memory snapshot but got dialoged by Visual Studio...
So, if we load the 3 GB memory dump in WinDbg, the tail end of !DumpHeap -stat shows us the following:
0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !DumpHeap -stat
Statistics:
MT Count TotalSize Class Name
......
000007fecfd94040 12 1784 System.Char[][]
000007fecfd88c80 16 1793 System.Boolean[]
000007fe75e05038 6 1824 System.Collections.Immutable.RefAsValueType`1[[System.Collections.Immutable.SecurePooledObject`1[[System.Collections.Generic.Stack`1[[System.Collections.Immutable.RefAsValueType`1[[System.Collections.Immutable.SortedInt32KeyNode`1[[System.Collections.Immutable.ImmutableHashSet`1+HashBucket[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]], System.Collections.Immutable]], System.Collections.Immutable]], System.Collections]], System.Collections.Immutable]][]
000007fe750fc2f8 38 1824 Microsoft.Extensions.Logging.LoggerMessage+<>c__DisplayClass4_0
000007fecfdc6a90 59 1888 System.Reflection.Emit.LineNumberInfo
000007fecf39f1e8 24 1920 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
000007fe7523ecf0 10 1920 Microsoft.Extensions.Caching.Memory.CacheEntry
000007fe751e78a8 24 1920 System.ComponentModel.ReflectTypeDescriptionProvider+ReflectedTypeData
000007fe74d9c9f8 6 1920 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+ChangeTokenInfo, Microsoft.Extensions.FileProviders.Physical]][]
000007fe7516bc28 2 1968 System.Xml.XmlWellFormedWriter+State[]
000007fe74e46e38 41 1968 Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine+<>c__DisplayClass1_0
000007fe7500c078 51 2040 Microsoft.Extensions.Logging.Logger
000007fe74fd0b98 23 2048 System.Reflection.RuntimeMethodInfo[][]
000007fe74c33ea0 18 2064 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe758fcce0 65 2080 System.Collections.Immutable.ImmutableArray`1+Builder[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758fbe10 65 2080 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]]
000007fe751c25c8 11 2112 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelBindingMessageProvider
000007fe756ee0f0 1 2160 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Razor.Language.Legacy.CSharpTokenType, Microsoft.AspNetCore.Razor.Language],[System.String, System.Private.CoreLib]][]
000007fe756c3378 1 2160 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
000007fe7545cfc0 1 2160 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Razor.Language.Legacy.CSharpKeyword, Microsoft.AspNetCore.Razor.Language]][]
000007fe75447890 1 2160 System.Collections.Generic.Dictionary`2+Entry[[System.Int32, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]
000007fe75012f60 1 2160 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.Extensions.Logging.Logger, Microsoft.Extensions.Logging]][]
000007fe74c3ba88 27 2160 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fecfda43c8 46 2208 System.Text.StringBuilder
000007fe75a5e4e0 40 2240 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithValueAndTrivia`1[[System.Int32, System.Private.CoreLib]]
000007fe754c9880 15 2280 System.IO.Pipelines.BufferSegment[]
000007fe74fc8f28 74 2368 Microsoft.Extensions.DependencyInjection.ServiceLookup.TransientCallSite
000007fecfdc7068 76 2432 System.Reflection.Emit.GenericFieldInfo
000007fe750fc398 38 2432 System.Action`2[[Microsoft.Extensions.Logging.ILogger, Microsoft.Extensions.Logging.Abstractions],[System.Exception, System.Private.CoreLib]]
000007fe75012b20 51 2448 Microsoft.Extensions.Logging.EventSource.EventSourceLogger
000007fe74fe2718 51 2448 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Extensions.Logging.Console.ConsoleLogger, Microsoft.Extensions.Logging.Console]]
000007fe751e5aa0 13 2496 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadata
000007fe74e42b40 2 2512 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]][]
000007fe74c383a8 6 2512 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Resources.ResourceLocator, System.Private.CoreLib]][]
000007fe75469a48 63 2520 Microsoft.AspNetCore.Razor.Language.RazorDiagnosticDescriptor
000007fe75e15588 3 2632 System.Runtime.CompilerServices.ConditionalWeakTable`2+Entry[[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis],[System.Collections.Generic.Dictionary`2[[Microsoft.CodeAnalysis.SyntaxTrivia, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.SyntaxNode, Microsoft.CodeAnalysis]], System.Private.CoreLib]][]
000007fecfdbdd38 24 2688 System.Diagnostics.Tracing.EventSource+OverideEventProvider
000007fe75222798 4 2736 Microsoft.CodeAnalysis.MetadataReference[]
000007fecfdaa1a8 50 2800 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeMethodInfo, System.Private.CoreLib]]
000007fe759734f0 70 2800 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol+SignatureData
000007fe754c8958 10 2800 System.IO.Pipelines.PipeCompletionCallback[]
000007fecfdc6f10 59 2832 System.Reflection.Emit.DynamicMethod+RTDynamicMethod
000007fecfdc6a30 59 2832 System.Reflection.Emit.ScopeTree
000007fecfdc68a0 59 2832 System.Reflection.Emit.SignatureHelper
000007fecfda45d0 118 2832 System.RuntimeTypeHandle
000007fecf3a0270 75 3000 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
000007fe75a69f88 95 3040 Roslyn.Utilities.TextKeyedCache`1+SharedEntryValue[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]]
000007fe755df028 31 3096 Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewBufferValue[][]
000007fe750e0ec0 5 3120 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection
000007fe7544cb30 49 3136 System.Action`2[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe75530e88 66 3168 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
000007fe754d40b0 5 3240 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseHeaders
000007fecfd84f20 12 3264 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Int32, System.Private.CoreLib],[System.IntPtr, System.Private.CoreLib]][]
000007fe74fe1b08 51 3264 Microsoft.Extensions.Logging.Console.ConsoleLogger
000007fe750fabd8 71 3408 Microsoft.Extensions.Logging.LoggerMessage+<>c__DisplayClass5_0`1[[System.String, System.Private.CoreLib]]
000007fe754a2e58 62 3472 System.Buffers.MemoryPoolBlock
000007fe75a704d0 73 3504 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.AttributeArgumentSyntax
000007fe754c71d0 10 3520 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs
000007fecfdb0ec8 63 3528 System.Threading.CancellationTokenSource+CallbackPartition
000007fe75751848 56 3584 System.Func`2[[Roslyn.Utilities.ObjectReader, Microsoft.CodeAnalysis],[Roslyn.Utilities.IObjectWritable, Microsoft.CodeAnalysis]]
000007fecfda84c8 151 3624 System.RuntimeMethodHandle
000007fecfda94c8 77 3696 System.Reflection.RuntimeAssembly
000007fe754d2e38 5 3720 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestHeaders
000007fe75530f30 14 3808 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]][]
000007fecfd95ee8 45 3888 System.Reflection.RuntimePropertyInfo[]
000007fe758b67e8 21 3944 Microsoft.CodeAnalysis.UnifiedAssembly`1[[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe74e42a78 84 4032 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe758c51a8 130 4160 System.Collections.Immutable.ImmutableArray`1+Builder[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fe758c1308 130 4160 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fecfd89520 15 4200 System.Threading.IThreadPoolWorkItem[]
000007fecfdc6db8 59 4248 System.Reflection.Emit.DynamicResolver
000007fe75a685b8 138 4416 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.LiteralExpressionSyntax
000007fe74fdb490 70 4480 System.Func`1[[System.String, System.Private.CoreLib]]
000007fe758c7910 142 4544 Microsoft.CodeAnalysis.Grouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]]
000007fe750fac80 71 4544 System.Action`3[[Microsoft.Extensions.Logging.ILogger, Microsoft.Extensions.Logging.Abstractions],[System.String, System.Private.CoreLib],[System.Exception, System.Private.CoreLib]]
000007fe74e4bfb0 114 4560 Microsoft.Extensions.DependencyInjection.ServiceLookup.IEnumerableCallSite
000007fe75ef2488 11 4664 System.ArraySegment`1[[System.Byte, System.Private.CoreLib]][]
000007fe758c83e0 118 4720 System.Collections.Generic.List`1[[System.Linq.IGrouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Linq]]
000007fe759631b8 1 4752 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.Metadata.MethodDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe74c3c8e8 2 4848 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fecfdac860 76 4864 System.Reflection.RuntimeModule
000007fe758960c0 204 4896 Microsoft.CodeAnalysis.MetadataLocation
000007fe758ce730 156 4992 System.Collections.Immutable.ImmutableArray`1+Builder[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758ca3f0 156 4992 Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe754c9010 15 5040 System.IO.Pipelines.Pipe
000007fe754a56a0 12 5088 System.Collections.Concurrent.ConcurrentQueue`1+Segment[[Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.IOQueue+Work, Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets]]
000007fecfdb9bb8 59 5192 System.RuntimeMethodInfoStub
000007fe75a5c3b0 96 5376 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithValueAndTrivia`1[[System.String, System.Private.CoreLib]]
000007fe75883288 112 5376 Microsoft.CodeAnalysis.AttributeDescription
000007fe74e2e320 84 5376 System.Func`2[[Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope, Microsoft.Extensions.DependencyInjection],[System.Object, System.Private.CoreLib]]
000007fecfdc75c8 226 5424 System.Reflection.Emit.OpCode
000007fecfdb0338 85 5440 System.Threading.CancellationTokenSource
000007fe7594a508 68 5440 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fecfdc6e80 59 5664 System.Reflection.Emit.DynamicMethod
000007fe75af1e80 1 5760 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.WellKnownType, Microsoft.CodeAnalysis]][]
000007fecfdb20f8 52 5824 System.Threading.OverlappedData
000007fecfd89778 41 6232 System.Threading.CancellationTokenSource+CallbackPartition[]
000007fe7589b690 204 6528 Microsoft.CodeAnalysis.MetadataLocation[]
000007fe758956b8 205 6552 Microsoft.CodeAnalysis.CSharp.Symbols.ModuleSymbol[]
000007fe74e47b28 168 6720 Microsoft.Extensions.DependencyInjection.ServiceLookup.SingletonCallSite
000007fe758c72c8 119 6728 System.Linq.IGrouping`2[[System.String, System.Private.CoreLib],[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]][]
000007fe75978168 98 6736 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.TypedConstant, Microsoft.CodeAnalysis]][]
000007fe74cd7ad8 8 6864 System.Collections.Hashtable+bucket[]
000007fe75ae3c90 1 6920 System.Collections.Generic.HashSet`1+Slot[[Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a518a0 304 7296 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken
000007fecfd94d88 213 7544 System.Reflection.RuntimeConstructorInfo[]
000007fe756c8048 319 7656 System.Collections.Generic.Dictionary`2+KeyCollection[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe756fb7c8 322 7728 System.Reflection.Internal.NativeHeapMemoryBlock+DisposableData
000007fe74d909d8 26 7952 Microsoft.Extensions.DependencyInjection.ServiceDescriptor[]
000007fe7589a668 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75898fb0 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75898368 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75896cb0 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75895490 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75893b60 204 8160 System.Collections.Concurrent.ConcurrentDictionary`2[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75778c88 204 8160 Microsoft.CodeAnalysis.ModuleReferences`1[[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe754c97e0 85 8160 System.IO.Pipelines.BufferSegment
000007fe75a57b98 1 8216 Microsoft.CodeAnalysis.CachingIdentityFactory`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.SyntaxKind, Microsoft.CodeAnalysis.CSharp]][]
000007fecfd94248 151 8496 System.Byte[][]
000007fe7593cee0 175 8520 Microsoft.CodeAnalysis.TypedConstant[]
000007fe75da59c8 1 8544 Microsoft.CodeAnalysis.RuntimeMembers.MemberDescriptor[]
000007fe75011ef0 51 8568 Microsoft.Extensions.Logging.LoggerInformation[]
000007fecfdc6cd0 59 8968 System.Reflection.Emit.DynamicILGenerator
000007fe74fc6eb0 194 9312 Microsoft.Extensions.DependencyInjection.ServiceLookup.ConstructorCallSite
000007fe754a58d0 12 9504 System.Collections.Concurrent.ConcurrentQueue`1+Segment+Slot[[Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.IOQueue+Work, Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets]][]
000007fe74e4bce0 235 9640 Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceCallSite[]
000007fe75727fb8 322 10304 Roslyn.Utilities.WeakList`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]]
000007fe757272a8 322 10304 Microsoft.CodeAnalysis.ModuleMetadata[]
000007fe75722850 322 10304 System.Reflection.Internal.NativeHeapMemoryBlock
000007fe75855b58 323 10328 Microsoft.CodeAnalysis.PEModule[]
000007fe75854430 323 10336 Microsoft.CodeAnalysis.AssemblyMetadata+Data
000007fe758644d8 1 10368 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[Microsoft.CodeAnalysis.CommonReferenceManager`2+ReferencedAssemblyIdentity[[Microsoft.CodeAnalysis.CSharp.CSharpCompilation, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol, Microsoft.CodeAnalysis.CSharp]], Microsoft.CodeAnalysis]], System.Private.CoreLib]][]
000007fe750f98c8 264 10560 Microsoft.Extensions.Logging.Internal.LogValuesFormatter
000007fe74e41e00 2 10800 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.DependencyInjection.ServiceLookup.IServiceCallSite, Microsoft.Extensions.DependencyInjection]][]
000007fecf3a03a8 282 11280 System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]]
000007fe75973b80 120 11520 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEParameterSymbol
000007fecfdaa128 213 11928 System.RuntimeType+RuntimeTypeCache+MemberInfoCache`1[[System.Reflection.RuntimeConstructorInfo, System.Private.CoreLib]]
000007fe756fdec8 322 12880 System.Reflection.PortableExecutable.CoffHeader
000007fe756fb178 322 12880 Microsoft.CodeAnalysis.ModuleMetadata
000007fe7588f810 332 13280 System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]]
000007fe75935c80 170 13600 System.Collections.Generic.Dictionary`2[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[System.Collections.Immutable.ImmutableArray`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Collections.Immutable]]
000007fe75889548 205 13640 Microsoft.CodeAnalysis.CSharp.Symbols.AssemblySymbol[]
000007fecfd94e00 71 13848 System.Reflection.RuntimeMethodInfo[]
000007fe74cd0438 261 14616 Microsoft.Extensions.DependencyInjection.ServiceDescriptor
000007fe7589b488 204 14688 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEGlobalNamespaceSymbol
000007fe757279d0 638 15312 System.WeakReference`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]]
000007fe75722b80 644 15456 Microsoft.CodeAnalysis.MetadataId
000007fe7589e558 89 15552 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]], System.Private.CoreLib]][]
000007fe7590ec68 355 16016 Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol[]
000007fe74e417e8 2 16688 System.Collections.Generic.Dictionary`2+Entry[[System.Type, System.Private.CoreLib],[Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory+ServiceDescriptorCacheItem, Microsoft.Extensions.DependencyInjection]][]
000007fe75972648 494 17688 Microsoft.CodeAnalysis.CSharp.Symbols.CSharpAttributeData[]
000007fe755de758 19 17864 Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewBufferValue[]
000007fe75860038 322 18032 System.Reflection.Metadata.Ecma335.NamespaceCache
000007fe7589e1b8 333 18616 System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]][]
000007fe74c39bc0 21 18672 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]
000007fe7590fec0 803 19272 Microsoft.CodeAnalysis.CSharp.Symbols.TypeMap
000007fe75728980 322 20608 Microsoft.CodeAnalysis.MetadataImageReference
000007fe75726468 322 20608 Microsoft.CodeAnalysis.PEModule+PEHashProvider
000007fe756fb5e0 322 20608 System.Reflection.PortableExecutable.PEReader
000007fecfdb2df0 665 21280 System.LazyHelper
000007fe75855ac8 322 23184 Microsoft.CodeAnalysis.PEAssembly
000007fe757266e8 322 23184 Microsoft.CodeAnalysis.AssemblyMetadata
000007fe756fd328 322 23184 System.Reflection.PortableExecutable.PEHeaders
000007fe75966120 68 25344 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe75890108 319 25520 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.Collections.Immutable.ImmutableArray`1[[System.Byte, System.Private.CoreLib]], System.Collections.Immutable]], System.Private.CoreLib]]
000007fe756c7008 319 25520 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]]
000007fe75915078 803 25696 Microsoft.CodeAnalysis.SmallDictionary`2[[Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeWithModifiers, Microsoft.CodeAnalysis.CSharp]]
000007fe75724fa0 644 25760 System.Lazy`1[[Microsoft.CodeAnalysis.IdentifierCollection, Microsoft.CodeAnalysis]]
000007fe75895de8 204 27744 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEModuleSymbol
000007fe7588f1c0 204 27744 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEAssemblySymbol
000007fe75722368 322 28336 System.Reflection.PortableExecutable.CorHeader
000007fecfd86b70 477 28504 System.String[]
000007fecfd88258 913 28616 System.RuntimeType[]
000007fe75968730 239 28680 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEPropertySymbol
000007fe758de2f8 816 29184 Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol[]
000007fecfdad200 304 31616 System.Reflection.RuntimeConstructorInfo
000007fecfd96168 1068 31728 System.Reflection.ParameterInfo[]
000007fecfd834e8 8 33144 System.UInt32[]
000007fecfdaaa20 327 34008 System.Reflection.RuntimePropertyInfo
000007fecfd83550 6 35692 System.UInt16[]
000007fe757281f8 639 35752 System.WeakReference`1[[Microsoft.CodeAnalysis.IAssemblySymbol, Microsoft.CodeAnalysis]][]
000007fe75a5f028 921 36840 Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken+SyntaxTokenWithTrivia
000007fecfda9fa8 245 37240 System.RuntimeType+RuntimeTypeCache
000007fe75724c48 322 38640 Microsoft.CodeAnalysis.PEModule
000007fe75724e80 644 41216 System.Func`1[[Microsoft.CodeAnalysis.IdentifierCollection, Microsoft.CodeAnalysis]]
000007fe758d9c20 697 44608 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.MetadataDecoder
000007fe756fbf50 322 46368 System.Reflection.PortableExecutable.SectionHeader[]
000007fe75a55848 1 49176 Roslyn.Utilities.TextKeyedCache`1+LocalEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a54ac8 1 49176 Roslyn.Utilities.TextKeyedCache`1+LocalEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe75895250 793 50752 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe75971bb8 697 55760 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEAttributeData
000007fe7589a428 1187 56976 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758d6068 2377 57048 System.Collections.Generic.Dictionary`2+ValueCollection[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7589a4f0 204 60048 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeReferenceHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe758d10c8 983 61936 Microsoft.CodeAnalysis.CSharp.Symbol[]
000007fe758622f0 848 63376 Microsoft.CodeAnalysis.AssemblyIdentity[]
000007fecfdace08 696 66816 System.Reflection.RuntimeParameterInfo
000007fecfdacd88 844 67520 System.Signature
000007fe75895318 204 70176 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.CodeAnalysis.MetadataTypeName+Key, Microsoft.CodeAnalysis],[Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe7594f0f0 677 70408 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PEMethodSymbol
000007fe75916608 1129 72256 Microsoft.CodeAnalysis.SmallDictionary`2+AvlNode[[Microsoft.CodeAnalysis.CSharp.Symbols.TypeParameterSymbol, Microsoft.CodeAnalysis.CSharp],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeWithModifiers, Microsoft.CodeAnalysis.CSharp]]
000007fecfdac078 2425 77600 System.Version
000007fe758f8d80 883 77704 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+UncommonProperties
000007fe756fe9a8 322 79856 System.Reflection.PortableExecutable.PEHeader
000007fe758fe070 1705 81840 Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol+InterfaceInfo
000007fe7590efe0 937 89952 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PETypeParameterSymbol
000007fe758ca188 862 89952 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe75936840 148 90960 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[System.Collections.Immutable.ImmutableArray`1[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata]], System.Collections.Immutable]][]
000007fecfda7e40 2317 92680 System.RuntimeType
000007fecfdaa468 893 92872 System.Reflection.RuntimeMethodInfo
000007fe7591d250 802 96240 Microsoft.CodeAnalysis.CSharp.Symbols.ConstructedNamedTypeSymbol
000007fe758c5728 2641 96508 System.Reflection.Metadata.TypeDefinitionHandle[]
000007fe758c95b8 1405 112400 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe758d8008 3052 115832 Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol[]
000007fe758c93b0 1319 116072 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENestedNamespaceSymbol
000007fecfda4420 5668 136032 System.Object
000007fe758981f0 204 178032 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe758cb980 2377 190160 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7575ee08 2423 193840 Microsoft.CodeAnalysis.AssemblyIdentity
000007fe758cf5b0 1169 224448 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolGeneric
000007fe756c8708 63 229968 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.ValueTuple`2[[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]], System.Private.CoreLib]][]
000007fecfd861c8 951 236136 System.Object[]
000007fe75a5a258 4 272000 Microsoft.CodeAnalysis.ArrayElement`1[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fecfd88ce8 129 300858 System.Char[]
000007fe758d5488 1810 422832 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe758ce998 12919 578824 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]
000007fe75898128 12308 590784 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe7585cc48 322 757344 System.Reflection.Metadata.MetadataReader
000007fe75a68a68 1 1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe75a54e60 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe75a53d30 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe7585e750 4 1146976 Roslyn.Utilities.StringTable+Entry[]
000007fecfd88d50 3882 1314856 System.Byte[]
000007fe758cf908 11139 1871352 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolNonGeneric
000007fecfda3770 31429 2486442 System.String
000007fecfde0c88 142268 3414432 System.Collections.Generic.ComparisonComparer`1[[System.Int32, System.Private.CoreLib]]
000007fe75687238 142268 3414432 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree+OutboundMatchClassifier
000007fe75689f80 142270 3414480 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer
000007fe75687430 142275 3414600 Microsoft.AspNetCore.Routing.RouteValueEqualityComparer
000007fe75686e70 142275 3414600 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree
000007fe75683688 142270 4552640 System.SZArrayHelper+SZGenericArrayEnumerator`1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Routing.Abstractions]]
000007fe75690128 142274 4552768 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756921e0 142270 5690784 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe75684e88 284554 6829296 Microsoft.AspNetCore.Routing.Template.TemplateSegment
000007fe756927a8 142268 7967008 System.Linq.EnumerableSorter`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe756907d0 142275 7967368 Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fecfda2170 142268 9105152 System.Comparison`1[[System.Int32, System.Private.CoreLib]]
000007fe756868e8 284555 9105752 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fecfdb0f38 284595 9107040 System.Threading.CancellationTokenRegistration
000007fe75693720 142270 10243440 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe7568d1c8 142269 11381520 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe75692ce0 142275 11382000 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75682ae0 142275 11382000 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Internal.OutboundMatchResult, Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
00000000004f8760 514256 12352146 Free
000007fe75685958 284536 13657728 System.Linq.Enumerable+SelectArrayIterator`2[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment, Microsoft.AspNetCore.Routing],[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe7568fb70 142269 13657824 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe7568f380 142269 13657824 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe75693bd8 142274 13658304 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe756816e0 426831 13658592 Microsoft.AspNetCore.Routing.Tree.OutboundMatch
000007fe75686268 284554 15935024 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe75685eb8 284555 15935048 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe756877f0 426810 17072400 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756851f0 426831 17073240 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe75684d70 426831 17073240 Microsoft.AspNetCore.Routing.Template.RouteTemplate
000007fe7568a3a0 569072 18210304 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+TreeBuilderContext[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe75692c38 569096 18211072 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756878a0 569076 22763040 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe756901d0 569097 22763880 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe7544c250 426806 23901136 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe75681770 711369 28454760 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe756862f8 711385 28455400 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe75687e90 569077 30730128 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe74e80ce8 426806 30730928 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe756914e0 569070 31867920 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe756834a0 569095 31869288 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe75688a18 426810 34144800 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe7551c2a0 853678 34147120 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe7568a448 569072 36420608 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe74e80438 569075 36420800 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe75691d98 569070 40973040 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe75689ca8 426810 40973760 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe75684b00 426831 40975776 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fe7568d9a8 569070 45525600 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe7568e708 853605 47801880 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fecfd82aa8 1711821 59556316 System.Int32[]
000007fe74d12b28 1715460 68618400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fecfd9a4b0 853692 75124192 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe74d128d8 1715464 109789696 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fecfd68868 1715471 109790144 System.Action
000007fecfd95b20 1715491 109791424 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fecfdb23c0 3168844 253507520 System.Threading.CancellationTokenSource+CallbackNode
Total 30766425 objects
And from my app that started this issue (1.7 GB dump):
0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !DumpHeap -stat
Statistics:
MT Count TotalSize Class Name
......
000007fed0a28d50 4588 1042037 System.Byte[]
000007fe7682a1f0 1 1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe76816950 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe76815820 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe76299348 26681 1067240 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe762b1690 27000 1080000 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76488c30 4 1146976 Roslyn.Utilities.StringTable+Entry[]
000007fe76296b40 21057 1179160 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe762923f8 29978 1199120 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76155e88 23569 1319864 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe76297770 39105 1467928 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fe762b29a0 26538 1486128 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe762b4be0 8873 1492488 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe76299938 26682 1588656 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe76296ef0 30080 1684480 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe7629bef0 26550 1699200 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75ca2568 26554 1699456 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe75ca2920 23588 1699552 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe76294128 29929 1771256 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe7629a4c0 23752 1900160 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe762b3258 26537 1910664 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe76169bf0 48147 1925880 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe7629f450 26545 2123600 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe762b10a8 8859 2129952 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe762b5098 8905 2146368 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe76296f80 54144 2165760 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe7629b750 23752 2280192 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe76295788 24064 2310144 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fe762b0240 47141 2639896 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fed0a43770 33901 3117148 System.String
000007fed0a22aa8 84229 3798748 System.Int32[]
000007fed0a3a4b0 48168 4237744 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fe75612c10 2739466 109578640 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe756129c0 2739473 175326272 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed0a08868 2739477 175326528 System.Action
000007fed0a35b20 2739501 175328064 System.Action`1[[System.Object, System.Private.CoreLib]]
00000000005a8d90 1558991 175859240 Free
000007fed0a523c0 3381845 270547600 System.Threading.CancellationTokenSource+CallbackNode
Total 17048248 objects
FWIW, I found that EnableEndpointRouting = false helps the situation (in the case of the fff.zip project). Memory consumption is at a slower rate but still shares the same monotonically increasing characteristics. (approx 700MB / 10 minutes).
In both cases, memory is never reclaimed by the OS.
EnableEndpointRouting = false:
@davidfowl , I'll send you an email soon.
@rynowak , I'll continue to do more testing with EnableEndpointRouting = false. I just wanted to make a quick post with what I found so far.
Thanks,
Brian
:boom: :fire: "Set it ablaze like a candle wick... Light it up, light it up..."
bchavez
on 23 Dec 2018
I have a hunch but I need to see the gc root of these:
C#
000007fe75612c10 2739466 109578640 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe756129c0 2739473 175326272 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed0a08868 2739477 175326528 System.Action
000007fed0a35b20 2739501 175328064 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed0a523c0 3381845 270547600 System.Threading.CancellationTokenSource+CallbackNode
This looks like it's the leak. Something isn't cleaning up cancellation token registrations (I've been on a mission cleaning this up since 2.2 but we still have a ways to go).
davidfowl
on 23 Dec 2018
Yep, got it:
davidfowl
on 23 Dec 2018
It looks like these aren't rooted though. I think they'll eventually get GCed. Still looking.
davidfowl
on 23 Dec 2018
I don't know if this is two sides to the same coin, but in my webapp, (generally following the same repro steps), I can get ASPNET to consume memory without any file saves or browser refreshes.
In the GIF below:
- The first ramp up is using the same repro steps posted.
- However, I stopped and waited for ASPNET to catch up... memory usage leveled off (the plateau/flat part in the pic).
- Then started the same repro steps again.. but about 10-20 iterations. Then stopped.
- Except, this time, ASPNET continued eating memory without any typing, saving, or browser refresh steps.
Just sit back and relax. There's a persistent 10-30% CPU usage as ASPNET goes on by itself. This is what is shown in the 2nd ramp up.
Waited up to 7GB (not doing anything) and still no end in sight:
Paused the debugger, caught Thread 7 in Array.Copy:
Created dump. Loaded in WinDbg:
0:000> .load C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.2.0\sos.dll
0:000> !threads -live
ThreadCount: 49
UnstartedThread: 0
BackgroundThread: 19
PendingThread: 0
DeadThread: 29
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
0 1 6c4 00000000004fcbf0 202a028 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA
15 2 32d0 0000000000561360 2b228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA (Finalizer)
16 3 1274 0000000000480ac0 102a228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA (Threadpool Worker)
17 7 3088 000000000d359270 10a9228 Preemptive 000000025D992928:000000025D9946F0 000000000045d670 4 MTA (Threadpool Worker)
18 8 2cb0 000000000d39de70 202b228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA
19 9 291c 000000000d482010 202b228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA
21 18 27e8 000000000d59ac70 10a9228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA (Threadpool Worker)
22 34 1218 000000001277d050 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
23 35 2ccc 000000001277d6e0 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
24 36 2c0c 000000001277dd70 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
25 37 2c94 000000001277e400 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
26 38 2d54 000000001277ea90 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
27 39 322c 000000001277f120 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
28 40 23a8 00000000127917c0 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
29 41 2d90 0000000012791e50 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
30 42 2e30 00000000127924e0 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
31 43 2d6c 0000000012792b70 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
32 44 33b8 0000000012793200 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
33 45 2434 0000000012793890 21228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 Ukn
34 33 2810 000000000d5998c0 8029228 Preemptive 0000000000000000:0000000000000000 000000000045d670 0 MTA (Threadpool Completion Port)
0:000> ~17s
ntdll!ZwWaitForSingleObject+0xa:
00000000`7742d9fa c3 ret
0:017> !clrstack
OS Thread Id: 0x3088 (17)
Child SP IP Call Site
000000000eacdca8 000000007742d9fa [HelperMethodFrame_1OBJ: 000000000eacdca8] System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32, Boolean)
000000000eacdf90 000007fed47de7f8 System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32) [E:\A\_work\287\s\src\mscorlib\src\System\Array.cs @ 231]
000000000eacdfd0 000007fed344236b System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].SetCapacity(Int32) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1192]
000000000eace030 000007fed34422e1 System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].IncreaseCapacity() [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1174]
000000000eace060 000007fed344250f System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].AddIfNotPresent(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1247]
000000000eace0e0 000007fed34411db System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].Add(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 469]
000000000eace110 000007fecfef6bc2 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateNode(TreeBuilderContext<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer, System.Collections.Generic.List`1<Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1<System.__Canon>>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 161]
000000000eace240 000007fecfef6623 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateTree(System.Collections.Generic.IReadOnlyList`1<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.IClassifier`1<System.__Canon>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 91]
000000000eace2b0 000007fecff09932 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree..ctor(System.Collections.Generic.IReadOnlyList`1<Microsoft.AspNetCore.Routing.Tree.OutboundMatch>) [/_/src/Microsoft.AspNetCore.Routing/Internal/LinkGenerationDecisionTree.cs @ 27]
000000000eace2f0 000007fecfef5f06 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.BuildOutboundMatches() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 95]
000000000eace340 000007fecfef5e53 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 84]
000000000eace380 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace3c0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace450 000007fecfeecc68 Microsoft.AspNetCore.Routing.CompositeEndpointDataSource.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/CompositeEndpointDataSource.cs @ 112]
000000000eace4c0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace500 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace590 000007fed2f6dc22 Microsoft.AspNetCore.Mvc.Internal.MvcEndpointDataSource.UpdateEndpoints() [/_/src/Microsoft.AspNetCore.Mvc.Core/Internal/MvcEndpointDataSource.cs @ 201]
000000000eace690 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace6d0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace760 000007fed2f792c6 Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection() [/_/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs @ 153]
000000000eace7d0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace810 000007fed48977f5 System.Threading.CancellationTokenSource.InternalRegister(System.Action`1<System.Object>, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 546]
000000000eace8a0 000007fed486aa40 System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 236]
000000000eace910 000007fed486a93e System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 192]
000000000eace950 000007fed33e9636 Microsoft.Extensions.Primitives.CompositeChangeToken.RegisterChangeCallback(System.Action`1<System.Object>, System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 48]
000000000eace9a0 000007fed33ec5d9 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 47]
000000000eace9e0 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacea60 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eaceaf0 000007fed33e992a Microsoft.Extensions.Primitives.CompositeChangeToken.OnChange(System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 117]
000000000eaceb50 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eacebe0 000007fed2e0c370 Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) [/_/src/FileProviders/Physical/src/PhysicalFilesWatcher.cs @ 27]
000000000eacec10 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacec90 000007fed484453a System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2440]
000000000eaced30 000007fed487c0d2 System.Threading.ThreadPoolWorkQueue.Dispatch() [E:\A\_work\287\s\src\mscorlib\src\System\Threading\ThreadPool.cs @ 588]
000000000eacf1b0 000007fed51aa673 [DebuggerU2MCatchHandlerFrame: 000000000eacf1b0]
0:017> !DumpHeap -stat
Statistics:
MT Count TotalSize Class Name
.....
000007fed4b68ce8 123 263756 System.Char[]
000007fe766161f8 353 270608 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]][]
000007fe7697fbe8 4 272000 Microsoft.CodeAnalysis.ArrayElement`1[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe76649530 4066 325280 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]]
000007fe7664d160 1720 330240 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolGeneric
000007fed4b8a468 3366 350064 System.Reflection.RuntimeMethodInfo
000007fed4b87e40 9832 393280 System.RuntimeType
000007fe7641be48 13774 440768 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+TreeBuilderContext[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fed4b661c8 2065 482992 System.Object[]
000007fe764165e0 10453 501744 System.Linq.Enumerable+SelectArrayIterator`2[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment, Microsoft.AspNetCore.Routing],[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe76419298 14440 577600 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76415b10 24970 599280 Microsoft.AspNetCore.Routing.Template.TemplateSegment
000007fe76419348 15535 621400 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76412368 19976 639232 Microsoft.AspNetCore.Routing.Tree.OutboundMatch
000007fe766632e8 2983 653232 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Collections.Immutable.ImmutableArray`1[[Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol, Microsoft.CodeAnalysis.CSharp]], System.Collections.Immutable]][]
000007fe762d5e88 11889 665784 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
000007fe764340f8 20874 667968 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe764329a0 13595 761320 System.Linq.OrderedEnumerable`2[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
000007fe76415e78 19976 799040 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplateSegment, Microsoft.AspNetCore.Routing]]
000007fe764159f8 19976 799040 Microsoft.AspNetCore.Routing.Template.RouteTemplate
000007fe76431690 20875 835000 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterion`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe76434be0 4864 841344 System.Collections.Generic.HashSet`1+Slot[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe765dac58 359 844368 System.Reflection.Metadata.MetadataReader
000007fe7664c548 19839 866632 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]
000007fe75e22920 12035 867736 System.Collections.Generic.HashSet`1+Slot[[System.String, System.Private.CoreLib]][]
000007fe7641bef0 13774 881536 System.Collections.Generic.HashSet`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe75e22568 13779 881856 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
000007fe76616130 18637 894576 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Reflection.Metadata.TypeDefinitionHandle, System.Reflection.Metadata],[Microsoft.CodeAnalysis.CSharp.Symbols.TypeSymbol, Microsoft.CodeAnalysis.CSharp]]
000007fe764123f8 22858 914320 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]]
000007fe76419938 15536 921664 Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]][]
000007fe76433258 13573 977256 System.Linq.OrderedEnumerable`1+<GetEnumerator>d__3[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
000007fe76416b40 17480 978848 Microsoft.AspNetCore.Routing.Template.TemplateSegment[]
000007fe7698e060 1 1048600 Microsoft.CodeAnalysis.Syntax.InternalSyntax.SyntaxNodeCache+Entry[]
000007fe7697a7f0 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxToken, Microsoft.CodeAnalysis.CSharp]][]
000007fe769796c0 1 1048600 Roslyn.Utilities.TextKeyedCache`1+SharedEntry[[Microsoft.CodeAnalysis.CSharp.Syntax.InternalSyntax.SyntaxTrivia, Microsoft.CodeAnalysis.CSharp]][]
000007fe7641f450 13709 1096720 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1+Criterion[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]]
000007fe765dc760 4 1146976 Roslyn.Utilities.StringTable+Entry[]
000007fe7641a4c0 14440 1155200 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe764310a8 4742 1183968 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
000007fe76417770 32462 1218560 Microsoft.AspNetCore.Routing.Template.TemplatePart[]
000007fed4b68d50 5018 1274282 System.Byte[]
000007fe76414128 22116 1301920 Microsoft.AspNetCore.Routing.Tree.OutboundMatch[]
000007fe76430240 23802 1332912 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]]
000007fe7641b750 14440 1386240 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValue, Microsoft.AspNetCore.Routing]][]
000007fe76416ef0 24970 1398320 Microsoft.AspNetCore.Routing.Template.TemplatePart
000007fe76435098 5454 1454928 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeNode`1[[Microsoft.AspNetCore.Routing.Tree.OutboundMatch, Microsoft.AspNetCore.Routing]], Microsoft.AspNetCore.Routing]][]
000007fe762e9bf0 39995 1599800 Microsoft.AspNetCore.Routing.RouteValueDictionary
000007fe76416f80 44946 1797840 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Template.TemplatePart, Microsoft.AspNetCore.Routing]]
000007fe76415788 19976 1917696 Microsoft.AspNetCore.Routing.Tree.OutboundRouteEntry
000007fed4b62aa8 53683 2569680 System.Int32[]
000007fe7664d4b8 16917 2842056 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol+PENamedTypeSymbolNonGeneric
000007fed4b7a4b0 40035 3521128 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
000007fed4b83770 39487 3634552 System.String
000000000054c350 5807424 291879606 Free
000007fe75792c10 15268935 610757400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe757929c0 15268942 977212288 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed4b48868 15268958 977213312 System.Action
000007fed4b75b20 15268971 977214144 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed4b923c0 28440798 2275263840 System.Threading.CancellationTokenSource+CallbackNode
Total 96275880 objects
Fragmented blocks larger than 0.5 MB:
Addr Size Followed by
00000003419c8240 0.6MB 0000000341a57b08 Microsoft.CodeAnalysis.CSharp.Symbols.Metadata.PE.PENamedTypeSymbol[]
@davidfowl , this memory dump is on the same server with filename dotnet.bitarmory.web.arraycopy.zip if you want to peek.
bchavez
on 23 Dec 2018
This call stack implies that the file watcher is triggering routing to recompute the route table after a file change occurs. Looking at the previous provided list of threads/stacks shows a LOT of threads with similar stacks.
I don't think it's a leak because all of the memory is still live.
So one of a few obvious possibilities seems possible:
- We have a broken behaviour when a file change occurs that will cause us to burn a thread by spinning over and over recomputing file changes. As you make more changes, more threads get stuck.
- We subscribe to the same change token a bunch of times, causing us to to update the route table
N time when a file change occurs. - File watcher is going BANANAS and triggering like a million notifications for some reason leading to lots of updates that need to be processed.
- You are legitimately triggering a bunch of file change notifications (somehow) which is causing the above.
I don't see anything obviously wrong in this call stack. DecisionTreeBuilder is thread-safe so there's no concurrency issue inside that.
@davidfowl - if you're looking into this also, there interesting bits are here - in order:
https://github.com/aspnet/AspNetCore/blob/release/2.2/src/Mvc/src/Microsoft.AspNetCore.Mvc.RazorPages/Internal/PageActionDescriptorChangeProvider.cs
https://github.com/aspnet/AspNetCore/blob/321327f84b2b22dcff2e9beb06a9a64236c5cced/src/Mvc/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs#L144
https://github.com/aspnet/AspNetCore/blob/master/src/Mvc/src/Microsoft.AspNetCore.Mvc.Core/Routing/MvcEndpointDataSource.cs#L184
https://github.com/aspnet/AspNetCore/blob/master/src/Http/Routing/src/CompositeEndpointDataSource.cs#L139
https://github.com/aspnet/AspNetCore/blob/master/src/Http/Routing/src/RouteValuesAddressScheme.cs#L76
000000000eacdca8 000000007742d9fa [HelperMethodFrame_1OBJ: 000000000eacdca8] System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32, Boolean)
000000000eacdf90 000007fed47de7f8 System.Array.Copy(System.Array, Int32, System.Array, Int32, Int32) [E:\A\_work\287\s\src\mscorlib\src\System\Array.cs @ 231]
000000000eacdfd0 000007fed344236b System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].SetCapacity(Int32) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1192]
000000000eace030 000007fed34422e1 System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].IncreaseCapacity() [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1174]
000000000eace060 000007fed344250f System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].AddIfNotPresent(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 1247]
000000000eace0e0 000007fed34411db System.Collections.Generic.HashSet`1[[System.__Canon, System.Private.CoreLib]].Add(System.__Canon) [E:\A\_work\20\s\corefx\src\System.Collections\src\System\Collections\Generic\HashSet.cs @ 469]
000000000eace110 000007fecfef6bc2 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateNode(TreeBuilderContext<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.DecisionCriterionValueEqualityComparer, System.Collections.Generic.List`1<Microsoft.AspNetCore.Routing.DecisionTree.ItemDescriptor`1<System.__Canon>>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 161]
000000000eace240 000007fecfef6623 Microsoft.AspNetCore.Routing.DecisionTree.DecisionTreeBuilder`1[[System.__Canon, System.Private.CoreLib]].GenerateTree(System.Collections.Generic.IReadOnlyList`1<System.__Canon>, Microsoft.AspNetCore.Routing.DecisionTree.IClassifier`1<System.__Canon>) [/_/shared/Microsoft.AspNetCore.Routing.DecisionTree.Sources/DecisionTreeBuilder.cs @ 91]
000000000eace2b0 000007fecff09932 Microsoft.AspNetCore.Routing.Internal.LinkGenerationDecisionTree..ctor(System.Collections.Generic.IReadOnlyList`1<Microsoft.AspNetCore.Routing.Tree.OutboundMatch>) [/_/src/Microsoft.AspNetCore.Routing/Internal/LinkGenerationDecisionTree.cs @ 27]
000000000eace2f0 000007fecfef5f06 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.BuildOutboundMatches() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 95]
000000000eace340 000007fecfef5e53 Microsoft.AspNetCore.Routing.RouteValuesAddressScheme.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/RouteValuesAddressScheme.cs @ 84]
000000000eace380 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace3c0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace450 000007fecfeecc68 Microsoft.AspNetCore.Routing.CompositeEndpointDataSource.HandleChange() [/_/src/Microsoft.AspNetCore.Routing/CompositeEndpointDataSource.cs @ 112]
000000000eace4c0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace500 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace590 000007fed2f6dc22 Microsoft.AspNetCore.Mvc.Internal.MvcEndpointDataSource.UpdateEndpoints() [/_/src/Microsoft.AspNetCore.Mvc.Core/Internal/MvcEndpointDataSource.cs @ 201]
000000000eace690 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace6d0 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eace760 000007fed2f792c6 Microsoft.AspNetCore.Mvc.Infrastructure.DefaultActionDescriptorCollectionProvider.UpdateCollection() [/_/src/Microsoft.AspNetCore.Mvc.Core/Infrastructure/DefaultActionDescriptorCollectionProvider.cs @ 153]
000000000eace7d0 000007fed33ec5ba Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 41]
000000000eace810 000007fed48977f5 System.Threading.CancellationTokenSource.InternalRegister(System.Action`1<System.Object>, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 546]
000000000eace8a0 000007fed486aa40 System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 236]
000000000eace910 000007fed486a93e System.Threading.CancellationToken.Register(System.Action`1<System.Object>, System.Object) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationToken.cs @ 192]
000000000eace950 000007fed33e9636 Microsoft.Extensions.Primitives.CompositeChangeToken.RegisterChangeCallback(System.Action`1<System.Object>, System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 48]
000000000eace9a0 000007fed33ec5d9 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0.<OnChange>b__0(System.Object) [/_/src/Primitives/src/ChangeToken.cs @ 47]
000000000eace9e0 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacea60 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eaceaf0 000007fed33e992a Microsoft.Extensions.Primitives.CompositeChangeToken.OnChange(System.Object) [/_/src/Primitives/src/CompositeChangeToken.cs @ 117]
000000000eaceb50 000007fed4897ae8 System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\CancellationTokenSource.cs @ 637]
000000000eacebe0 000007fed2e0c370 Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) [/_/src/FileProviders/Physical/src/PhysicalFilesWatcher.cs @ 27]
000000000eacec10 000007fed4791999 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [E:\A\_work\287\s\src\mscorlib\shared\System\Threading\ExecutionContext.cs @ 167]
000000000eacec90 000007fed484453a System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) [E:\A\_work\287\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2440]
000000000eaced30 000007fed487c0d2 System.Threading.ThreadPoolWorkQueue.Dispatch() [E:\A\_work\287\s\src\mscorlib\src\System\Threading\ThreadPool.cs @ 588]
000000000eacf1b0 000007fed51aa673 [DebuggerU2MCatchHandlerFrame: 000000000eacf1b0]
I think you might be on to something @rynowak.
- Do you know where I could enable more tracing for the File Watcher? Perhaps we could get a trace of the file changes that are triggering recompilation.
- Are there any significant changes to the File (System?) Watcher behavior between Windows 7 and Windows 10 that might have been overlooked?
Earlier I tried to set VSIISEXELAUNCHER_TRACEFOLDER for VSIISExeLauncher.exe but Visual Studio wasn't having it: https://twitter.com/bchavez/status/1076929078597283840
Also, I'll try setting up ProcMon to monitor the file paths in the directory to see if I can pickup anything during the autopilot memory ramp up.
bchavez
on 24 Dec 2018
Do you see the leak when you turn off file watching: https://github.com/aspnet/AspNetCore/blob/release/2.2/src/Mvc/src/Microsoft.AspNetCore.Mvc.Razor/RazorViewEngineOptions.cs#L198-L236? As of 2.2, it's disabled by default in non-Development environments.
pranavkm
on 24 Dec 2018
Hi @pranavkm ,
I added the following code before .AddMvc and I still see the issue.
Let me know if I should be configuring it a different way.
With AllowRecompilingViewsOnFileChange = false, it seems the behavior persists.
Here's a 5-min MP4 video: https://files.bitarmory.com/screen_2600.mp4
The video shows how long each edit takes which progressively gets worse on each edit. At the very bottom of the video, there's a white-background area, that's the browser/website for the Razor page.
- By edit 14 and 15, ASPNET seems to start lagging behind.
- Edit 15 - 20, ASPNET really lags behind.
- Particularly interesting is when I get to edit 30 notice how long it takes for edit 31 to appear in the browser after refresh. By edit 31, I've basically lost control of ASPNET.
:green_book: Green gutter color indicates changes saved in VS:
:orange_book: Orange gutter color indicates changes edited (but not saved yet) in VS:
Flash of :white_large_square: white, indicates a browser refresh.
bchavez
on 24 Dec 2018
Are we sure change tokens are being registered (or disposed of) correctly? And that we don't have any kind of daisy chaining of change registrations? If we had daisy chained callbacks it would seem consistent with what I was experiencing as continued subsequent edits took more time to appear in the browser.
Here's what I'm seeing inside CancellationTokenSource:
There seems to be a never ending list of node = node.Next in the for loop (of type CallbackNode) in which our special thread appears to be burning CPU on. This is consistent with the !DumpHeap -stat stats where +CallbackNode appears to be the largest allocation:
000007fe75792c10 15268935 610757400 Microsoft.Extensions.Primitives.ChangeToken+<>c__DisplayClass0_0
000007fe757929c0 15268942 977212288 System.Func`1[[Microsoft.Extensions.Primitives.IChangeToken, Microsoft.Extensions.Primitives]]
000007fed4b48868 15268958 977213312 System.Action
000007fed4b75b20 15268971 977214144 System.Action`1[[System.Object, System.Private.CoreLib]]
000007fed4b923c0 28440798 2275263840 System.Threading.CancellationTokenSource+CallbackNode
Specifically, when node.ExecuteCallback() is called inside CancellationTokenSource for loop, it takes us back here:
Following the jump, ultimately, we land here which causes the re-computation of routes:
Rinse and repeat. This is the hot path the thread is on.
Also, I breakpointed the file change events here and didn't notice anything abnormal popping off during automatic monotonically increasing memory usage.
I'll continue to do more debugging. :beetle: :bug: :hammer:
PS. Fun fact, the node.Id in the CancellationTokenSource for loop is counting backwards as the for loop progresses. :sunglasses:
NodeID: 1340668
NodeID: 1340667
NodeID: 1340666
NodeID: 1340665
NodeID: 1340664
NodeID: 1340663
NodeID: 1340662
...
:beach_umbrella: :trumpet: Beach Boys - Good Vibrations (Nick Warren bootleg)
bchavez
on 24 Dec 2018
Oh wait I see a problem. That code is supposed to be called once ever. The underlying code re-registers for triggering on its own and this does it as well.
davidfowl
on 24 Dec 2018
Yea, pretty interesting behavior. Now that I have the for loop in CancellationTokenSource:ExecuteCallbackHandlers() wired up I'm getting some pretty interesting behavior.
Every iteration of the for loop, when node.Id % 1000 = 0, I print a message. It appears, that only saving the Razor file can cause the callback registrations to double up. Here's the output:
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 16000
node.Id: 15000
node.Id: 14000
node.Id: 13000
node.Id: 12000
node.Id: 11000
node.Id: 10000
node.Id: 9000
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
>>>>>>>>>>>>>>(modify razor file & save)
node.Id: 32000
node.Id: 31000
node.Id: 30000
node.Id: 29000
node.Id: 28000
node.Id: 27000
node.Id: 26000
node.Id: 25000
node.Id: 24000
node.Id: 23000
node.Id: 22000
node.Id: 21000
node.Id: 20000
node.Id: 19000
node.Id: 18000
node.Id: 17000
node.Id: 16000
node.Id: 15000
node.Id: 14000
node.Id: 13000
node.Id: 12000
node.Id: 11000
node.Id: 10000
node.Id: 9000
node.Id: 8000
node.Id: 7000
node.Id: 6000
node.Id: 5000
node.Id: 4000
node.Id: 3000
node.Id: 2000
node.Id: 1000
Finished for loop in CancellationTokenSource:ExecuteCallbackHandlers(). Interlocked.MemoryBarrier() done.
Kinda neat. The situation can get out of hand pretty quick. So far, I'm pretty satisfied we have enough diagnostic information for the issue now. Y'all keepin' my debug skills freshhh :watermelon:. :sunglasses:
I can finally put my mind at rest; gonna to take a few day off now to enjoy the holidays. :christmas_tree: :christmas_tree: :christmas_tree: :christmas_tree:
Happy Holidays friends,
Brian Chavez
:walking_man: :walking_woman: Missing Persons - Walking in L.A.
bchavez
on 25 Dec 2018
Wow thanks for the investigation here Brian. That's definitely a bug.
rynowak
on 27 Dec 2018
👍3
Just throwing this in here that I'm also having the same issue. My dotnet.exe reached 3 GB's of RAM after wondering for like 15 minutes why my cshtml edits weren't even making it to the browser (browser kept reloading a very old copy!). I learned a lot reading this thread. Keep it up guys!
ronester
on 4 Jan 2019
👍2
I have the same issue in .net core 3.0. I cannot see changes when I refresh the .cshtml page.. (Markup and javascript changes.). I need to re-run the project.
Visual Studio version - 2019 preview
.Net Core 3.0
Windows 10 professional latest
Thank you
deepumi
on 4 Jan 2019
👍3
Is this fix available on 2.2 branch?
abdulkareemnalband
on 3 Apr 2019
@abdulkareemnalband Change 3e5b37f22cd7ba2c81b1ac69d55ffae7ea09423f is included since version v2.2.2.
poke
on 3 Apr 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
guardrex
路
3Comments
rynowak
路
3Comments
mj1856
路
3Comments
ermithun
路
3Comments
Pixel-Lord
路
3Comments
Most helpful comment
I have the same issue in .net core 3.0. I cannot see changes when I refresh the .cshtml page.. (Markup and javascript changes.). I need to re-run the project.
Visual Studio version - 2019 preview
.Net Core 3.0
Windows 10 professional latest
Thank you