Runtime: Memory leak with AWS SDK for .NET using .NET Core 3.1.300 on macOS 10.15 Catalina
Description
I'm running into a memory leak when using the AWS SDK for .NET to send data to the AWS Kinesis Data Stream endpoint. Looking at the memory leak report generated using macOS's built-in leaks utility, part of the leak might be caused by System.Security.Cryptography.Native.Apple.dylib.
I have created a min-reproducible example for the memory leak, found in this repository. The operation requirements as well as detailed instructions can be found in the repository's README.
On some machines, the memory usage of the app can grow to >1GB. On other machines, the memory usage does not grow noticeably, but the amount of leaked memory (determined by the leaks tool continues to grow. The leak in both cases has the same sources - I've verified this using the leaks tool output from both groups of machines. This high memory usage causes customers' machines to slow down considerably and affect productivity.
Configuration
- .NET Core version: 3.1.300
- OS version: macOS Catalina 10.15.5
- Architecture: x64
- Configuration-specific: No, the memory leak has been seen on macOS 10.14 Catalina as well, with .NET Core versions 3.0, 3.1.1, and 3.1.2.
Regression?
Not sure - we have seen this issue previously on .NET Core 3.0, but there was only one reported incident. It looks like the leak occurs even if the total memory usage doesn't grow noticeably, so it's possible that this just wasn't noticed in previous releases of our app.
Other information
The leak report for the min-reproducible example can be found in the project README. I'm including the sections responsible for the most memory usage (found in the leak reports from other affected systems) below. There are two main sections:
SSL handshake attempt
STACK OF 1 INSTANCE OF 'ROOT LEAK: <SecTrust>':
42 libsystem_pthread.dylib 0x7fff6b1cbb8b thread_start + 15
41 libsystem_pthread.dylib 0x7fff6b1d0109 _pthread_start + 148
40 libcoreclr.dylib 0x1010c12a4 CorUnix::CPalThread::ThreadEntry(void*) + 436
39 libcoreclr.dylib 0x10126db8f ThreadpoolMgr::WorkerThreadStart(void*) + 1311
38 libcoreclr.dylib 0x101240154 ManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 276
37 libcoreclr.dylib 0x101249b20 ManagedThreadBase::ThreadPool(void (*)(void*), void*) + 32
36 libcoreclr.dylib 0x101249503 ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 323
35 libcoreclr.dylib 0x1012a3538 QueueUserWorkItemManagedCallback(void*) + 184
34 libcoreclr.dylib 0x101288639 MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 1657
33 libcoreclr.dylib 0x10143c8fb CallDescrWorkerInternal + 124
32 ??? 0x109384766 0x7fffffffffffffff + 9223372041304426343
31 ??? 0x10924ed8d 0x7fffffffffffffff + 9223372041303158158
30 ??? 0x109ee4798 0x7fffffffffffffff + 9223372041316353945
29 ??? 0x109ee629a 0x7fffffffffffffff + 9223372041316360859
28 ??? 0x10937930d 0x7fffffffffffffff + 9223372041304380174
27 ??? 0x10938423e 0x7fffffffffffffff + 9223372041304425023
26 ??? 0x108f57eed 0x7fffffffffffffff + 9223372041300049646
25 ??? 0x107ef0400 0x7fffffffffffffff + 9223372041282847745
24 ??? 0x109397586 0x7fffffffffffffff + 9223372041304503687
23 ??? 0x10939905c 0x7fffffffffffffff + 9223372041304510557
22 ??? 0x1093992bb 0x7fffffffffffffff + 9223372041304511164
21 ??? 0x107ef0400 0x7fffffffffffffff + 9223372041282847745
20 ??? 0x109397586 0x7fffffffffffffff + 9223372041304503687
19 ??? 0x10939905c 0x7fffffffffffffff + 9223372041304510557
18 ??? 0x1093992bb 0x7fffffffffffffff + 9223372041304511164
17 ??? 0x107ef0400 0x7fffffffffffffff + 9223372041282847745
16 ??? 0x109397452 0x7fffffffffffffff + 9223372041304503379
15 ??? 0x109397631 0x7fffffffffffffff + 9223372041304503858
14 ??? 0x109397af5 0x7fffffffffffffff + 9223372041304505078
13 ??? 0x10939894e 0x7fffffffffffffff + 9223372041304508751
12 ??? 0x109398ac2 0x7fffffffffffffff + 9223372041304509123
11 ??? 0x10934c526 0x7fffffffffffffff + 9223372041304196391
10 System.Security.Cryptography.Native.Apple.dylib 0x101e816ee 0x101e7d000 + 18158
9 com.apple.security 0x7fff3da8c909 SSLHandshake + 185
8 com.apple.security 0x7fff3da8ca29 SSLHandshakeProceed + 185
7 libcoretls.dylib 0x7fff68609999 tls_handshake_process + 85
6 libcoretls.dylib 0x7fff6860a0eb SSLProcessHandshakeRecordInner + 219
5 com.apple.security 0x7fff3dcacdac tls_verify_peer_cert + 71
4 com.apple.security 0x7fff3dcaccff sslCreateSecTrust + 47
3 libcoretls_cfhelpers.dylib 0x7fff6861c23e tls_helper_create_peer_trust + 222
2 com.apple.security 0x7fff3da5ff43 SecTrustCreateWithCertificates + 918
1 com.apple.CoreFoundation 0x7fff310e9663 _CFRuntimeCreateInstance + 597
0 libsystem_malloc.dylib 0x7fff6b181d9e malloc_zone_malloc + 140
and thread start code.
STACK OF 10 INSTANCES OF 'ROOT LEAK: malloc<144>':
7 libsystem_pthread.dylib 0x7fff6b1cbb8b thread_start + 15
6 libsystem_pthread.dylib 0x7fff6b1d0109 _pthread_start + 148
5 libcoreclr.dylib 0x1010c12a4 CorUnix::CPalThread::ThreadEntry(void*) + 436
4 libcoreclr.dylib 0x10126fcc6 ThreadpoolMgr::GateThreadStart(void*) + 118
3 libcoreclr.dylib 0x1012e53a6 EETlsSetValue(unsigned int, void*) + 22
2 libcoreclr.dylib 0x1011938cd CExecutionEngine::CheckThreadState(unsigned int, int) + 61
1 libcoreclr.dylib 0x10109a698 HeapAlloc + 40
0 libsystem_malloc.dylib 0x7fff6b181d9e malloc_zone_malloc + 140
I have opened https://github.com/aws/aws-sdk-net/issues/1629 on the GitHub repo for the AWS SDK for .NET. However, because these two stack traces seem to point to .NET SDK native code, I have opened this issue directly with the .NET repo as well.
rharpavat
All 36 comments
I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.
Dotnet-GitSync-Bot
on 2 Jun 2020
Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.
![msftbot[bot] picture](https://avatars2.githubusercontent.com/in/26612?v=4&s=40)
msftbot[bot]
on 2 Jun 2020
After some more investigation we were able to trace the issue back to the AmazonServiceClient class that all the individual classes (AmazonKinesisClient, etc.) inherit from. The AmazonServiceClient uses the HttpWebRequest class to send data to AWS. However, looking at the docs for the class, Microsoft does not recommend using it for development, instead recommending the HttpClient class.
I was able to reproduce the memory leak with a sample program that just created a new HttpWebRequest every 5 seconds to http://www.contoso.com/ and got the response. However, when making this same request using a single HttpClient, I wasn't able to see the memory leak.
rharpavat
on 4 Jun 2020
We investigated further and discovered that the AmazonServiceClient uses the recommended HttpClient class in the .NET Core implementation. Looking at the stack trace from the test program, we see the line:
10 System.Security.Cryptography.Native.Apple.dylib 0x101e816ee 0x101e7d000 + 18158
in the leak report. Can we investigate if this library has a bug that might be causing this memory leak? We are not seeing this leak on Ubuntu, so I think it might be in macOS-specific code.
Also, are y'all able to reproduce the memory leak on a test machine with the sample program?
rharpavat
on 15 Jun 2020
I will try to reproduce this soon. Thanks for for the steps on how to reproduce and sample repo.
vcsjones
on 16 Jun 2020
I took a look at the repro. Just to make sure, I set up a Kinesis stream in AWS with valid IAM credentials that could PUT to it.
One thing that I noticed in the repro is that there is a task that is not being awaited here.
So the SendRecord was essentially a fire-and-forget, but the Thread.Sleep prevented it from growing very rapidly.
I fixed up the example to use async / await and I was able to publish a few thousand messages to Kinesis, and memory usage never exceeded 30 MB on macOS 10.15. You can see the exact code I ran here (minus the IAM credentials of course).
I'm not familiar enough with async / await to know if a missing await (fire and forget) could leak memory in circumstances.
vcsjones
on 17 Jun 2020
.NET Core versions 3.0, 3.1.1, and 3.1.2.
Also, I was testing with .NET Core 3.1.5. If you can still reproduce the memory leak with my suggested changes, it would be good to see if it reproduced on the latest version of 3.1.
vcsjones
on 17 Jun 2020
@vcsjones Thanks for the quick response, I really appreciate it! I made the changes you suggested and updated to the latest 3.1.5 release of .NET Core (changes pushed to the GitHub repo too). Note that the memory leak can be reproduced by running the program as-is (i.e. with invalid AWS credentials or stream name) - even trying to make that connection seems to cause the leak.
Running the altered program causes memory usage to shoot up to ~70MB within the first 5 seconds (probably because we took out the Thread.Sleep() call) and then grows by 0.2-0.3MB/s (monitored via Activity Monitor with a refresh rate of 1s). The leak report looks the same as described in my sample GitHub repo, just with more leaked objects of the SecTrust type.
To provide some context here, we see the memory leak across all of our test systems, but it only grows massively (to the order of GB) on some of those machines. Unfortunately we haven't been able to pinpoint what's different about those particular machines that's causing this leak to grow so reliably. The software we're using this code in is part of our goal to deploy to a fleet of 80k client macOS systems, and this memory leak has delayed that goal by 3 months. These are all user machines, so we can't proceed without resolving the memory leak since GBs of memory being used by our process causes a bad customer experience.
Again, I really appreciate all your help in troubleshooting this issue - please let me know if there's any help I can provide. Thanks!
rharpavat
on 19 Jun 2020
I actually just ran another test - looks like the memory leak doesn't happen when the records are successfully sent. i.e. I ran the test program with valid AWS credentials/stream name, and the memory usage stayed <30MB. The leak report file also didn't report the SecTrust type leaks.
Could you see if providing invalid credentials to the program makes the leak happen on your machine as well?
rharpavat
on 19 Jun 2020
Could you see if providing invalid credentials to the program makes the leak happen on your machine as well?
Ah, I didn't try that! I will give that a try soon. Thank you for the further investigation.
vcsjones
on 19 Jun 2020
@vcsjones Did you get a chance to try and reproduce the issue? I think it happens when the request fails for any reason.
Also, what tools do you guys use when troubleshooting .NET Core issues on macOS? Seems like some of the dotnet trace tools don't support macOS.
rharpavat
on 24 Jun 2020
I _think_ so? If it is it is slow for me, I'm working on trying to get a managed debugger and sos working on my mac again.
Will your program eventually crash and OOM? Or is memory usage just unusually high?
vcsjones
on 24 Jun 2020
Hmm I don't think I've tested that scenario, all the users experiencing the issue killed the app after a while because it was noticeably slowing their systems down. Max memory usage we had reported was 4.7GB.
rharpavat
on 25 Jun 2020
@vcsjones Were you able to find a potential source for this memory leak? Just to clarify, providing invalid credentials is only one way of causing the leak - it seems like any failure causes the leak to happen. The error we're actually seeing the most often is:
2020-05-20 02:03:26.9177 ERROR KinesisStream client exception after 2 attempts: System.Net.Http.HttpRequestException: nodename nor servname provided, or not known
@stacktrace_id 2730 257189 814922068
---> System.Net.Sockets.SocketException (0xFFFDFFFF): nodename nor servname provided, or not known
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
This may be also related to #34080. On that issue, memory also seems to be leaking when using attached C code e.g. without .NET involved. I started looking at it but I got interrupted with some other work.
wfurt
on 20 Aug 2020
few notes: Who the request fails with 'nodename nor servname provided' than SSL is not even tried - so there is no way how we would leak in System.Security.Cryptography.Native.Apple.dylib (note this can be cause by known bugs in systemd or transient lookup failures)
I modified repro to do 111 request, dispose the IAmazonKinesis and wait for input.
Interestingly that does not dispose underlying HttpClients and that leaves ConnectionPool and that keeps SslStreams and that keeps references to various crypto objects.
few examples here from gcroot
00007000033D1DF0 0000000109C24B82 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 794]
rbx:
-> 0000000181C82318 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[Amazon.Runtime.Internal.Transform.IWebResponseData, AWSSDK.Core],[Amazon.Runtime.HttpWebRequestMessage+<GetResponseAsync>d__20, AWSSDK.Core]]
-> 0000000181C8A538 System.Threading.Tasks.Task+ContingentProperties
-> 0000000181C8A588 System.Threading.Tasks.TaskExceptionHolder
-> 0000000181C8A5B8 System.Collections.Generic.List`1[[System.Runtime.ExceptionServices.ExceptionDispatchInfo, System.Private.CoreLib]]
-> 0000000181C8A5D8 System.Runtime.ExceptionServices.ExceptionDispatchInfo[]
-> 0000000181C8A5F8 System.Runtime.ExceptionServices.ExceptionDispatchInfo
-> 0000000181C8A4B0 Amazon.Runtime.Internal.HttpErrorResponseException
-> 0000000181C89FA0 Amazon.Runtime.Internal.Transform.HttpClientResponseData
-> 0000000181C89FE8 Amazon.Runtime.Internal.Transform.HttpResponseMessageBody
-> 0000000181C045B8 System.Net.Http.HttpClient
00007000033D1F90 0000000109C23D5D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 201]
r15:
-> 0000000181C82270 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Net.Http.HttpResponseMessage, System.Net.Http],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]]
-> 0000000181C045B8 System.Net.Http.HttpClient
00007000033D1FE0 000000010A8430A6 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 366]
rbx:
-> 0000000181C82270 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Net.Http.HttpResponseMessage, System.Net.Http],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]]
-> 0000000181C045B8 System.Net.Http.HttpClient
00007000033D1FE0 000000010A8430A6 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]].MoveNext(System.Threading.Thread) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs @ 366]
rbp-20: 00007000033d1ff0
-> 0000000181C82270 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Net.Http.HttpResponseMessage, System.Net.Http],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]]
-> 0000000181C045B8 System.Net.Http.HttpClient
00007000033D2020 0000000109C24B82 System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 794]
rbx:
-> 0000000181C82270 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Net.Http.HttpResponseMessage, System.Net.Http],[System.Net.Http.HttpClient+<FinishSendAsyncUnbuffered>d__77, System.Net.Http]]
-> 0000000181C045B8 System.Net.Http.HttpClient
HandleTable:
0000000101901230 (strong handle)
-> 0000000181C8AEE0 Amazon.Kinesis.AmazonKinesisException
-> 0000000181C8A4B0 Amazon.Runtime.Internal.HttpErrorResponseException
-> 0000000181C89FA0 Amazon.Runtime.Internal.Transform.HttpClientResponseData
-> 0000000181C89FE8 Amazon.Runtime.Internal.Transform.HttpResponseMessageBody
-> 0000000181C045B8 System.Net.Http.HttpClient
0000000101901258 (strong handle)
-> 0000000181C8AEE0 Amazon.Kinesis.AmazonKinesisException
-> 0000000181C8A4B0 Amazon.Runtime.Internal.HttpErrorResponseException
-> 0000000181C89FA0 Amazon.Runtime.Internal.Transform.HttpClientResponseData
-> 0000000181C89FE8 Amazon.Runtime.Internal.Transform.HttpResponseMessageBody
-> 0000000181C045B8 System.Net.Http.HttpClient
I don't know for sure that causes the leak reported by the 'leaks' utility but it is difficult to make progress with all the objects hanging around.
I also added
c#
GC.Collect();
GC.WaitForPendingFinalizers();
to the loop but that won't help with live objects.
Is there some better way how to force proper cleanup @rharpavat?
wfurt
on 1 Sep 2020
@wfurt By default, the AWS .NET SDK uses a global cache of HttpClient instances: https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/Amazon.Runtime/Pipeline/HttpHandler/_netstandard/HttpRequestMessageFactory.cs#L219 . These instances are assigned to the client objects (i.e. IAmazonKinesis) every time a request is made, and is re-added to the pool after the request is done.
There are a couple of ways to manually create and destroy HttpClient instances without using the global cache. One way is as following:
// create my own HttpClientFactory
class MyHttpClientFactory : HttpClientFactory
{
public override HttpClient CreateHttpClient(IClientConfig clientConfig)
{
return new HttpClient();
}
// return true here will make the client dispose the HttpClient instance after EVERY request.
public override bool DisposeHttpClientsAfterUse(IClientConfig clientConfig)
{
return true;
}
}
var client = new AmazonKinesisClient(new AmazonKinesisConfig
{
CacheHttpClient = false, // this will tell AmazonKinesisClient to NOT use the global cache
HttpClientFactory = new MyHttpClientFactory()
});
Please let me know if this isn't what you're looking for.
dhhoang
on 1 Sep 2020
@wfurt By default, the AWS .NET SDK uses a global cache of
HttpClientinstances: https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Core/Amazon.Runtime/Pipeline/HttpHandler/_netstandard/HttpRequestMessageFactory.cs#L219 . These instances are assigned to the client objects (i.e.IAmazonKinesis) every time a request is made, and is re-added to the pool after the request is done.There are a couple of ways to manually create and destroy
HttpClientinstances without using the global cache. One way is as following:// create my own HttpClientFactory class MyHttpClientFactory : HttpClientFactory { public override HttpClient CreateHttpClient(IClientConfig clientConfig) { return new HttpClient(); } // return true here will make the client dispose the HttpClient instance after EVERY request. public override bool DisposeHttpClientsAfterUse(IClientConfig clientConfig) { return true; } } var client = new AmazonKinesisClient(new AmazonKinesisConfig { CacheHttpClient = false, // this will tell AmazonKinesisClient to NOT use the global cache HttpClientFactory = new MyHttpClientFactory() });Please let me know if this isn't what you're looking for.
Also, HttpClientFactory class in the code above is Amazon.Runtime.HttpClientFactory, not the .NET one.
dhhoang
on 1 Sep 2020
yes, essentially this is what I was looking for.
With that and the #41657 fix I can get clean run:
Shining:sslLeak furt$ leaks 70229
Process: kinesis_dotnet_macos_memoryleak [70229]
Path: /Users/USER/*/kinesis_dotnet_macos_memoryleak
Load Address: 0x1026ad000
Identifier: kinesis_dotnet_macos_memoryleak
Version: ???
Code Type: X86-64
Parent Process: bash [1033]
Date/Time: 2020-09-01 11:46:29.280 -0700
Launch Time: 2020-09-01 11:46:09.410 -0700
OS Version: Mac OS X 10.15.5 (19F101)
Report Version: 7
Analysis Tool: /Applications/Xcode.app/Contents/Developer/usr/bin/leaks
Analysis Tool Version: Xcode 11.5 (11E608c)
Physical footprint: 22.4M
Physical footprint (peak): 22.4M
----
leaks Report Version: 4.0
Process 70229: 20233 nodes malloced for 4402 KB
Process 70229: 0 leaks for 0 total leaked bytes.
yes, essentially this is what I was looking for.
With that and the #41657 fix I can get clean run:Shining:sslLeak furt$ leaks 70229 Process: kinesis_dotnet_macos_memoryleak [70229] Path: /Users/USER/*/kinesis_dotnet_macos_memoryleak Load Address: 0x1026ad000 Identifier: kinesis_dotnet_macos_memoryleak Version: ??? Code Type: X86-64 Parent Process: bash [1033] Date/Time: 2020-09-01 11:46:29.280 -0700 Launch Time: 2020-09-01 11:46:09.410 -0700 OS Version: Mac OS X 10.15.5 (19F101) Report Version: 7 Analysis Tool: /Applications/Xcode.app/Contents/Developer/usr/bin/leaks Analysis Tool Version: Xcode 11.5 (11E608c) Physical footprint: 22.4M Physical footprint (peak): 22.4M ---- leaks Report Version: 4.0 Process 70229: 20233 nodes malloced for 4402 KB Process 70229: 0 leaks for 0 total leaked bytes.
I'm still wondering why leak would report leak when we don't dispose (but still hold references to) HttpClient objects. Isn't HttpClient supposed to be reused? We can't simply create new instance for each request because that would be really bad for performance.
dhhoang
on 1 Sep 2020
I'm not sure it would. I just wanted to make sure I have as little objects as possible.
wfurt
on 1 Sep 2020
FYI: This is now fixed in 6.0 in PR #41657 (it was merged on 9/1). We are discussing options to backport it into 5.0.
karelz
on 8 Sep 2020
could you verify the fix @dhhoang with daily builds https://github.com/dotnet/installer ???
This will be needed for possible back-port.
wfurt
on 10 Sep 2020
@wfurt Is the fix in the daily build for 6.0.x or a previous version? If possible, an updated 3.1.x build with the fix would be ideal as that would help us validate the fix in our actual integ environment.
Thanks for the quick turnaround on this issue!
rharpavat
on 11 Sep 2020
it is in daily build only for now. To get permission for back port, we were asked for validation. I _could_ build and share private bits (or provide instructions how to do). Essentially, the fix is in one shared library which needs be replaced.
wfurt
on 11 Sep 2020
hi wfurt@,
It will be great if you can provide private bits for 3.1. It makes validation easy for us otherwise we have to build code against 6.0 and there may be some other compilation or compatibility issues. Generally migration to a newer version is a bigger task and we plan it 1-2 sprint task.
Thanks for your great support on this issue.
Regards,
Harvir
singhharvir
on 11 Sep 2020
@wfurt for comment above
danmosemsft
on 14 Sep 2020
You can grab runtime bits here . That is CI build from proposed change. Depending how your app is structured, you can either use it directly to run your assembly or you can grab System.Security.Cryptography.Native.Apple.dylib and replace it in your normal distribution.
wfurt
on 15 Sep 2020
@singhharvir will you be able to validate by Friday 9/18?
karelz
on 15 Sep 2020
any luck with verification @singhharvir @rharpavat ?
wfurt
on 18 Sep 2020
Sorry for the late response. Yes, we have planned to validate it on 9/18.
singhharvir
on 18 Sep 2020
thanks. let us know.
wfurt
on 18 Sep 2020
@wfurt I replaced the existing .NET Core 3.1 installation with the provided bits and ran the min-repro program. After running for ~2 hrs, not seeing the memory leak anymore. Will keep this running till tomorrow and validate that it fully resolves the issue. Thanks for the quick turnaround, really appreciate it!
rharpavat
on 19 Sep 2020
good to know. thanks.
wfurt
on 19 Sep 2020
Fixed in:
- 3.1.9 in PR https://github.com/dotnet/corefx/pull/42985
- 5.0 (RC2) in PR #41989
- 6.0 in PR #41657
karelz
on 21 Sep 2020
fixed for 5.0 with #41989
wfurt
on 21 Sep 2020
Related issues
yahorsi
路
3Comments
v0l
路
3Comments
matty-hall
路
3Comments
noahfalk
路
3Comments
omariom
路
3Comments
Most helpful comment
fixed for 5.0 with #41989