Runtime: System.Diagnostics.PerformanceCounter leaks memory

[FeodorFitsner]

Our .NET Core app uses System.Diagnostics.PerformanceCounter class to poll Hyper-V VM's statistics.
The typical usage pattern is shown below. Counters are being polled in a loop every second until cancelled. The code is shown for 1 counter, but there are 10 counters polled in the same loop.

Everything was fine in .NET Core 2.2 until the app was migrated to 3.0 RTM and the code below started leaking "unmanaged" memory.
<ServerGarbageCollection>false</ServerGarbageCollection> is set during the publishing.

private async Task PollPerformanceCounters(CancellationToken cancellationToken)
{
    var _workerName = "vm-1";
    PerformanceCounter nicBytesReceivedCounter = null;

    // here are 10 more counters like above
    // ...

    try
    {
        var nicCategoryName = "Hyper-V Virtual Network Adapter";
        var nicInstanceName = PerformanceCounterCategory.GetCategories()
            .Single(category => category.CategoryName == nicCategoryName)
            .GetInstanceNames().FirstOrDefault(i => i.StartsWith(_workerName));

        nicBytesReceivedCounter = new PerformanceCounter(nicCategoryName, "Bytes received/sec", nicInstanceName);

        while (!cancellationToken.IsCancellationRequested)
        {
            // query NIC performance counters
            var bytesReceivedSec = Convert.ToInt32(nicBytesReceivedCounter.NextValue());

            // Save counter in the database
            SaveMetadata();

            await Task.Delay(1000);
        }
    }
    finally
    {
        // dispose counters
        if (nicBytesReceivedCounter != null)
        {
            nicBytesReceivedCounter.Dispose();
        }
    }
}

Are we doing something wrong here (like wrong pattern or something is not called additionally to free resources) or there is a regression/bug in .NET 3.0?

[danmosemsft]

@buyaa-n could you please look at this today? @Anipik is not back until next week, and it would be good to figure it out this week, in case we need to put it in 3.1.

[buyaa-n]

@FeodorFitsner thank you for reporting this. I have tried to reproduce the issue with several version of similar code you provided but don't see a memory leak. Also compared with 2.0 version memory usage, nothing suspicious. Could you please provide more info for repro, sample project would be very helpful or some more info about how did you found the leak, how are you running the task above etc

[FeodorFitsner]

Thanks for looking into that! I'm working on the repro right now.

[ericstj]

@FeodorFitsner do you have an update? Our time window is quickly closing for getting a fix in 3.x. Looking at your sample code I see two potential sources of a leak: the call to NextValue, or whatever you are doing in SaveMetadata(). If you can get a repro of the leak in very basic console app it'd help isolate it.

@buyaa-n I do see a change that's somewhat suspicious of changing memory usage characteristics: https://github.com/dotnet/corefx/commit/c4f331961bfc6317a16a6ec43b53c780bd3c8dc4 /cc @MaxShoshin
I wonder if you call NextValue() in a loop if you can repro it.

[FeodorFitsner]

Yes, I've found the source of memory leak. So, these are memory/CPU Hyper-V counters:

// CPU
var cpuCategoryName = "Hyper-V Hypervisor Virtual Processor";
var cpuGuestRunTimeCounters = PerformanceCounterCategory.GetCategories()
    .Single(category => category.CategoryName == cpuCategoryName)
    .GetInstanceNames().Where(i => i.StartsWith(_workerName))
    .Select(name => new PerformanceCounter(cpuCategoryName, "% Guest Run Time", name)).ToList();

// Memory
var memoryGuestVisiblePhysicalCounter = new PerformanceCounter("Hyper-V Dynamic Memory VM", "Guest Visible Physical Memory", _workerName);
var memoryCurrentPressureCounter = new PerformanceCounter("Hyper-V Dynamic Memory VM", "Current Pressure", _workerName);

// query memory performance counters
var memoryGuestVisibleMbytes = Convert.ToInt32(memoryGuestVisiblePhysicalCounter.NextValue());
var memoryCurrentPressure = Convert.ToInt32(memoryCurrentPressureCounter.NextValue());

// CPU
var cpuGuestRuntime = Convert.ToInt32(cpuGuestRunTimeCounters.Select(c => c.NextValue()).Average());

_workerName in the code above is the name of Hyper-V VM.

I don't know if that helps you guys to find an issue in .NET Core logic or maybe it's Hyper-V specific problem, but in our app commenting that piece makes huge difference. Otherwise, with those counters queried memory is leaking like 500 MB per day. The interesting thing I tried doing a .NET Core console app (and I still have it, can share), but looks like it doesn't surface the problem.

Let me know if you need anything to further investigate the issue. Thank you!

[buyaa-n]

@ericstj I did call NextValue() within a loop, but not sure if that loop is same as you suggesting. My machine don't have "Hyper-V" counters so I just used Lan/Ethernet counter for testing as in the original issue. But didn't run with a memory counter.
@FeodorFitsner is your console app is using those "Hyper-V" counters same as in your other app?

[FeodorFitsner]

Yes, these particular Hyper-V counters is the issue. Other counters don't leak. Please find attached a console app I tried using to reproduce the issue.
PerfCountersWmi.zip

[buyaa-n]

Thank you @FeodorFitsner it would be helpful, by my understanding memory leak is not happening with the console app but happening with your other app right?

[FeodorFitsner]

It looks like, yes.

[danmosemsft]

@buyaa-n it is very likely a bug in the HyperV counter implementation. Do you have Hyperv installed? If not, try installing it (it’s part of windows) and create a VM . See whether you can get the leak to happen.

[FeodorFitsner]

Moreover, I suspect it leaks on Windows Server 2019 (1809) specifically. We run the same code on Windows Server 2016 (though built against .NET Core 2.2) - no leaks so far.

[danmosemsft]

@FeodorFitsner does it leak using 2.2 on 2019? I assume so, and this is a bug introduced in windows 2019, right?

If you have a support contract with Microsoft that would be the most effective way to get a fix for it. Otherwise we can help file a bug internally.

[buyaa-n]

@danmosemsft yes I have enabled Hyper-V and installed win10 instance, and running the sample provided by @FeodorFitsner since around 6pm, just checked so far no leak. From sample I was not able to create Network adapter counters because somehow no instance found, but was able to create memory and CPU counters for that Hyper-V VM, instead Network adapter loading more memory counters as he mentioned those counters are leaking

[buyaa-n]

My repro efforts with given sample not showing trace of leak, @FeodorFitsner is it possible to share with us the memory dump you are seeing the leak? Also not sure if you are already tried this: forcing GC completely after disposing counters might give better result

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

[FeodorFitsner]

Do you have instructions how to collect the dump and where to securely submit it? Will try forcing GC in the meantime.

[marklio]

Do you have instructions how to collect the dump and where to securely submit it? Will try forcing GC in the meantime.

Expect instructions for collecting the dump shortly. Can you share your security concerns? Who would you like to have access to look at your dump? (this might affect how we suggest to submit it) The easiest thing would be to create a repro that is free of security concern on your end, and you can easily share that via OneDrive or other mechanism. If your only repro contains sensitive information, we will need to go through different (more restrictive and difficult to deal with) channels to handle it appropriately.

[marklio]

As for capturing a dump, it sounds like you've got a process that's leaking memory. The easiest thing to do in that case is just look at the process in the details view of the task manager, right-click, and select create dump file. It will create the dump file, and tell you the path that it is saved. (NOTE: If you are running x86 on a 64-bit machine, you need to run the 32-bit taskmgr.exe.) For other info on collecting dumps, here's a pretty good article that @buyaa-n pointed me at: https://michaelscodingspot.com/how-to-create-use-and-debug-net-application-crash-dumps-in-2019/

[FeodorFitsner]

@marklio Would be OK if I wrap a dump into zip with password, attach zip here and then send password to your email (on your GitHub profile)?

[marklio]

@marklio Would be OK if I wrap a dump into zip with password, attach zip here and then send password to your email (on your GitHub profile)?

Again, it comes down to what your security and/or privacy concerns are. What matters to us is that your concerns are addressed.

We have a GDPR-compliant file attach scenario built into the problem reporter here: https://developercommunity.visualstudio.com/content/problem/post.html?space=61

First, create and submit a report (without attaching). You could copy/paste the details from this bug, or just be brief. Then, create a comment and mark it as private, and attach the file as part of that comment. Then, you can comment here with a link to the issue. We can get the file from the feedback item and move forward with this bug.

Password protected zips aren't terribly secure (and even if they were today, anyone could grab it and wait until someone cracked them). If you think you have security or privacy sensitive data in your dump, you should let us know so we can treat it properly. If not, then it doesn't matter how you provide the dump.

[marklio]

OK, I got it straightened out. Editing the comment above with good instructions.

[FeodorFitsner]

Thanks for the instructions! We are deploying leaking code now to a controlled environment and I'll share the dump tomorrow.

[FeodorFitsner]

@marklio I've created the report with dump attached in a "Microsoft only" comment as you suggested: https://developercommunity.visualstudio.com/content/problem/803951/systemdiagnosticsperformancecounter-leaks-memory.html

Thank you for looking into that! I'm eager to know about your findings. Maybe it's something we could/should fix on our side.

[marklio]

Great. I can see it. @buyaa-n , let me know if you have trouble seeing the dump.

[buyaa-n]

Yes I see the dump, thank you @marklio @felipepessoto!

[buyaa-n]

Hello @FeodorFitsner, is your issue solved by any chance? If not, would you mind leave a memory dump with some GFlags option set as mentioned in the VS developer community issue