Powershell: [Bug in PS 5,6 and 7] Memory leak when Select-Object on $Global scope array of PSCustomObject

[RainbowMiner]

Steps to reproduce

function Initialize-Something {
    $Global:TestArray = @()
    $Global:TestArray += [PSCustomObject]@{abc=1;def=2}
    $Global:TestArray += [PSCustomObject]@{abc=2;def=3}
    $Global:TestArray += [PSCustomObject]@{abc=3;def=4}
}

function Invoke-Something {
    for($i=0; $i -lt 1000; $i++) {
        $Global:TestArray | Select-Object > $null
    }
}

Initialize-Something
Invoke-Something

Expected behavior

Garbage collection should clear heap afterwards

Actual behavior

Every call to Invoke-Something collects more objects of the following type in managed memory:

{System.Collections.Concurrent.ConcurrentDictionary<string, System.Management.Automation.PSMemberInfoInternalCollection<System.Management.Automation.PSMemberInfo>>.Node}

Most notably, the "_key" contains the following.

"Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@Selected.System.Management.Automation.PSCustomObject@@@System.Management.Automation.PSCustomObject@@@System.Object"

After each run of Invoke-Something, every newly created object of the above type seems to inherit this list, but increased by one more Selected.System.Management.Automation.PSCustomObject. This creates an exponential growth, over time.

Environment data

Name                           Value
----                           -----
PSVersion                      6.1.0
PSEdition                      Core
GitCommitId                    6.1.0
OS                             Microsoft Windows 10.0.18362
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

.. but the same happens with pwsh core 6.2.3, as well. Same thing happens, when running pwsh on a linux platform.

[iSazonov]

Can you repo with latest PowerShell 7 build?

[RainbowMiner]

Good question, I don't have a pwsh 7 install, yet. I will try this first thing, tomorrow morning.

[RainbowMiner]

One additional remark: the leak also happens, when using -first, -last, -index etc. on Select-Object, but does not happen, when using -Property or -ExpandProperty. This might be, because the latter two create new objects.

[RainbowMiner]

Can you repo with latest PowerShell 7 build?

Yes, same thing happens with PowerShell 7, too.

[iSazonov]

Perhaps it is related #7768

[RainbowMiner]

It's a pretty severe bug which did cost me many hours of searching. I had never suspected such a basic function like select-object to cause this much trouble. If you have a long running script, that does select-object on a global PSCustomObject array e.g. every 60 seconds, it will fill up all your machine's ram within 24 hours.

Regarding issue #7768, I stumbled upon this issue during my search for the memory leak. Actually, that one addresses the use of -ExpandProperty and -Property - if using select-object with one or both of those parameters, the memory leak does not happen. Most probably, because a new object will be forwarded through the pipeline.

It's happening only, when using blank select-object or positional select-object, which should simply forward a selection of the original objects

[iSazonov]

I can not repo on PS 7.0. GC works great.
I tried:
```powershell
1.. 1000 | % { Invoke-Something }

and

1.. 1000 | % { Initialize-Something; Invoke-Something }

[RainbowMiner]

For me, it doesn't work out.

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.0.0-rc.1
PSEdition                      Core
GitCommitId                    7.0.0-rc.1
OS                             Microsoft Windows 10.0.18362
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Before:

MemUsage  MemDiff MemText
--------  ------- -------
22071784 22071784 Memory usage: 21,0 MB (22.071.784 Bytes [phone])

Now I run this code:

1..20 | %{Invoke-Something}

After:

MemUsage  MemDiff MemText
--------  ------- -------
37715576 ‭15643792‬ Memory usage: 36,0 MB (37.715.576 Bytes +‭15643792‬)

Now I run this code again:

1..20 | %{Invoke-Something}

MemUsage  MemDiff MemText
--------  ------- -------
54916776 ‭17201200‬ Memory usage: 52,4 MB (54.916.776 Bytes +‭17201200‬)

I am using the following function to get the memory usage. There is no change to the above values, if I would call [System.GC]::Collect() one or multiple times.

function Get-MemoryUsage
{
    $memusagebyte = [System.GC]::GetTotalMemory($true)
    $memdiff = $memusagebyte - [int64]$Global:last_memory_usage_byte
    [PSCustomObject]@{
        MemUsage   = $memusagebyte
        MemDiff    = $memdiff
        MemText    = "Memory usage: {0:n1} MB ({1:n0} Bytes {2})" -f  ($memusagebyte/1MB), $memusagebyte, "$(if ($memdiff -gt 0){"+"})$($memdiff)"
    }
    $Global:last_memory_usage_byte = $memusagebyte
}

[iSazonov]

What is $Global:last_memory_usage_byte in your script? I don't see that the variable is assigned.

[RainbowMiner]

Ah, that's being set to 0 at the start and set within the above function. I had stripped some code from the Get-MemoryUsage function, before posting it here. I unintentionally did cut out the reset line (I just re-added it)

I did another 100 runs:

1..100 | %{Invoke-Something}

Now the memory is here:

 MemUsage   MemDiff MemText
 --------   ------- -------
145399880  90483104 Memory usage: 138,7 MB (145.399.880 Bytes [phone])

[RainbowMiner]

The memory dump on Pwsh 7 is a bit different, than that I aquired from the Pwsh 6 dumps:

Tons of strings of type System.Collections.Generic.List<string>

Tons of strings of type System.Management.Automation.Runspaces.ConsolidatedString

So, basically a similar result, compared to PS6, but in a different order and objects. But it all comes down to a giant collection of strings with the value "Selected.System.Management.Automation.PSCustomObject"

[iSazonov]

PowerShell is a script engine and it does many allocations by design.
Your script creates global varable and not clear/remove it - as result memory is not freed. I can not confirm a memory leak.

If you see a memory leak, please make a simple repo on PowerShell 7.0 latest build so that we can reproduce and measure.

[doctordns]

@RainbowMiner - have you tried

Initialize-Something
Invoke-Something
Remove-Item -Path Variable:testarray*

then take a look at memory usage.

[lzybkr]

I can reproduce the problem and it feels like a bit of a corner case but maybe has an easy fix.

This function:

https://github.com/PowerShell/PowerShell/blob/59db1f619edb1bd85a784c0f019a8bf75574bd84/src/Microsoft.PowerShell.Commands.Utility/commands/utility/Select-Object.cs#L657

probably needs to be changed.

It doesn't seem useful to add the Selected. type to pass-thru objects like this (so you could add a test that psoObj is not InputObject, but that might be insufficient, e.g. I also don't think it makes sense to add Selected. if the type is already Selected.. Maybe the type prefix should not be added when InputObject.BaseObject is a PSCustomObject.

[iSazonov]

@lzybkr Thanks for your investigating!

Maybe the type prefix should not be added when InputObject.BaseObject is a PSCustomObject.

Why do we add the prefix at all exclusively to PSCustomObject-s?

[lzybkr]

@iSazonov - if the cmdlet is passing objects through unmodified - it doesn't make sense to add the prefix, so I think the condition was written with that intent as it creates custom objects from the selected properties.

[iSazonov]

@lzybkr It is not clear why we add the prefix at all. Using Select-Object implies that this cmdlet always creates custom objects (except in edge bypass case). Also adding the type prefix is not documented. So I think we could remove the feature.

[lzybkr]

@iSazonov I don't know the history but removing the prefix would be a breaking change and it might be hard to search for (125000 hits on the word Selected in GitHub - most probably not related but some might be.)

[iSazonov]

Yes, formally it is a breaking change. I found some scripts using the feature https://github.com/search?q=Select-Object+Selected+pstypenames&type=Code - there are only a few.

I think PowerShell Committee can already make a conclusion. /cc @SteveL-MSFT

My arguments for removing the feature:

• it is not documented
• it is a breaking change in grace area
• it is bad design because this cmdlet always creates custom objects
• users can use a workaround manually adding a custom type or a custom property to mark objects

[bpayette]

@iSazonov

it's not documented

Not specifically but as a general rule, when PSCustomObjects are produced, information is added to TypeNames so you can identify the origin of the object. So this is an entirely reasonable behaviour.

it is a breaking change in a grace area

No it's not. You showed that it would absolutely break people.

it is bad design because this cmdlet always creates custom objects

It creates custom objects because these are projections - there is no nominal type to return. It might be possible to do what LINQ does: synthesize anonymous types and unify on property names but that would be a breaking change and I'm not sure generating transient types is the best solution for this scenario. It must be possible to optimize this (used interned strings or something.)

users can use a workaround manually adding a custom type or a custom property to mark objects

Since you're ultimately doing the same thing, how is that going to fix the problem?

[SteveL-MSFT]

@PowerShell/powershell-committee reviewed this, we agree that simply removing the Selected. type IS a breaking change so any fix would need to be made without breaking existing users.

[iSazonov]

Since you're ultimately doing the same thing, how is that going to fix the problem?

Th problem is that users in 99.999% cases do not need/not use (how GitHub search shows) and do not know the feature, and if they works with long time live custom object they can catch the issue with leak. In other words, it should be __opt-out__ or just delegated to users.

This is all the more amazing because we do nothing of the kind in Where-Object, Foreach-Object,
and specially in Add-Member.

[bpayette]

To be clear, Select-Object should _only_ be adding the Selected prefix to PSCustomObjects it projects (creates). Everything else is a bug. Adding it 10000 times is a bug. Adding it to objects that it did not project is a bug. Does this make sense?

[msftbot[bot]]

:tada:This issue was addressed in #11548, which has now been successfully released as v7.1.0-preview.1.:tada:

Handy links:

• Release Notes