Powershell: Memory leak when foreach-object parallel

Created on 10 Sep 2020  路  2Comments  路  Source: PowerShell/PowerShell

Steps to reproduce

$scriptThrottleLimit = 4
$scriptTimeoutLimit = 0
$ErrorActionPreference = Continue
$test = Get-Aduser -Filter *
$test.count
14483

 $test.samaccountname | Foreach-Object -Parallel {
     Write-Output $_
 } -ThrottleLimit $scriptThrottleLimit -TimeoutSeconds $scriptTimeoutLimit

Expected behavior

works normally, process all object

Actual behavior

After processing some pack of objects, server goes to out of memory, then pwsh stop using cpu (0-1%) and stop processing pipe. Only stop process works. I tried different numbers in throttlelimit, no change, it just slow memory leak.
https://prnt.sc/ueti4c
http://prntscr.com/uetj45
i think it not clean after each of parallel instance

Environment data


Name                           Value
----                           -----
PSVersion                      7.0.3
PSEdition                      Core
GitCommitId                    7.0.3
OS                             Microsoft Windows 10.0.14393
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0鈥
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0
Issue-Question WG-Remoting

All 2 comments

/cc @PaulHigin for information.

This version of PowerShell does not have the perf improvement for ForEach-Object -Parallel, and creates and destroys a runspace for each loop iteration (perf improvement in 7.1 preview reuses runspaces by default).

But I have tested both versions for leaks, and haven't found any.

However, running scripts in parallel does use a lot of resources. Try adding [System.GC]::Collect() to your loop to help ensure CLR recovers resources.

I notice two things in your test repro above:

  1. You store output from Get-ADUser, which may use a lot of resources depending on how large the returned objects are. Instead you could just stream directly to For-EachObject -Parallel.
  1. Your loop script writes to output/console, which is a serial operation. So your repro expends a tremendous amount of resources to parallelize something that is immediately re-serialized, and essentially provides no benefit.
Was this page helpful?
0 / 5 - 0 ratings