Powershell: Powershell .net class loads slow since MS-Security Update April 2019

[fadanner]

Since May 2019 Security Rollup for Windows Server 2016, access to .net object via powershell is very slow, when a powershell variable breakpoint is set in the powershell.

For example loading speed of [System.Environment] takes over ten times with breakpoint set against no breakpoint set.

Since the behavior only occurs with latest security updates installed (May 2019 (KB4494440) or June 2019 (KB4503267)) i think it is related to the update.

Did some troubleshooting with procmon until now. For me it looks like when PSBreakpoint is set, there are multiple reads from HKLM\System\CurrentControlSet\Control\Session Manager\Environment\__PSLockdownPolicy can't repoduce this when there is no breakpoint set and can't reproduce this effect without one of the updates installed.

Following powershell code runs more than 10 times slower when i install Windows Security Rollup update Mai / June 2019 on my Windows Server 2016.

$measure1 = measure-command{
  foreach($i in 1..10000){
    [System.Environment]
  }
} |select -ExpandProperty TotalSeconds

Set-PSBreakpoint -Variable "test" -Action {$null}

$measure2 = measure-command{
  foreach($i in 1..10000){
    [System.Environment]
  }
} |select -ExpandProperty TotalSeconds

Write-Host "First measurement took $measure1 seconds, Second measurement took $measure2"

>>First measurement took 0.0482474 seconds, Second measurement took 0.8722697

Tested on:

PSVersion: 5.1.14393.2969
PSEdition: Desktop
PSCompatibleVersions: {1.0, 2.0, 3.0, 4.0...}
BuildVersion:10.0.14393.2969
CLRVersion:4.0.30319.42000
WSManStackVersion:3.0
PSRemotingProtocolVersion:2.3
SerializationVersion:1.1.0.1

Can anyone observe this behavior as well?

Do you have any suggestions to workaround this problem?

Is it related to some settings or installed software?

[bergmeister]

Please fill in the issue template as things like the PS Version are very relevant to this.

[iSazonov]

/cc @KirkMunro @PaulHigin

[fadanner]

Added $PSVersionTable to the issue

[bergmeister]

@fadanner This repository is not for Windows PowerShell as the issue template says, I assume the issue does not apply for PowerShell Core?

[KirkMunro]

@fadanner Thank you for explaining the issue you were seeing so clearly. I love it when someone indicates they took the time to isolate the problem, in this case trying with/without the OS updates, using procmon, etc. That is very much appreciated.

As @bergmeister suggested, it would be most helpful to know if this is an issue for PowerShell Core or not. Are you able to try that out on a test server, with PowerShell 6.2 installed? It can be installed side-by-side with PowerShell 5.1.

Also, you suggested that you don't see the slowdown without the OS updates. Using the same PowerShell versions, can you share those times as well to illustrate how much of a difference we're looking at before/after the update, based on your tests?

[nohwnd]

@bergmeister @KirkMunro I think we are seeing the same bug in pester/Pester#1318. The impact does not seem to be limited to accessing classes, but instead all code slows down when even a single breakpoint is present.

I can see that the problem impacts both Windows PowerShell 5.1 and PowerShell Core 6.2.2 and have a simple repro and measurements.

I was able to replicate this on Windows 10 Bussiness version 1903 (updated July 2019), straight from MSDN (gwmi win32_operatingsystem -> version 10.0.18362).

I can make more tests in case you need them, to confirm what the problem is, and to try possible solutions.

[iSazonov]

@nohwnd If you have a test environment you could collect traces (before appling the security patch and after) with PerfView utility and share traces - I expect that it will very simple to find the expensive system call.
Update: Perhaps it is more easy to get traces for Windows PowerShell - we can use WPR/WPA from Windows Performance Toolkit

[DavidGibsonNorth]

Can anyone observe this behavior as well?
@fadanner Yes, replicated by me

First measurement took 0.0102665 seconds, Second measurement took 1.6129433

$PSVersionTable
Name                           Value
----                           -----
PSVersion                      5.1.14393.3053
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.3053
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

.

[nohwnd]

@iSazonov I never used PerfView is there anything specific that I should setup / measure. Or just running Collect > Run with Powershell.exe -f repro.ps1 and then sharing the merged logs is enough?

I don't have a version of the vm before patches becasue I was never able to update from a clean station, and so I was able to replicate only on the build that I mention above, that already has the patch installed. But I have a second workstation that does not have the problem, will collecting the same thing on both be useful for you?

[iSazonov]

@nohwnd We need _compare_ two traces to find regression.

Or just running Collect > Run with Powershell.exe -f repro.ps1

No need start PowerShell every time. You can start PowerShell, prepare test, then start collect by Collect > Collect, run test, after the test done press Stop collection, save zip file and repeat all steps for new test in the same PowerShell session.

[iSazonov]

Tracing says a lot:

[bergmeister]

I suspect this was the fix for CVE-2018-12130, a.k.a. the Zombie Load Attack, that was fixed as part of the May Security update. Details about the specific KB numbers in which it was included can be found here: https://portal.msrc.microsoft.com/en-us/security-guidance/advisory/adv190013
It was mentioned that this fix brings a performance degradation of up to 40%, so I'm not surprised that one can generate more extreme outliers.

[Glober777]

@SteveL-MSFT, will the fix for the issue be back-ported to Windows PowerShell as is, or should a related record still be created in UserVoice?

[SteveL-MSFT]

@Glober777 I expect #10269 to be back-ported to 5.1. The root issue of the general perf impact isn't understood yet so can't comment on that.

[msftbot[bot]]

This issue has been marked as fixed and has not had any activity for 1 day. It has been closed for housekeeping purposes.