Powershell: Memory leak executing remote commands from Linux (Docker) to Windows

[rthardin]

Steps to reproduce

Start a fresh Docker container with either the latest stable or latest preview of the mcr.microsoft.com/powershell:centos-7 container, on either OS X or Linux (see Environment data for more detail):

docker run -it mcr.microsoft.com/powershell:centos-7 bash

or

docker run -it mcr.microsoft.com/powershell:6.1.0-preview.3-centos-7 bash

Install the prerequisites for SPNEGO/NTLM Authentication on OMI as per this guide: https://github.com/Microsoft/omi/blob/master/Unix/doc/setup-ntlm-omi.md

yum install -y epel-release && yum install -y krb5-workstation gssntlmssp

From pwsh, create a new session with negotiated authentication, and run any remote commands on the remote computer (using hostname here as an example). In this example, the remote machine is running Windows Server 2016.

# Open a session with negotiated authentication.
$pwd = ConvertTo-SecureString -AsPlainText -Force -String "..."
$cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "...", $pwd
$session = New-PSSession -ComputerName "..." -Credential $cred -Authentication Negotiate

# Execute some remote commands and observe increasing memory usage.
for ($i=0; $i -le 100; $i++) {
    Invoke-Command -Session $session -ScriptBlock { hostname }
}

# Clean up the session and observe that memory is not freed.
$session | Remove-PSSession

Expected behavior

Some increase in memory utilization by pwsh as shown by top while the loop is executing, stabilizing as garbage is collected. Eventually, the memory used by pwsh should reduce until it's near where it started.

Actual behavior

Linear increase in memory utilization by pwsh as shown by top while the loop is executing, which seems to have no upper-bound (until the host is out of memory). The memory used by pwsh is not reduced after ~6 hours of sitting idle. As a rough estimate, I've seen an increase in memory of about one megabyte per call to Invoke-Command (e.g. the usage procedure in the Steps to reproduce section increased memory used by pwsh from about 70 MB to about 150 MB in 100 calls to Invoke-Command).

Environment data

PowerShell versions:

> $PSVersionTable                                                                                                                                       

Name                           Value                                                                                                                       
----                           -----                                                                                                                       
PSVersion                      6.0.2                                                                                                                       
PSEdition                      Core                                                                                                                        
GitCommitId                    v6.0.2                                                                                                                      
OS                             Linux 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018                                                              
Platform                       Unix                                                                                                                        
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                     
PSRemotingProtocolVersion      2.3                                                                                                                         
SerializationVersion           1.1.0.1                                                                                                                     
WSManStackVersion              3.0       

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.1.0-preview.3
PSEdition                      Core
GitCommitId                    v6.1.0-preview.3
OS                             Linux 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 ...
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Docker on OS X:

> docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:13:02 2018
 OS/Arch:      darwin/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:22:38 2018
  OS/Arch:      linux/amd64
  Experimental: true

Docker on Linux (CentOS):

> sudo docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

[iSazonov]

Related #7112.

[larssb]

I'm experiencing this issue as well. Only difference seems to be the NTLM/Kerberos related apps I install in my image. Which is Ubuntu bionic based.

They are:

• krb5-user
• cifs-utils

I've tried the following localize the culprit of the memory issue I see when executing my PowerShell code inside a Docker container.

• WinDbg analyzing a full mem. dump. Taken outside a container. As the code is the same. Though there is a difference in the PowerShell version. Inside the container it is v7 preview 2 and outside it is v5.1. Couldn't find the issue. Might be my less than average skills in debugging memory dumps.
• Used GC.Collect(). No difference
• Tried configuring the System.GC.Server & System.GC.Concurrent settings in the pwsh.runtimeconfig.json file. Made no difference. The container was restarted and calling e.g. [System.Runtime.GCSettings]::IsServerGC returns True after the restart.
• Went through my code to see if there might be variables I do not remove, or other objects that might be hogged so that they are not cleared from memory when garbage collection occurs. Optimized in many places. The issue remained.
• Been through the issues in this repository and found some very interesting issues besides this one (which is the one being closest to the issue I'm experiencing). The issues are:
  
  
  
  • ConvertFrom-JSON high memory consumption
  
  
  • Invoke-RestMethod is slow and uses a lot of memory - my code also uses the Invoke-RestMethod. However, it should not be a problem according to the close note on the issue. However, I'm using ConvertFrom-JSON soooo
  
  
  • Performance of Windows PowerShell is better then PowerShell Core
  
  

The code I execute makes great use of Invoke-Command over a remote PowerShell session from the Linux Ubuntu container to different Windows servers. As @rthardin I also make sure to remove the PS session after use.

I don't see this issue as being thaaaat tightly related to #7112 @iSazonov .... read through that issue and yes there are similarities. However, the big difference is the use of Invoke-Command and New-PSSession and the challenges in regards to memory consumption that seems to come with.

I'll happily provide more info. Feel free to ask.

[larssb]

Any news on this?

[larssb]

@SteveL-MSFT I imagine that the PS team at MSFT is very busy with the upcoming 7GA release. But, is there any chance I could get a little comment on this issue. Por favor 💯

Thank you and have a great day.

[larssb]

Any news on this? A couple of more months have passed by. @iSazonov ? Or maye you @rthardin

Thank you so much.

[vexx32]

/cc @PaulHigin is this something within your area of expertise?

[iSazonov]

The code I execute makes great use of Invoke-Command over a remote PowerShell session from the Linux Ubuntu container to different Windows servers.

@larssb

1. Do you see the issue with latest PowerShell 7.0 Preview?
2. Do you see the issue without container (direct connect from Linux to Windows Server)?

[larssb]

We'll be updating to version 7 of PowerShell this week. Then we'll know more. Unless there is some fixes or the like in the latest v7 preview, that makes you ask @iSazonov ?

I'll see what I can do on your second question.

Thank you ... also to you @vexx32 for chirping in.

[iSazonov]

@larssb There were some performance fixes.

Now you could collect .Net traces on Linux and analyze with PerfView https://michaelscodingspot.com/dotnet-trace/

Michael's Coding Spot

Performance Profiling of .NET Core 3 applications on Linux with dotnet-trace and PerfView - Michael's Coding Spot

Performance issues never seem to disappear from the world, no matter how fast new computers become. To deal with those issues we need proper tools. We have some great tools on Windows. On .NET Core with Linux, things are not so great. But they're getting better.

[PaulHigin]

A couple of comments:

When measuring memory pressure, be sure and run [gc]::Collect() to ensure you are seeing a real memory leak, and not just the CLR neglecting to garbage collect freed objects.

This is WinRM over OMI remoting, which has been deprecated and is something we are no longer supporting. So if there is a memory leak in the implementation, it is unlikely we would address it. Instead, we are encouraging cross platform remoting through SSH.

[larssb]

Hmm @PaulHigin. Interesting. I wasn't aware of the lack of support for WinRM over OMI. I've tried [gc]::Collect() to get the debugging effect you mention. Still an issue.
However, if what Microsoft support is remoting through SSH I'll see if I can work with that relative to the infrastructure on which I have to do the remoting.

Looking at OpenSSH Server Configuration..... It looks like it's only from Windows 2019 server that SSH is fully supported. What if one uses 2016 or 12? Is there anything to do there. Besides using WinRM over OMI? I would prefer to use remoting through SSH. As that is what you say you are supporting.

Thank you very much.

[iSazonov]

@larssb You can download OpenSSH here https://github.com/PowerShell/Win32-OpenSSH/releases
WiKi says that it works on Windows Server 2008 R2.

GitHub

PowerShell/Win32-OpenSSH

Win32 port of OpenSSH. Contribute to PowerShell/Win32-OpenSSH development by creating an account on GitHub.