Powershell: Ctrl+C doesn't work as expected with SSH Remoting

[PaulHigin]

In PowerShell Ctrl+C should stop a running command. But in an SSH interactive remote session Ctrl+C terminates the SSH process and closes the session. This behavior is the same for both Windows and Linux clients.

This is happening because both PowerShell and SSH processes are attached to the console and so they both get the SIGINT signal. PowerShell sends a PSRP stop signal over the channel as expected but the SSH process also handles the signal and terminates the process which closes the remote session.

Steps to reproduce

PS> Enter-PSSession -HostName <hostName> -UserName <userName>
[<hostName>]: PS > 1..100 % { sleep 1; "Output $_" }
Output 1
Output 2
# While script is running type Ctrl+C to stop it

Expected behavior

Running command stops and remote session prompt is displayed.

Actual behavior

Session is ended with error:

System.IO.IOException: Pipe is broken

Also the exception is unhandled causing PowerShell process to crash.

Environment data

Name                           Value
----                           -----
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
SerializationVersion           1.1.0.1
PSRemotingProtocolVersion      2.3
BuildVersion                   3.0.0.0
WSManStackVersion              3.0
PSVersion                      6.0.0-alpha
CLRVersion
GitCommitId                    v6.0.0-alpha.10-8-gdd2394b1e6088f6c82054f5cae5520e532010fd4-dirty
PSEdition                      Core

[PaulHigin]

This has been fixed as part of the work for issue #2827 for a Windows client. When creating the ssh.exe process we can use the CREATE_NEW_PROCESS_GROUP process creation flag which prevents the console Ctrl+C signal from propagating to that process. But there is no support for this in managed code used on Linux clients so it is not fixed there yet.

[vors]

I'm hitting this problem.
Would be really useful to get fix for unix.

Unhandled Exception: System.IO.IOException: Broken pipe
   at System.IO.UnixFileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.UnixFileStream.WriteNative(Byte[] array, Int32 offset, Int32 count)
   at System.IO.UnixFileStream.FlushWriteBuffer()
   at System.IO.UnixFileStream.Flush(Boolean flushToDisk)
   at System.Management.Automation.Remoting.OutOfProcessTextWriter.WriteLine(String data)
   at System.Management.Automation.Remoting.Client.OutOfProcessClientCommandTransportManager.SendStopSignal()
   at System.Management.Automation.Internal.ClientPowerShellDataStructureHandler.SendStopPowerShellMessage()
   at System.Management.Automation.Runspaces.Internal.ClientRemotePowerShell.StopAsync()
   at System.Management.Automation.PowerShell.CoreStop(Boolean isSyncCall, AsyncCallback callback, Object state)
   at System.Management.Automation.PowerShell.BeginStop(AsyncCallback callback, Object state)
   at System.Management.Automation.RemotePipeline.Stop()
   at Microsoft.PowerShell.Executor.Cancel()
   at Microsoft.PowerShell.Executor.CancelCurrentExecutor()
   at Microsoft.PowerShell.ConsoleHost.HandleBreak()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

But there is no support for this in managed code used on Linux clients so it is not fixed there yet

Is it a CoreCLR limitation? Should we start conversation with dotnet people?

[iSazonov]

It seems ssh can set raw mode and send Ctrl-C to remote host as character.
http://unix.stackexchange.com/questions/102061/ctrl-c-handling-in-ssh-session

[PaulHigin]

This is interesting. I can verify that interactive SSH sessions don't handle the Ctrl+C signal and appear to pass it to the remote session as expected.

But in any case I don't want ssh to handle Ctrl+C at all. Instead it should be handled only in the PowerShell client process but it appears to get passed to the ssh child process as well.

In Windows this was an easy fix by simply preventing Ctrl+C from being passed from the PowerShell process to the child ssh process. I am not sure what the mechanism is for Linux but it may be a similar issue.

[iSazonov]

@PaulHigin Unix has the same behavior as Windows based on process groups
To move the process in a new process group, the process must make a call setpgid(0,0)

I believe that it can be easily and quickly be fixed in CoreCLR (add support for CREATE_NEW_PROCESS_GROUP flag)

I may be wrong, but it seems the above mentioned ssh RAW mode uses setpgid(0,0) https://github.com/openssh/openssh-portable/blob/79e4829ec81dead1b30999e1626eca589319a47f/sshpty.c#L112
If so, we implicitly have "fix" :-) If you have the opportunity to check, look at PGID for ssd in raw mode ps xao pid,ppid,pgid,sid,comm.

[PaulHigin]

@iSazonov This is great. Thanks very much for the information. I will look into this and check with the CoreCLR group about adding CREATE_NEW_PROCESS_GROUP support.

But I don't understand your comment about SSH RAW mode. RAW mode doesn't seem to be well documented and it is not clear if it can be used with subsystems. But even if it can it still passes the signal to the SSH process and I presume on to the SSHD remote session, and I want to prevent this.

[iSazonov]

I prefer the CoreCLR fix and you too - so that we can both forget about ssh raw mode even if it works :-)

[PaulHigin]

setpgid() doesn't allow the process group id to be set if the process has already started executing code. In addition there are restrictions to what process group id it can be set to (must be a process in the same session). I didn't see anywhere where you can create a process with a new group process id so I am not sure if this is possible. In any case I need to make a request to .Net to see if CREATE_NEW_PROCESS_GROUP can be supported. My guess is this can't be done in the beta 1 timeframe.

[iSazonov]

@PaulHigin Maybe comments from http://pubs.opengroup.org/onlinepubs/009695399/functions/setpgid.html will be useful.

The new processes call setpgid() to alter their own process groups after fork() but before exec.

Since it would be confusing to an application to have its process group change after it began executing (that is, after exec), and because the child process would already have adjusted its process group before this, the [EACCES] error was added to disallow this.

[PaulHigin]

@iSazonov Thanks again. It looks like you can specify a new process group id if done at the right time
"The setpgid() function shall either join an existing process group or create a new process group..."
I have created an issue/request for dotNet Standard to add this (CREATE_NEW_PROCESS_GROUP) support.

[iSazonov]

@PaulHigin Please add link on the Issue for tracking.

[PaulHigin]

Here is the link: https://github.com/dotnet/corefx/issues/17412

[PaulHigin]

Follow up: I have created an API request to dotnet/corefx/issues to add support for process creation flags. It is not clear if this proposal will be accepted or if the dotnet/corefx team will have resources to review/implement. But there are other options. One is to review the existing Linux process creation code and submit a pull request to support process creation flags. The other is to use Linux API PInvoke in PowerShell to do our own process creation on Linux.

But in any case I feel there is not enough time to solve this issue for beta-1 and so I am moving this issue to beta-2. We will need to document this issue for the beta-1 release.

[joeyaiello]

Totally fair to move out to beta2, especially given that we'll be focusing especially on remoting scenarios for that release.

I'm going to work to engage with CoreFX with you, @PaulHigin. Let's chat offline.

[joeyaiello]

@PaulHigin it looks like this works now (with a weird error about the pipeline that doesn't show up locally). I still leave this to you to close, though, because it still looks like you're talking to CoreFX folks about this issue.

[jaiello-centos]: PS /home/jaiello> 1..100 | % { sleep 1; "Output $_" }
Output 1
Output 2
Output 3
Output 4
The pipeline has been stopped.
    + CategoryInfo          : OperationStopped: (:) [], PipelineStoppedException
    + FullyQualifiedErrorId : PipelineStopped

[PaulHigin]

It currently only works on Windows clients. It is still broken on Linux clients.

[joeyaiello]

@PaulHigin Really? Like Linux SSH clients? Or Linux PowerShell clients? Both appear to work for me right now...

[PaulHigin]

Yes. On Linux/Mac OSes Ctrl+C is handled by the SSH child process and closes the process/connection. We want to prevent this and let PSRP (PowerShell remoting protocol) handle the Ctrl+C signal.

[joeyaiello]

Okay, I see it now. Not sure why I couldn't get it before, but this looks like it's only for PSRP over SSH. Not generic Linux SSH clients. For posterity, the error I see right now is:

The background process reported an error with the following message: "The named pipe target process has ended.".
    + CategoryInfo          : ResourceUnavailable: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : System.Management.Automation.Remoting.PSRemotingDataStructureException

[iSazonov]

@PaulHigin @vors If we put the remote pseudo-terminal in raw mode - is it resolve the problem?

ssh -t remotehost command args - put local terminal in raw mode. Then PowerShell put remote pseudo-terminal in raw mode with cfmakeraw - Is it possible?

[PaulHigin]

This is now fixed for both Windows and Linux platforms.