Vscode-remote-release: Remote-SSH: Fails to discover WSL SSH

[UnicodeTreason]

• VSCode Version: 1.37.0-insider
• Local OS Version: Windows 10 1803
• Remote OS Version: N/A
• Remote Extension/Connection Type: SSH

Steps to Reproduce:

1. Have Remote-SSH installed
2. Connect to Host
3. VSCode fails to find any SSH with error "An SSH installation couldn't be found"
4. I checked the Remote - SSH Output and can't even see it attempting to look for WSL SSH
   Got error from ssh: spawn ssh ENOENT
   Testing ssh with C:\WINDOWS\System32\OpenSSH\ssh.exe -V
   Got error from ssh: spawn C:\WINDOWS\System32\OpenSSH\ssh.exe ENOENT
   Testing ssh with C:\Program Files\Git\usr\bin\ssh.exe -V
   Got error from ssh: spawn C:\Program Files\Git\usr\bin\ssh.exe ENOENT
   Testing ssh with C:\Program Files (x86)\Git\usr\bin\ssh.exe -V
   Got error from ssh: spawn C:\Program Files (x86)\Git\usr\bin\ssh.exe ENOENT
   Finding installed ssh failed: ssh installation not found
   ssh installation not found

Which is unexpected as the documentation here: https://code.visualstudio.com/docs/remote/troubleshooting#_installing-a-supported-ssh-client

Specifically states it'll use WSL SSH

These closed issues also make it look as though it WAS added recently:
https://github.com/microsoft/vscode-remote-release/issues/359
https://github.com/microsoft/vscode-remote-release/issues/448

[Chuxel]

@roblourens Did the update to dynamically find the SSH client on Windows include a WSL check or did that get cut/missed?

[roblourens]

Yeah I missed that entirely. Can you suggest how I should discover/invoke WSL ssh?

Also would have to figure out how config files will work. WSL ssh would not look at windows' ssh config files by default and vice versa.

[davidwin]

Enabling the WSL ssh would be _awesome_, since it should make it possible to take advantage of ControlMaster on Windows. We rely on WSL ssh heavily for a homegrown remote development solution, and connection multiplexing really helps to make things zippy.

[Chuxel]

@roblourens I've got a PR out to update docs in the mean time.

You should be able to detect WSL SSH with the following command line call:

where wsl && wsl which ssh

Non-zero exit means it's not there.

[UnicodeTreason]

Thanks for jumping onto this one guys, saves us having to run two SSH configurations.

[diablodale]

Additional test case as this is implemented...

• Force usage of WSL ssh

I tried three alternate settings.json of

  "remote.SSH.path": "C:\\Windows\\System32\\wsl.exe ssh"
  "remote.SSH.path": "\\\\wsl$\\Ubuntu-18.04\\usr\\bin\\ssh"
  "remote.SSH.path": "/usr/bin/ssh"

They all failed with the Remote - SSH output log reporting for each their location...

Testing ssh with C:\Windows\System32\wsl.exe ssh -V
Got error from ssh: spawn C:\Windows\System32\wsl.exe ssh ENOENT
The specified path C:\Windows\System32\wsl.exe ssh is not a valid SSH binary

I have a very little tested workaround to force WSL ssh. Use a settings.json of

  "remote.SSH.path": "c:/njs/ssh.bat"

with an ssh.bat of

@echo off
set v_params=%*
set v_params=%v_params:\=/%
set v_params=%v_params:c:=/mnt/c%
REM set v_params=%v_params:"=\"%
C:\Windows\system32\wsl.exe ssh %v_params%

[diablodale]

Hello. A regression in use of the setting remote.SSH.path to use WSL occurred in VSCode Windows release 1.37.0; where no problem existed in the previous 1.36.x. Putting it in this issue as it is highly related to this discussion.

The setting remote.SSH.path no longer accepts correctly a path like c:/njs/ssh.bat
In 1.37 on Windows, the setting must be with backslashes.

Setup

Version: 1.37.0 (system setup)
Commit: 036a6b1d3ac84e5ca96a17a44e63a87971f8fcc8
Date: 2019-08-08T02:33:50.993Z
Electron: 4.2.7
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 10.0.18362

Repo

1. Setup the ssh.bat file in a directory, and VSCode remote.SSH.path as in https://github.com/microsoft/vscode-remote-release/issues/937#issuecomment-514714615
2. Run and connect using Remote SSH

Result

The connection fails. In the Remote SSH output window is the error

[10:35:33.341] Install and start server if needed
[10:35:33.404] > 
[10:35:33.405] Got some output, clearing connection timeout
[10:35:33.903] > ]0;C:\WINDOWS\SYSTEM32\cmd.exe
[10:35:33.912] >
...
[10:35:33.920] > 'c:' is not recognized as an internal or external command,
> operable program or batch file.

Expected

No error and a successful connection

Workaround

Change the setting of the path to use backslashes

"remote.SSH.path": "c:/njs/ssh.bat"    <--- fails

"remote.SSH.path": "C:\\njs\\ssh.bat"    <--- works

[roblourens]

Ugh windows. Forked that into https://github.com/microsoft/vscode-remote-release/issues/1148, thanks.

[andrzejnovak]

I can attest that using ssh through wsl would be amazing. Apart from the obvious non-duplication of configs, I need to connect to a host which only allows kerberos authentication.

Is there a documented workaround for now? I tried the ssh.bat file hack mentioned above, but got nowhere. There's an error stating The specified path is not a valid SSH binary

[8549]

Setting remote.ssh.path to \\wsl$\Ubuntu-18.04\usr\bin\ssh doesn't work, either.

[UnicodeTreason]

Seems like this one's a doozy, everyone's efforts are appreciated.

[sweepyoface]

The script provided by @diablodale doesn't seem to work for me. When I go through the dialog to create a new remote SSH connection, it prompts me for the command and then the config file (of which I use a path to the WSL one) and then nothing happens. No new window opens up and no connection is added.

[diablodale]

@sweepyoface , contact me via email if you want my help diagnosing. It is a workaround, and I prefer not to muddy this issue w/ support convo on a workaround. :-)

[andrzejnovak]

@diablodale actually if you could that would be great, this clearly collected a bunch of ppl who'd like a fix but it doesn't work, presumably many more didn't bother to comment

[diablodale]

Install/Discussion for my BAT workaround is at https://gist.github.com/diablodale/54756043c395d712053cf0d50a86086a

[andrzejnovak]

Can confirm @diablodale 's shim works great. Presumably it something that could be added natively to the extensions?

[cchan]

Can also confirm @diablodale 's shim works great. Just wanted to additionally report that it doesn't play nice with Use Local Server, which might be a separate issue from this.

[lancslingwood]

Change the last line of @diablodale 's script to:
C:\Windows\system32\bash.exe -ic "%~n0 %v_params%"
Your ssh-agent will now work! This just loads your profile so the env vars are set.

[cchan]

That works great @lancslingwood!

I made a couple more changes to be able to use the same ssh config between vscode and WSL2, so here's the full instructions I've followed so far on Windows 10 v2004 if it's helpful to anyone:

• Use @diablodale's shim, saved to C:\Users\Clive\ssh.BAT (home directory is in the Windows PATH by default, and for me it gets called first ahead of my OpenSSH installation)

• With the last line of that file modified so it grabs the env variables and can use the ssh-agent (@lancslingwood):

C:\Windows\system32\bash.exe -ic "%~n0 %v_params%"

• vscode config (if you saved ssh.BAT somewhere else, you need to set remote.SSH.path as well.) Make sure to set your User config, not your Remote or Workspace configs.

"remote.SSH.configFile": "\\\\wsl$\\Ubuntu\\home\\clive\\.ssh\\config"

• Make sure bash knows what to do with the config file path when vscode passes it with ssh -F:

sudo ln -s / //wsl$/Ubuntu

All this feels like a massive Frankenstein but at least it's an invisibly scripted Frankenstein now :P

[diablodale]

If your specific setup needs to run .profile and/or .bashrc there are several combinations of bash features which cause those files to load but also creates side-affects. Your specific setup/needs can guide you on this. 👍
https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html

-i indicates an interactive shell and additional code/features happen when bash is marked interactive. One example: the default .bashrc has code near # If not running interactively which prevents running most of that file. If you want the fastest execution of the shim's command with a more "pure" _non_-interactive session, it is usually not appropriate to indicate it is interactive. I use the shim for many things that need fast execution, e.g. pylint, so I do not use the -i approach.

Instead, I use the approach as written in the shim using wsl.exe and use the bash environment variable BASH_ENV=~/.bashrc. I set that variable as a Windows user environment variable and WSLENV=BASH_ENV so it is passed to the WSL subsystem. Now when I run something with wsl.exe it is non-interactive non-login yet runs ~/.bashrc. I discovered in some circumstances of truly interactive and/or login sessions, the .bashrc will then load twice. I stopped that by putting an if test at the top of .bashrc to catch those circumstances. 🤪

[sweepyoface]

• Make sure bash knows what to do with the config file path when vscode passes it with ssh -F:

sudo ln -s / //wsl$/Ubuntu

Heads up, I had to first sudo mkdir //wsl$ to get this to work.

[monado3]

These solutions don't work for me.

Here're my settings and codes. Can anyone fix?

• wsl2 version

$ wslsys                                                                                                                                      
Release Install Date: 0x5ed27425
Branch: vb_release
Build: 19041
Full Build: 19041.1.amd64fre.vb_release.191206-1406
Uptime: 0d 0h 14m
Linux Release: Ubuntu 20.04 LTS
Linux Kernel: Linux 4.19.104-microsoft-standard
Packages Count: 1127

• C:\Users\monad\Codes\ssh.BAT (I've confirmed this has 0 byte output.)

SETLOCAL EnableExtensions
SETLOCAL DisableDelayedExpansion
set v_params=%*
set v_params=%v_params:\=/%
set v_params=%v_params:c:=/mnt/c%
REM set v_params=%v_params:"=\"%
REM C:\Windows\system32\wsl.exe %~n0 %v_params%
C:\Windows\system32\bash.exe -ic %~n0 %v_params%

The reason I changed wsl.exe into bash.exe -ic is my default shell is fish, which has over 0 byte output when executing true.BAT.

• settings.json of VSCode

    "remote.SSH.path": "C:\\Users\\monad\\Codes\\ssh.BAT",
    "remote.SSH.configFile": "\\\\wsl$\\Ubuntu\\home\\monado\\.ssh\\config",

• at Ubuntu

$ la /wsl\$/                                                                                                                                     
total 8.0K
drwxr-xr-x  2 root root 4.0K Jun  5 00:49 ./
drwxr-xr-x 20 root root 4.0K Jun  5 11:59 ../
lrwxrwxrwx  1 root root    1 Jun  5 00:49 Ubuntu -> //

• the output when I executed remote-ssh

[12:08:49.089] Log Level: 2
[12:08:49.092] [email protected]
[12:08:49.092] win32 x64
[12:08:49.094] SSH Resolver called for "ssh-remote+kugenuma", attempt 1
[12:08:49.095] SSH Resolver called for host: kugenuma
[12:08:49.095] Setting up SSH remote "kugenuma"
[12:08:49.114] Using commit id "5763d909d5f12fe19f215cbfdd29a91c0fa9208a" and quality "stable" for server
[12:08:49.115] Install and start server if needed
[12:08:51.068] Checking ssh with "C:\Users\monad\Codes\ssh.BAT -V"
[12:08:51.252] ssh exited with code: 255
[12:08:51.253] The specified path C:\Users\monad\Codes\ssh.BAT is not a valid SSH binary
[12:08:51.253] Checking ssh with "ssh -V"
[12:08:51.255] Got error from ssh: spawn ssh ENOENT
[12:08:51.255] Checking ssh with "C:\WINDOWS\System32\OpenSSH\ssh.exe -V"
[12:08:51.288] > OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
[12:08:51.299] Using SSH config file "\\wsl$\Ubuntu\home\monado\.ssh\config"
[12:08:51.299] Running script with connection command: "C:\WINDOWS\System32\OpenSSH\ssh.exe" -T -D 50953 -F "\\wsl$\Ubuntu\home\monado\.ssh\config" kugenuma bash
[12:08:51.302] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[12:08:51.394] > CreateProcessW failed error:2
> ]0;C:\WINDOWS\System32\cmd.exe
[12:08:51.394] Got some output, clearing connection timeout
[12:08:51.412] > posix_spawn: No such file or directory
> The process tried to write to a nonexistent pipe.
[12:08:51.854] "install" terminal command done
[12:08:51.855] Install terminal quit with output: The process tried to write to a nonexistent pipe.
[12:08:51.855] Received install output: The process tried to write to a nonexistent pipe.
[12:08:51.857] Resolver error: The process tried to write to a nonexistent pipe
[12:08:51.868] ------

[RyanPersson]

Well, my setup WAS working with the ssh.BAT shim. However it stopped working a couple days ago & I'm getting a similar error to @monado3 .
Checking ssh with "C:\\scripts\\ssh.BAT -V" ssh exited with code: 1 The specified path C:\\scripts\\ssh.BAT is not a valid SSH binary

Running C:\\scripts\\ssh.BAT -V in cmd returns OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020
& I am able to use ssh.BAT to open ssh in wsl FROM cmd. So I know that the shim works. But remote-ssh's check if it's a valid binary is stopping it from even getting a chance to run...

[UnicodeTreason]

@roblourens Any progress on this to report? I assume WSL 2 might shake things up a little.

[fxzxmic]

I used my poor knowledge of C language to write a program, according to your own needs to compile it into ssh.exe. Hope there are other awesome people can help me optimize, I'm just a novice.
Because this program just forwards ssh.exe's parameters to WSL or other bash environment. So, you can fully use the configuration used in WSL or other bash environments before. For example, the key file uses the internal location of WSL rather than the absolute location of key file in Windows environment.

#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[]) {
  int count;
  char ch[100] = "bash.exe -c \"ssh";
  for (count = 1; count < argc; ++count) {
    char ch0[100] = " ";
    strcat_s(ch0, sizeof(ch0), argv[count]);
    strcat_s(ch, sizeof(ch), ch0);
  }
  strcat_s(ch, sizeof(ch), "\"");
  system(ch);

  return 0;
}

[RyanPersson]

@fxzxmic's program is working for me!

Note that if you're compiling it on WSL/WSL 2, you need to use a windows compiler like mingw.

Steps I used to get it working:
1: Install a windows C/C++ compiler:
sudo apt-get install mingw-w64

2: Save @fxzxmic's code as ssh.c

3: Compile into a windows exectutable:
x86_64-w64-mingw32-g++ -o ssh.exe ssh.c

4: Set the path to ssh.exe in the remote-ssh settings:
C:\\some\\path\\ssh.exe

[cihatyildiz]

I had the same issue, I don't know why this showed up. But it has been fixed when I restart the system :)