Vscode-remote-release: Failed to establish connection to remote SSH host

[underchemist]

I've been experimenting with the new remote development extension since I heard because it is the thing I've been most excited about for VS Code!

I have a similar environment as #15 and #36 where the remote development environment is a secured i.e. no or very select outside access. So I expect to not be able to properly set up the remote ssh server as eventually it will need to access the marketplace domains.

However I think I am experiencing another error before I get to that point. After attempting to connect, I get this in the remote extension log:

SSH Resolver called for "ssh-remote+wipml01"
SSH Resolver called for host: wipml01
Setting up SSH remote "wipml01"
Using commit id "473af338e1bd9ad4d9853933da1cd9d5d9e07dc9" and quality "insider" for server
Install and start server if needed
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> 
>          IMPORTANT NOTICE
>          ----------------
> 
> 
> 
> Use of this equipment and access to or use of any of the Company's computing or network facilities is subject to <company name>'
> s Corporate Security and Use Policies. Copies of these policies can be obtained by telephoning or online 
> 
> The company reserves the right to monitor, audit, or filter any e-mail, computer hardware or software, or network facilities as requi
> red, in order to enforce this policy. By logging on to this system you agree to comply with these policies and consent to any such mo
> nitoring, auditing or filtering.
> 
> 
> Enter passphrase for key 'C:\Users\ya006948\.ssh\id_rsa':
> 
> 
> bash: cannot set terminal process group (51108): Inappropriate ioctl for device
> bash: no job control in this shell
> 
> bash: /tmp/tmp.jM6fSydug6/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh: Permission denied
> 
"install" terminal command done
Received install output: bash: /tmp/tmp.jM6fSydug6/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh: Permission denied
Failed to parse remote port from server output: bash: /tmp/tmp.jM6fSydug6/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh: Permission denied

I'm not really sure what the issue might be. It looks like there are some permission issues, however the /tmp directory on the remote host has permission bits set to drwxrwxrwt, which seems very permissive. The other thing that seems to be an issue is the

> bash: cannot set terminal process group (51108): Inappropriate ioctl for device
> bash: no job control in this shell

Initially, I had oh-my-zsh as the default login shell but I disabled that as I thought it might be an issue but I get the same results with default bash login shell. I don't really know enough about the remote server implementation to pinpoint much further than this. Hopefully someone can point me in the right direction to getting past this point in the remote vs code server setup and I can fail on the inability to connect to extension marketplace.

• VSCode Version: Insiders 1.34
• Local OS Version: Windows 10 1709 (OS build 16299.1087)
• Remote OS Version: CentOS 7.6.1810
• Remote Extension/Connection Type: SSH
• SSH Client: installed from git for windows.

Steps to Reproduce:

1. Click Remote SSH: connect to host...
2. Select host defined in ssh config in %USERPROFILE%\.ssh\config
3. vs code terminal window opens prompting for identity file password, input password
4. A new vs code window opens, also opening a terminal window and asking for identity file password.
5. vs code server remote server begins installing as evidenced by notification in lower right hand corner
6. Error window with Could not establish connection to <hostname>

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Yes

[winddd]

I have the same problem.

[Binaryify]

same

[Boddlnagg]

I had the same problem, even without a password prompt (i.e. I have correctly configured key based authentication).

It seems that the problem was this in my ~/.bash_profile:

if command -v /bin/zsh > /dev/null; then
    export SHELL=/bin/zsh               
    exec /bin/zsh -l                    
fi                                      

It works after I've removed that.
I should find another way to configure zsh as my default shell ...

[underchemist]

Looking at my VMs /etc/fstab it looks like the /tmp dir is mounted with noexec. So I guess something similar to #17 would it be possible to change the directory the install remote server code is run from?

[Pistahh]

I have bash as my default shell (I cannot change it) but from .profile it executes zsh immediately, this breaks vscode remote. This is the workaround I use (same method should work with other shells too):

1. I created a new ssh keypair for vscode for use (I created it passwordless)
2. on the target server I added it to authorized_keys this way:
   environment="VSCODE=yes" ssh-rsa AAAA.....
3. I changed my .profile to

[[ "$VSCODE" == "yes" ]] && return

[[ -x /usr/bin/zsh ]] && SHELL=/usr/bin/zsh exec /usr/bin/zsh

1. I enabled PermitUserEnvironment yes in /etc/ssh/sshd_config

2. configured vscode to use the vscode private key

So when the vscode key is used, my ssh session sets the VSCODE env var, .profile checks it and if set it doesn't switch to zsh.

[Boddlnagg]

I was able to fix my .bash_profile by checking for interactive mode with one of the methods described here: https://www.gnu.org/software/bash/manual/html_node/Is-this-Shell-Interactive_003f.html

I then start zsh only in interactive mode. This works fine.

(Update: It actually doesn't work, see https://github.com/microsoft/vscode-remote-release/issues/39#issuecomment-493536460)

[underchemist]

If I follow @Boddlnagg link and modify my bash_profile to only run zsh from interactive shell then don't get a permission error with /tmp (which is strange, because my /tmp dir is still mounted with noexec) but I still have the

> bash: cannot set terminal process group (51108): Inappropriate ioctl for device
> bash: no job control in this shell

error

[underchemist]

I don't have the ability to change PermitUserEnvironment to yes in my environment.

[jdputsch]

Like Pistahh, I must switch shells in my bashrc and use a similar mechanism.

I do not have the ability to change my shell otherwise.

In addition I do not have the ability to change PermitUserEnvironment.

Perhaps a solution similar to that of issue #27, do not startup the connection with "-i", so unexpected commands (such as cd, zsh) in .bash_profile no longer interfere, would help here.

In the meantime, I'm stuck working on duplicating much of my zsh enviornment in Bash such that I can get the right search path, pyenv settings, etc. available when I cannot switch to zsh for the initial connection due to this bug/issue.

[underchemist]

Just as a brief update:

• Changing PermitUserEnvironment to yes yielded no change in behavior
• No instance of jailshell seems to be running from ps aux | grep jail (from https://github.com/microsoft/vscode-remote-release/issues/128#issuecomment-489859938)
• I can reproduce the behavior with ssh -T <sshhost> /bin/bash -ilc echo $SHELL where I am prompted to put in my password, I get no job control errors as described above and then the process hangs I and have to ^C to return to my local shell.

[roblourens]

So it doesn't actually echo $SHELL? Can you try removing -T, or using -t instead, or playing with the -il arguments to bash, and see what combo is breaking it?

[underchemist]

So I gathered that this was the command the remote SSH extension is running locally to start/install the remote vs code server on the host. I've played around with different options (-T and -t) for ssh and (combinations of -ilc for bash). I'll list some example outputs and instead of testing with /bin/bash -ilc echo $SHELL I found the sh script that vs code tries to run so I'll use that in the examples. An interesting thing to note off the top:

• The behaviour I described before where it is "hanging" after no job control error messages is incorrect. There's no terminal prompt but if I type a command and hit enter i.e. ls I get the directory listings in the remote host

• command: ssh -T -vvv <sshhost> /bin/bash -ilc /tmp/tmp.dWzyEiRT1x/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh

• debug output:
  vscodessh_debug_T.txt

• again to note there is no prompt at the bottom of this output but I can type in ls and see the contents of my home directory on the ssh host.

• command ssh -t -vvv ya006948@<hostname> /bin/bash -ilc /tmp/tmp.dWzyEiRT1x/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh

• debug output:
  vscodessh_debug_t_flag.txt

• Here the key difference is that I end up in a full terminal shell

• command ssh -T -vvv ya006948@<hostname> /bin/bash -lc /tmp/tmp.dWzyEiRT1x/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh

• debug output:
  vscodessh_debug_no_i_flag_bash.txt

• I get a permissions denied now, I think because the /tmp directory is mounted with noexec. So here it is again where I've copied the script to my home directory.

• command ssh -T -vvv ya006948@<hostname> /bin/bash -lc /home/ya006948/tmp.dWzyEiRT1x/vscode-remote-install.473af338e1bd9ad4d9853933da1cd9d5d9e07dc9.sh

• debug output
  vscodessh_debug_no_i_flag_bash_from_home.txt

• it will hang here because my server does not have outside internet access, but that is a totally separate issue.

Apologies for the late response, and also for the really verbose debug text files. I don't have access to pastebin from work.

Another thing of note: if I manually download the vscode-remoteserver.tar.gz and install it manually into ~/.vscode/bin I can get the server to start. However when I try to connect to remote host through vs code insiders I still get stuck at the part where nothing seems to happen. It isn't running the initialization script once it's connect unlike when I remove the -i flag from the bash command executed after connection.

[roblourens]

Thanks for trying that. I still have no idea what's going on. It seems like it could be related to something in your bash_profile

If I follow @Boddlnagg link and modify my bash_profile to only run zsh from interactive shell

With the tests above, are you still running zsh from bash_profile? What if you remove that entirely?

[underchemist]

GOT shame bells

So that got past the issue of hanging/not running any command after the ssh connection was established. Sorry to have gone on a long goose chase here. So I was finally able to make connection through the insiders build with several caveats. I'll describe the workflow:

• download vscode-server-linux-x64.tar.gz manually
• untar in ~/.vscode-remote/bin/473af338e1bd9ad4d9853933da1cd9d5d9e07dc9
• connect to remote host in vscode ui
• permission denied since /tmp is noexec
• remount with exec on (don't tell my vm admin)
• now the startup script will run, it will "detect" that vscode-server has already been installed due to a package.json existing in ~/.vscode-remote/bin/<commit-id> 👌
• server starts up, vscode remote window appears and can browse remote filesystem.
• haven't experimented with other extensions, I'll have to install them manually since this vm doesn't have outside internet access

tl;dr on workarounds I have to do in order to connect

• On a new remote host, must manually install vscode-server.tar bundle if it doesn't have outside internet access
• Not use zsh (or configure my bash_profile properly so it can detect if it's being run interactively or not)
• remount my /tmp dir with exec option (from noexec). Would be nice to have a configurable option to choose where to run the startup script from
• haven't played with this extensively but will presumably have to figure out how to manually install extensions for the time being if I want them to be run remotely.

Thanks so much for the help, I'll try to update in a few days once I've played around with it a bit more.

[underchemist]

From my two full days of development with remote-ssh so far I would say that is issue can be closed. I think the root issue was that remote-shh is incompatible with a zsh starting shell and a lot of my other issues were masked by the fact that I thought I had successfully implemented logic to only start zsh shell from my bash_profile if connecting user wasn't doing so through vs code's remote ssh connection.

[roblourens]

Thanks for the details and for following up. I will close it, some of these issues have come up in other issues too and I am building a list of these and will figure out where to track them...

So far it seems like launching a different shell from bash_profile is not uncommon and I need to figure out what to do about that.

[Boddlnagg]

@underchemist

I thought I had successfully implemented logic to only start zsh shell from my bash_profile if connecting user wasn't doing so through vs code's remote ssh connection

Were you subsequently able to to implement logic that actually works? Because what I suggested in my previous comment (https://github.com/microsoft/vscode-remote-release/issues/39#issuecomment-489158955), which first looked like it helped, doesn't seem to work either.

[roblourens]

We run the install script in interactive mode so that won't work.

I think the solution is to not launch a new shell from bash. That just makes it impossible to actually use bash which you may want to do some day. You can use chsh to set the default shell.

[Boddlnagg]

I tried chsh but it doesn't work in my environment. I have to see whether I can do anything to make it work. Thanks for your answer, anyway!

[roblourens]

Oh ok please follow up if you figure out what the issue is with chsh

[netbitsolutions]

I fixed mine. This error is due to current user not having right to set files in directory outside thier home. To fix, do the following:

In your .bashrc in the users home folder, add this

export TMPDIR=~/tmp/

[Philippe-Collignon]

I moved env zsh -l from .bashrc to .bash_profile and it works.

[pkokp8]

same.
I'm using zsh as my default shell, and I remove these from my .bash_profile to use bash:

[ -f $HOME/.local/bin/zsh ] && {
    echo "Type n/N to run bash: "
    line=""
    read -t 3 line
    [ "$line" != n ] && [ "$line" != N ] && exec $HOME/.local/bin/zsh -l
}

Then vscode remote works!
I have no permission to run chsh. But when I change to use zsh again, it still works. Anyway, it works now.

[rodolfo-gonzalez]

this works on fish shell

if not status is-interactive
    /bin/bash
end