Vscode-remote-release: Does not support agent forwarding - committing on remote host via SSH fails

[rcarmo]

• VSCode Version: Version 1.34.0-insider (1.34.0-insider)
• Local OS Version: Mac OS X Mojave 10.14.4 (18E226)
• Remote OS Version: Ubuntu Linux 18.04
• Remote Extension/Connection Type: SSH

Steps to Reproduce:

1. Open remote folder/project workspace with git repository via SSH
2. Try to commit something to an even more remote repo (say GitHub) that relies on SSH auth.

Since the remote server does not have your SSH key and the VS Code connection does not use agent forwarding, key validation by the Git server (GitHub in this case) fails because the remote machine cannot talk to your local SSH agent and does not have valid keys.

This is a breaking issue for anyone who has to work via jump boxes.

Does this issue occur when you try this locally?: N/A
Does this issue occur when you try this locally and all extensions are disabled?: N/A

[kieferrm]

@rcarmo If you ignore VS Code Remote for a bit, how have you setup your SSH box so that your git cli on the SSH box works well over ssh to GitHub?

[rcarmo]

The top of the ~/.ssh/config on my Mac and WSL userland reads like this:

Host *
    AddKeysToAgent yes
    UseKeychain yes
    IdentityFile ~/.ssh/id_rsa
    controlmaster auto
    controlpath /tmp/ssh-%r@%h:%p
    ForwardAgent yes
    Compression yes
    ServerAliveInterval 120
    ServerAliveCountMax 3
    TCPKeepAlive yes

VSCode on the Mac is parsing this file correctly, since it is actually using the hosts I have defined in it as suggestions.

In general, doing what I do (SSHing to a remote machine and then committing via SSH to GitHub using the key on my local laptop) works as long as I have an equivalent local config to the above or explicitly type ssh -A for agent forwarding. This works in Linux (since time imemorial, that specific file above is easily around eight years old), macOS or WSL.

(Just to make it crystal clear, I have to do _zero_ setup on the remote/intermediate box. No extra keys, no added configs, nothing. OpenSSH knows how to deal with agent forwarding and ask my local agent on my laptop for keys, as long as my local machine tells it it's doing agent forwarding)

The only occasion I've had it break is when I forget to add my .tmuxrc to local and remote machines to pass on (and update) the right environment variables when I open a new tmux window.

Besides committing to remote repos, using agent forwarding is also (very) commonly used to log in to machines beyond jump boxes - and the terminal inside VS Code Remote, lacking the right environment variables, cannot talk to my local agent either.

[TomFrost]

Piling on here -- I have agent-forwarding explicitly defined in the host configuration that VSCode is connecting to:

Host work-remote
    User tom
    HostName tom.myworkdomain.net
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent yes
    LocalForward 127.0.0.1:5432 127.0.0.1:5432

VSCode observes all of these things, but ignores ForwardAgent. This is befuddling, since the docs insinuate that VSCode is using the locally installed ssh command, and if I open a terminal window on this machine and ssh work-remote with no other arguments, my agent gets forwarded automatically.

It would be a _very_ high-impact change for me and others I work with to allow the local ssh's default functionality to handle this. Or, alternatively, if VSCode Remote must parse this file and handle the connection itself, passing -A to ssh when ForwardAgent yes is present would also get the job done.

(Quick note, since it's easy to see a bunch of bug reports and feel unappreciated -- this project is _amazing_ and has, in the span of the 4 minutes it took to set up, redefined my development process. This is seriously incredible work, team.)

[roblourens]

This doesn't work by default because what we do isn't quite the same as logging in with a terminal. The ForwardAgent property causes SSH to set the SSH_AUTH_SOCK environment variable in the remote environment. First we open a connection to install the remote agent. If the agent is not already running, we start it. The agent and the remote EH processes that it spawns will get the SSH_AUTH_SOCK from this very first connection. But that SSH connection is short lived so the value of SSH_AUTH_SOCK is invalid later.

But once the agent is running and we know which port it is listening on, we start a second SSH connection to forward that port to the local machine. That SSH connection stays open and we want SSH_AUTH_SOCK from that connection. So we need to

• Log $SSH_AUTH_SOCK when creating the port tunnel
• Somehow set that variable in the environment of the remote EH

Remote-SSH is a local extension so we can't do this directly but I think the way to do it would be to have a remote helper extension, send the socket path to it, and have it modify the remote EH environment?

[nathan-sain]

@roblourens I didn't see your answer while I was writing this....

Using VSCode insiders build on Windows 10 using native Windows OpenSSH, I also am unable to use agent forwarding while connected via Remote-SSH. Checking the environment in a VSCode terminal shows

SSH_AUTH_SOCK=/tmp/ssh-m7litxvEqu/agent.56974
SSH_CLIENT='XXX.XXX.XXX.XXX 50429 22'
SSH_CONNECTION='XXX.XXX.XXX.XXX 50429 10.10.1.33 22'

But the path /tmp/ssh-m7litxvEqu/agent.56974 does not exist on the remote host. Additionally the file name agent.56974 would indicate that agent forwarding would use a sshd process running with PID 56974 but there is not sshd running on that process. The other SSH environment variables also indicate that the ssh session was established using a client connection on port 50429 from my workstation. But running netstat | grep ssh shows

tcp        0    160 ansible:ssh             XXX.XXX.XXX.XXX:50432   ESTABLISHED

So it would appear that VSCode establishes a sshd connection to setup / start the server, but then closes that connection, leaving the environment that the server is running in inaccurate.

Would it be possible to use SSH multiplexing as a remedy to this problem?

[roblourens]

Exactly correct @nathan-sain.

I would suspect that using ControlMaster would fix this but I see that the OP seems to have it set up.

[TomFrost]

@roblourens Thanks for the great explanation! I'd assumed a remote server was started with each new connection, but needing two connections to determine the forwarding port makes a ton of sense. The ugly side-effect is that two different machines making two different SSH connections to the same server instance can't each forward their own agent -- a bummer, since I do that with my office machine and a separate laptop. But given that solving that would require rearchitecture, accepting that only one agent can forward at one time seems sane.

With the above shortcoming accepted as a given, the solve for this could be as simple as writing SSH_AUTH_SOCK to a temp file that the server has an inotify watch on. But I suppose that's not as clean as an extension to the server process ;-)

[rcarmo]

One iteration of my tmux scripts did exactly that (pull off and set SSH_AUTH_SOCK from a file, but inotify tends to get swamped in database servers and machines with lots of writes - and I often forgot to bother with sysctl -w a new max limit ;)

[roblourens]

Using ControlMaster with the steps here does work for me:

https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods

Meaning I can run ssh-add -l in a remote vscode terminal and see my local keys. First I had to kill the remote agent using the command "Kill VS Code Server on Host" to make sure we start a new one with the right environment. I think this would be my recommended workaround for now. I can add a note to the docs.

@rcarmo so you are using controlmaster for this connection? Can you check the value of SSH_AUTH_SOCK in a vscode terminal?

[rcarmo]

Sure thing. Here's my Docker builder on my home LAN:

rcarmo@rogueone:~/Development/alpine-node$ set | grep SSH_
SSH_AUTH_SOCK=/tmp/ssh-sS32r23wcx/agent.31416
SSH_CLIENT='192.168.1.111 51626 22'
SSH_CONNECTION='192.168.1.111 51626 192.168.1.201 22'
rcarmo@rogueone:~/Development/alpine-node$ ls -al /tmp/ssh-*
ls: cannot access '/tmp/ssh-*': No such file or directory
rcarmo@rogueone:~/Development/alpine-node$ ssh-add -l
Error connecting to agent: No such file or directory

Either way, after killing the server using the "Kill VS Code Server on Host" I get pretty much identical results. But in my case every time I issue that command it seems to fail and I get a prompt to reload that VS Code window.

Oh, and the feature-request label is... suboptimal. General expectations are that this should work.

[rcarmo]

A bit more experimentation yielded a positive result inside the VS Code terminal with:

    ControlPersist  600

Set in my Mac's ~/.ssh/config file. However, a 600 second timeout is frankly excessive, and leaving control sockets lying around tied to a daemon (even with nothing connected to it, and it being battle-tested, etc.) so I opted to set it to ControlPersist 5, which works.

I can now use ssh-add -l to commit manually via the terminal actually I can't but I did get the right key added to my agent - I was committing to the wrong git remote, which is my CI system and runs in the same box.

This is what I get in my terminal now:

rcarmo@rogueone:~/Development/alpine-python$ set | grep SSH_
SSH_AUTH_SOCK=/tmp/ssh-H2e7wJyIm8/agent.28181
SSH_CLIENT='192.168.1.104 57043 22'
SSH_CONNECTION='192.168.1.104 57043 192.168.1.201 22'
rcarmo@rogueone:~/Development/alpine-python$ ssh-add -l
2048 SHA256:2Fke2KiPZYa4JXdUXJsouBdex4wmcfHZ768Mfu2wLu4  (RSA)
rcarmo@rogueone:~/Development/alpine-python$ git push
bind: No such file or directory
unix_listener: cannot bind to path: /home/rcarmo/.ssh/sockets/[email protected]
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Clicking on the "sync" button in VS Code fails with the following error:

Looking for git in: git
Using git 2.17.1 from git
> git rev-parse --show-toplevel
> git config --get commit.template
Open repository: /home/rcarmo/Development/alpine-python
> git status -z -u
> git symbolic-ref --short HEAD
> git rev-parse master
> git rev-parse --symbolic-full-name master@{u}
> git rev-list --left-right master...refs/remotes/origin/master
> git for-each-ref --format %(refname) %(objectname) --sort -committerdate
> git remote --verbose
> git check-ignore -z --stdin
> git pull --tags origin master
bind: No such file or directory
unix_listener: cannot bind to path: /home/rcarmo/.ssh/sockets/[email protected]
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

That's the next step, I think, and will require the server component to invoke git with the right environment (so the terminal workaround, if it worked, would only be a partial solution).

incidentally, if anyone wants to test remote Node debugging on ARM32 containers, I'm doing builds for Raspberry Pis (armv6 and armv7) and can try adding ARM64 :)

[roblourens]

However, a 600 second timeout is frankly excessive, and leaving control sockets lying around tied to a daemon (even with nothing connected to it, and it being battle-tested, etc.) so I opted to set it to ControlPersist 5, which works.

Oh, that makes sense. That will work but at some point the socket will time out and the next connection will create a new one, and the agent environment will be wrong after that point.

I don't quite understand where this is coming from: /home/rcarmo/.ssh/sockets/[email protected]

Are you also using ControlMaster on your remote? Is that a path for your local machine for some reason?

[rcarmo]

I'm not using ControlMaster on the remote - it has no SSH configuration whatsoever besides authorized_keys to allow me to log in.

But mind you, the socket is valid for as long as an existing connection is in place. Also, IIRC one of the reasons I didn't use ControlPersist is that it can get "stuck" due to a stalled process, and it broke my Ansible deployments (it was locked out of SSH until the timeout expired). This was a while back, can't remember the details.

As to the path, it is quite likely to be the default SSH socket path in Ubuntu, which means the environment isn't being propagated to terminal subprocesses.

[TomFrost]

Confirmed that I got this working with ControlMaster (Though Rob's link seems to be private to MS staff-- I believe he was pointing to https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods).

It would be great to work on this a bit more though, as this solution wouldn't allow me to, say, close my laptop, drive home, and continue working without an exceptionally long timeout. Having the server able to update the SSH_AUTH_SOCK when it changes -- if adn only if it points to a nonexistent file -- would be ideal. That way multiple local machines connecting to the same remote remains usable as long as the first doesn't drop while the second is still connected. That's a solvable case in the future as well though.

[rcarmo]

Confirmed that I got this working with ControlMaster (Though Rob's link seems to be private to MS staff-- I believe he was pointing to https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods).

Well, I can read the original link, and saw nothing _different_ from my config other than ControlPersist and the socket path - which shouldn't matter - but after a second read, it _is_ the same path as the one that git complains about...

My takeaway right now is that the documented workaround in that link makes an assumption for ControlPath (that may not work out for everyone, even though it seems to match OpenSSH defaults - which I'm still researching)

[nathan-sain]

Unfortunately, Win32-OpenSSH does not support ControlMaster so for those of us suck running VSCode on Windows there does not seem to be a workaround at the moment.

[rcarmo]

@nathan-sain Well, I use the WSL SSH client for everything, so maybe that is feasible (not near any Windows machines right now).

[rcarmo]

Update: I created ~/.ssh/sockets on my Mac, set my ~/.ssh/config to the _exact_ contents of the document (plus the Mac specific options for the keychain), and no, it doesn't work for me.

[jrobsonchase]

I've found that this works consistently for both ssh and gpg forwarding:

RemoteForward /run/user/2000/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent
RemoteForward /run/user/2000/gnupg/S.gpg-agent.ssh /run/user/1000/gnupg/S.gpg-agent.ssh
SetEnv SSH_AUTH_SOCK=/run/user/2000/gnupg/S.gpg-agent.ssh

Since the socket is always in the same place, you don't have to worry about SSH_AUTH_SOCK only being valid for a single ssh session. It's not quite as "clean" as the ForwardAgent option, but it seems to get the job done.

[rcarmo]

@jobsonchase those run paths are for Linux, right? I’m going to have a go at fiddling with SetEnv, but I have a feeling that fiddling with SSH configs risks breaking other tools (like Ansible, like I mentioned above).

[jrobsonchase]

@rcarmo Correct, specifically the flavors that have switched to using the XDG_RUNTIME_DIR rather than GNUPGHOME for the sockets. They're generally the ones that also start the agent with systemd --user, Arch in my case.

It's kind of unfortunate that there's no way (that I've found at least) to figure out the UID of the remote user to know where to put the sockets, or a way to specify the gpg-agent socket location like there is for the ssh-agent socket. You could put the S.gpg-agent.ssh (or whatever ssh agent socket you're actually using) in an easier-to-find location like /tmp/<username>/ssh-agent-socket, but I believe gpg removed the ability to specify a different socket path, so you're stuck putting it where it's expected to be.

On the topic of Ansible, I doubt this config would break anything - it's effectively the same as adding ForwardAgent yes to your config, just with a consistent socket path.

[singerb]

+1 for supporting agent forwarding; right now I commit on my remote sandbox machine with git CLI, since I have to run pre-commit hooks and it's easier to just use the environment there than to try and replicate it locally (getting a node_modules/ and.git/` folder synchronized with file sync tools is a nightmare). Being able to use the VS Code git UI to commit remotely and run the hooks in that environment would be great, but it relies on agent forwarding, naturally.

[benliddicott]

Workaround for Windows client, Linux server

I'm working around this as follows. @roblourens something similar may work for the product?

Connecting from code-insiders, I do not configure agent forwarding. Instead I have a single additional connection with agent forwarding (from a command prompt, but I suppose PuTTY would also work), and use a symbolic link to direct vscode to use that.

At the top of my .bashrc on the destination server:

if [ -z "$SSH_AUTH_SOCK" -o "$SSH_AUTH_SOCK" = "$HOME/.ssh/auth_sock" ] ; then
  # vscode connection
  # Direct vscode to use symlink, which we can set later.
  export SSH_AUTH_SOCK=$HOME/.ssh/auth_sock
else
  # agent forwarding connection. 
  # Direct agent symlink to the real port.
  ln -fs $SSH_AUTH_SOCK $HOME/.ssh/auth_sock
fi

It still has the shortcoming that it only allows a single agent forwarded to any destination.

Edit: I notice X11 forwarding also doesn't work, but using PuTTY and setting DISPLAY solves that.
Edit 2: Fixed case where it was occasionally overwritten

[ardikaveh]

ControlMaster method also doesn't work when connecting from multiple machines to the same remote server(work, home). I have to reboot the remote server for it to work again.

Why can't the terminal not automatically connect to the remote machine? If it behaved as it does without remote ssh there wouldn't be an issue.

[AgSync-Aaron]

I too am unfortunately running into this issue as well. I don't mean to pile on, I just don't have much to add to the conversation. If I can help test, or get you some other specific setup information I will.

When opening an ssh connection from a command prompt the agent forwarding works great. Just not through VS Code.

---

OpenSSH: OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Version: 1.34.20-insider (user setup)
Commit: d6aa8eaf30c71df32d4543290675808440def630
Date: 2019-05-09T22:10:02.019Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17763

[Toxicable]

Also running into this issue I have this in my config file:

Host *
  ForwardAgent yes

Host LINUX_HOST
  User LINUX_USER
  HostName LINUX_HOST
  ForwardAgent yes
  LocalForward 127.0.0.1:4200 127.0.0.1:4200
  LocalForward 127.0.0.1:3000 127.0.0.1:3000

If I manually ssh ssh LINUX_HOST then I can use git on the other side - proving the agent is forwarded
If I remote with VSC then I am unable to use git

For now my work around is to use git in a seprate powershell window but it would be awesome be able to use git inside VS Code

[TomFrost]

@Toxicable and others who can't get this to work at all -- per the above, add these lines to your ssh config:

    ControlMaster auto
    ControlPath  /tmp/%r@%h-%p
    ControlPersist  600

Then in VSCode, Open the command palette (Cmd+Shift+P) and choose "Remote-SSH: Kill VS Code Server on Host".

A better solution (particularly one that supports multiple machines, like work + home) would be great, but for now this is an adequate workaround. It just requires that you re-kill the server if your connection drops for more than 10 minutes, or whatever your ControlPersist is set to.

From: https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods

[rcarmo]

I hesitate to recommend the ControlPersist workaround given its blocking behaviour. In fact, I disagree with it being a workaround unless it's set to something much shorter...

[benliddicott]

@TomFrost @Toxicable @AgSync-Aaron @rcarmo @ardikaveh @jrobsonchase
I have a workaround I am using daily, with no issues, see my comment above. I'm using it from Windows->Linux but there is no reason it shouldn't work for Linux->Linux or another combination.

[TomFrost]

Note that as of recently, the ControlMaster solution is no longer working for me. Perhaps a recent update changed that functionality?

[vilic]

It seems that the -N option is not compatible agent forwarding, which actually eliminates potential workaround without an extra connection outside of the extension.

[benliddicott]

@vilic I just get on with my work. So I need an extra connection outside vscode? So what? It works.

[vilic]

@benliddicott We are trying to integrate remote development into our daily workflow, so we are trying to make it as easy to use as possible. If the second connection is not provided with -N option, a modified version of your script would just work without the user setting up another connection. Isn't that a better situation?

[PatrickLang]

+1

My private keys are derived from GPG and stored on my Yubikey. I need SSH agent forwarding to:

1. Authenticate to GitHub using ssh
2. Connect to my deployed test machines using the terminal window

Normally I accomplish this with ssh -A user@remote to enable agent forwarding.

[roblourens]

The main output of this for this month is an extension api proposal that we give remote resolvers a way to say that the new extension host should be started with some environment. Example:

export class ResolvedAuthority {
    readonly host: string;
    readonly port: number;
    readonly remoteEnv?: { [key: string]: string };

    constructor(host: string, port: number, remoteEnv?: { [key: string]: string });
}

This is the type a resolver returns, and it should be able to give a list of env vars for the new EH.

This is also used for reconnection so we would need to implement this for the initial EH creation and also for reconnecting to a running EH. For agent forwarding, both would be necessary. When we reconnect and open a new tunnel, we will get a new SSH_AUTH_SOCK and we would need to patch this in the EH.

[jkjl]

Combining @TomFrost's and @benliddicott's workarounds to avoid the need for an extra connection and having to kill the server after the connection drops:

SSH config:

Host *
    ControlMaster auto
    ControlPath /tmp/%r@%h-%p
    ControlPersist 5

At the top of the remote bashrc:

# only when piping a script to bash
if [ "$-" = hBc ] && [ "${BASH_EXECUTION_STRING:-}" = bash ] \
    && [ -S "${SSH_AUTH_SOCK:-}" ] && ! [ -L "$SSH_AUTH_SOCK" ]
then
    ssh_auth_sock_fixed="$HOME/.ssh/ssh_auth_sock"
    [ "$(SSH_AUTH_SOCK="$ssh_auth_sock_fixed" ssh-add || echo $?)" = 2 ] \
        && ln -snf -- "$SSH_AUTH_SOCK" "$ssh_auth_sock_fixed"
    export SSH_AUTH_SOCK="$ssh_auth_sock_fixed"
fi

Tested on macOS 10.14 and Ubuntu 18.04 clients and Ubuntu 16.04 server. The bashrc-part is only needed when connecting after the initial connection has dropped, but it has to be there before launching the server.

I hesitate to recommend the ControlPersist workaround given its blocking behaviour. In fact, I disagree with it being a workaround unless it's set to something much shorter...

ControlPersist=2 seemed to work reliably, but it probably depends on the client/server/connection lag.

It seems that the -N option is not compatible agent forwarding, which actually eliminates potential workaround without an extra connection outside of the extension.

Yes, the agent is not forwarded when using -N unless a connection with forwarding is reused.

[zobo]

(me too). Would love to see this solver, although I see its not as straight forward. My current use-case just touches the terminal (bash, git push in terminal) so what I do is just
export SSH_AUTH_SOCK=/tmp/ssh-(one of the existing socks).

I'm guessing a solution would be to not allow the vscode server to keep running outside an active SSH connection. I understand this changes some use cases, but as an option, perhaps?

Another thing I'd suggest or try is bundling a custom SSH client (perhaps based on https://godoc.org/golang.org/x/crypto/ssh) that could do multiplexing. That would be useful if each terminal would actually live inside a separate ssh session, but I'm seeing terminals and others are actually forks from the vscode-server.

A third, rather simple option would be to symlink a fake SSH_AUTH_SOCK.

1. vscode-server starts, gets current enviroment.
2. creates a /tmp/ssh-vscodeRANDOM/agent.pid symlink to current SSH_AUTH_SOCK and changes SSH_AUTH_SOCK to ssh-vscode... before passing it to other child processes.
3. ssh connection drops, new ssh connection opens
4. new connection (after binding to vscode-server) updates /tmp/ssh-vscodeRANDOM/agent.pid to new SSH_AUTH_SOCK

This way there is no need to manipulate ENV of any child processes..

I haven't done any deep dives into the vscode-server code to know if I'm missing a big wall or something, but let me just say that what you did here is amazing. For me (Win client, Linux workspace) this is the best of both worlds, experience like no other...

[ryansully]

Expanding on @benliddicott's workaround, I combined some of his code with a fix I previously had for getting SSH agent forwarding to work in tmux.

Add the following code to their respective files on your host *nix system:

~/.ssh/rc

#!/usr/bin/env bash

# Fix SSH auth socket location so agent forwarding works with tmux and VS Code
if test "$SSH_AUTH_SOCK"; then
  ln -sf $SSH_AUTH_SOCK ~/.ssh/auth_sock
fi

~/.bashrc (or whatever shell init file you prefer, e.g. ~/.bash_profile, ~/.profile, etc)

export SSH_AUTH_SOCK=$HOME/.ssh/auth_sock

With this I got agent forwarding working on a Linux host from Windows/Mac/Linux clients. If there's any concern or issue with this I'm happily open to suggestions.