Vscode-remote-release: Remote - Containers over SSH (to remote container host)

[ItalyPaleAle]

• VSCode Version: 1.50.1
• Local OS Version: macOS 10.15.7
• Remote OS Version: Debian 10
• Remote Extension/Connection Type: Docker

Steps to Reproduce:

1. Set a docker.host variable to a remote host via SSH (e.g. ssh://user@host), as per documentation: https://code.visualstudio.com/docs/remote/containers-advanced#_a-basic-remote-example . I am using a SSH key (if it matters, it's a SSH key part of my GPG key, in a YuibiKey as per this guide)
2. Confirm that VS Code can connect to the remote Docker host: in the Docker extension tab, I can see all the containers running the remote host
3. In the Remote extension tab, select "Container" and hit the "reload" button. After minutes, the list of containers still doesn't appear, and the progress indicator still keeps moving. I can also see my YubiKey's LED flashing, which means that it's constantly trying to (re-?) connect to the SSH host.

Does this issue occur when you try this locally?: N/A
Does this issue occur when you try this locally and all extensions are disabled?: N/A

[chrmarti]

Have you tried reloading the window?

[ItalyPaleAle]

@chrmarti Yes I tried reloading the window and also re-launching VS Code

[chrmarti]

Could you append the log from when you are trying to connect? (F1 > Remote-Containers: Show Log)

[chrmarti]

One issue could be that using Docker with the ssh:// protocol has to reestablish an SSH connection each time we run the Docker CLI and that could have too much overhead depending on network connection speed and the time it takes to authenticate with the YuibiKey.

If the log doesn't show obvious errors, could you try using an SSH tunnel, that will create a single SSH connection and reuse that: https://code.visualstudio.com/docs/remote/troubleshooting#_using-an-ssh-tunnel-to-connect-to-a-remote-docker-host

[chrmarti]

There also appears to be a way to share the first SSH connection by configuring SSH to do so: https://stackoverflow.com/a/30242522/7857763. Haven't tried that yet.

[chrmarti]

Following https://puppet.com/blog/speed-up-ssh-by-reusing-connections/, I have added:

Host *
ControlMaster auto
ControlPath ~/.ssh/sockets/%r@%h-%p
ControlPersist 600

And created the sockets folder manually.

That brought the time it takes for the Remote-Containers resolver to run with a local SSH server down from around 4.2 to around 2.3 seconds. (Enabling 'Trace' logging for Remote-Containers shows a [2319 ms] Stop (2299 ms): Resolving remote message in the Dev Container log.)

@ItalyPaleAle Could you give that a try?

[ItalyPaleAle]

Hi @chrmarti sorry for the delay, I was OOF Mon-Wed and catching up yesterday.

The first update is that it actually does work, it's just very slow. Watching the logs, I saw that it is working, but it took a couple of minutes before it showed the results (actually 2 minutes, per timing in the logs).

Logs

  [6 ms] Remote-Containers 0.145.1 in VS Code 1.50.1 (d2e414d9e4239a252d1ab117bd7067f125afd80a).
  [32163 ms] Start: Run: docker version --format {{.Server.Version}}
  [37376 ms] 19.03.13
  [37377 ms] Start: Run: docker events --format {{json .}} --filter event=create --filter event=destroy --filter event=die --filter event=health-status --filter event=kill --filter event=pause --filter event=rename --filter event=resize --filter event=restart --filter event=start --filter event=stop --filter event=unpause --filter event=update --filter event=exec_create
  [37381 ms] Start: Run: docker ps -q -a
  [46471 ms] Start: Run: docker inspect --type container 9cf478d8559b a892c39e6775 e1d7b151a0e5
  [54240 ms] Start: Run: git rev-parse --show-toplevel
  [54251 ms] Start: Run: git rev-parse --show-toplevel
  [54260 ms] Start: Run: git rev-parse --show-toplevel
  [54264 ms] Start: Run: git rev-parse --show-toplevel
  [54279 ms] Start: Run: git rev-parse --show-toplevel
  [54281 ms] Start: Run: git rev-parse --show-toplevel
  [54304 ms] Start: Run: docker-compose version --short
  [54760 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=***_devcontainer --filter label=com.docker.compose.service=development
  [62344 ms] Start: Run: docker inspect --type container /***_dev_1
  [69851 ms] Start: Run: git rev-parse --show-toplevel
  [69869 ms] Start: Run: docker-compose version --short
  [70300 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=devcontainer_devcontainer --filter label=com.docker.compose.service=dev
  [78021 ms] Start: Run: git rev-parse --show-toplevel
  [78039 ms] Start: Run: docker-compose version --short
  [78472 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=devcontainer_devcontainer --filter label=com.docker.compose.service=dev
  [86051 ms] Start: Run: git rev-parse --show-toplevel
  [86069 ms] Start: Run: docker-compose version --short
  [86497 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=***_devcontainer --filter label=com.docker.compose.service=dev
  [93938 ms] Start: Run: git rev-parse --show-toplevel
  [93956 ms] Start: Run: docker-compose version --short
  [94390 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=***_devcontainer --filter label=com.docker.compose.service=dev
  [101853 ms] Start: Run: git rev-parse --show-toplevel
  [101871 ms] Start: Run: docker-compose version --short
  [102304 ms] Start: Run: docker ps -q -a --filter label=com.docker.compose.project=***_devcontainer --filter label=com.docker.compose.service=dev
  [109730 ms] Start: Run: git rev-parse --show-toplevel
  [109737 ms] Start: Run: git rev-parse --show-toplevel
  [109753 ms] Start: Run: docker ps -q -a
  [117403 ms] Start: Run: docker inspect --type container 9cf478d8559b a892c39e6775 e1d7b151a0e5
  [125177 ms] Start: Run: docker inspect --type container 9cf478d8559b055e3c57391cec3f6db8629a22cca3e11002d6752c0690204f4e
  [125177 ms] Start: Run: docker inspect --type container a892c39e677540b45926db80648c28d0024f9953f9e9de6e7d8e29d9d84820d0
  [125178 ms] Start: Run: docker inspect --type container e1d7b151a0e5636d44bce8ee8610518605491d14ba378d74a9b3861992df67a7

Now, onto the solutions you proposed...

SSH tunnel

Using an external SSH tunnel seems to work and is much faster. I start a tunnel with:

# The Docker daemon is listening on 127.0.0.1:2379 on the remote server
ssh -N -v -L 127.0.0.1:2379:127.0.0.1:2379 ***@***

Then setting docker.host to localhost:2379.

This shows the containers in a bit less than 15 seconds.

Re-using SSH connections

Following your instructions above, enabling ControlMaster does seem to be working too and lowers the time to about 25 seconds.

Here's how I configured it:

Adding this to my .ssh/config:

Host foo.com
  User dev
  RemoteForward /run/user/1000/gnupg/S.gpg-agent /Users/alessandro/.gnupg/S.gpg-agent.extra
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600

Then running mkdir .ssh/sockets

VS Code is configured with docker.host set to ssh://[email protected]

Moral of the story

Sounds like this was always working but was just incredibly slow. Thanks for pointing me to workarounds.

I would personally prefer the second workaround (using ControlMaster) because it does not depend on a SSH tunnel I create in a separate terminal window, but instead lets VS Code manage the SSH connection. I will rely on that.

In both cases, I only tested this to confirm "I can see the containers" and haven't spent much time coding inside the dev containers in a remote SSH host. I'll play with that more tonight.

Not sure if there's something that can be done however to make this faster (less than 2 minutes) without workarounds. Somehow, the Docker extension seems to not have issues and it responds in seconds.

[chrmarti]

Fixing by reducing the number of roundtrips to the remote Docker daemon. Performance should become comparable to the Docker extension.

The SSH options still make sense because we need to contact the Docker daemon several times when creating and attaching to a container. We can add that to our documentation (filed #3938 to track).

[chrmarti]

@ItalyPaleAle Could you verify that the performance of the Remote-Explorer has improved?

[ItalyPaleAle]

Confirming the performance has greatly improved and it's now usable for my scenario too. Thanks!

[rzhao271]

Marking it as verified per comment above ^