Vscode-remote-release: SSH agent fails when attached to running container on remote host

[markedwards]

Issue Type: Bug

1. Attach to running container using a remote docker host. In my case the remote is an EC2 machine running Amazon Linux, running a python:3.8-slim-buster container.
2. Observe that the ssh agent is working by opening a terminal and testing with ssh -v [email protected]. My local ssh key is available in the container, and authentication succeeds.
3. After some time (appears to be roughly 60 minutes), communication with the ssh agent begins to hang indefinitely at debug1: rekey after 134217728 blocks, and the command has to be terminated with control-c.
4. After some additional time, communication with the agent fails outright and the command fails with permission denied, due to no valid keys being available in the container.
5. Beyond step 3, no successful agent communication is possible until the VSCode workspace is closed and a new connection to the container is established as per step 1.

VS Code version: Code 1.44.0 (2aae1f26c72891c399f860409176fe435a154b13, 2020-04-08T08:23:56.137Z)
OS version: Darwin x64 19.4.0
Remote OS version: Linux x64 4.14.171-136.231.amzn2.x86_64

System Info

|Item|Value|
|---|---|
|CPUs|Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz (4 x 2300)|
|GPU Status|2d_canvas: enabled
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
metal: disabled_off
multiple_raster_threads: enabled_on
oop_rasterization: disabled_off
protected_video_decode: unavailable_off
rasterization: enabled
skia_renderer: disabled_off_ok
video_decode: enabled
viz_display_compositor: enabled_on
viz_hit_test_surface_layer: disabled_off_ok
webgl: enabled
webgl2: enabled|
|Load (avg)|2, 2, 2|
|Memory (System)|8.00GB (0.34GB free)|
|Process Argv||
|Screen Reader|no|
|VM|0%|

|Item|Value|
|---|---|
|Remote|Container 829264618005.dkr.ecr.us-east-…|
|OS|Linux x64 4.14.171-136.231.amzn2.x86_64|
|CPUs|AMD EPYC 7571 (2 x 2199)|
|Memory (System)|3.81GB (1.31GB free)|
|VM|0%|

Extensions (9)

Extension|Author (truncated)|Version
---|---|---
vscode-docker|ms-|1.0.0
remote-containers|ms-|0.112.0
remote-ssh|ms-|0.51.0
remote-ssh-edit|ms-|0.51.0
remote-wsl|ms-|0.44.2
vscode-remote-extensionpack|ms-|0.20.0
python|ms-|2020.3.71659
vscode-graphql|Pri|0.2.14
vscodeintellicode|Vis|1.2.6

[markedwards]

Here is sample output when communication with the agent hangs:

$ ssh -v [email protected]
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to bitbucket.org [18.205.93.0] port 22.
debug1: Connection established.
debug1: identity file /home/gidev/.ssh/id_rsa type -1
debug1: identity file /home/gidev/.ssh/id_rsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_dsa type -1
debug1: identity file /home/gidev/.ssh/id_dsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_ecdsa type -1
debug1: identity file /home/gidev/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_ed25519 type -1
debug1: identity file /home/gidev/.ssh/id_ed25519-cert type -1
debug1: identity file /home/gidev/.ssh/id_xmss type -1
debug1: identity file /home/gidev/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version conker_405f5f426f-dirty celery-worker-112
debug1: no match: conker_405f5f426f-dirty celery-worker-112
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:...
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/gidev/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
^C
$ 

[markedwards]

Note that when this happens, SSH_AUTH_SOCK is still set in the container shell, and it points to an existing socket beginning with /tmp/vscode-ssh-auth-. Also, agent communication remains perfectly fine in the remote host itself throughout. The issue appears to be limited to the specific VSCode container connection.

Also, a colleague running the exact same VSCode version, remote host EC2 AMI, and docker image does not experience this issue. He is running OS X 10.14.6 whereas I'm running 10.15.4, although I have no idea if that is the significant difference.

[chrmarti]

Does reloading the VS Code window make it work again? (F1 > Developer: Reload Window)

[markedwards]

Does reloading the VS Code window make it work again? (F1 > Developer: Reload Window)

Yes, that works temporarily (same result as step #5 in my repro above).

[markedwards]

@chrmarti, anything I can do to provide further information or help debug this?

[chrmarti]

Not at the moment. I have improved logging in that area, but that change first needs to be published in a new version of the extension.

[markedwards]

@chrmarti Happy to run your logging changes locally if it would help.

[markedwards]

Is it possible to push this investigation forward yet? Thanks.

[chrmarti]

We'll have a new version for testing next week. Thanks.

[vscodebot[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[markedwards]

Just pinging this. The issue shouldn't really be marked as closed, as we are waiting for a release to investigate further.

[chrmarti]

@markedwards Thanks. Could you retry with 0.118.0 or later. This currently requires VS Code Insiders (https://code.visualstudio.com/insiders/).

[markedwards]

I'm set up with 0.118.0 and waiting to see the issue appear. Is there some logging in particular you want?

[markedwards]

Definitely the same behaviour in Insiders as in the current Code release (but I think that's expected). Let me know how I can help debug this.

[chrmarti]

I expect there to be some error(s) in the Dev Containers log at the time the SSH agent connection stops working. (F1 > Remote-Containers: Show Log.)

[markedwards]

Below is a log, but I don't see any errors. At some point during the long list of "Start: Run:..." lines the SSH agent stopped working.

[24 ms] Start: Resolving remote
[26 ms] Setting up container /me_platform_1

[27 ms] Start: Run: docker inspect --type container /me_platform_1
[1822 ms] Start: Check Docker is running
[1822 ms] Start: Run: docker info
[5188 ms] Start: Inspecting container
[5188 ms] Start: Run: docker inspect --type container c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81
[6764 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh
[6767 ms] Start: Run in container: uname -m
[10335 ms] x86_64
[10335 ms] 
[10335 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[10429 ms] PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
[10429 ms] 
[10430 ms] Start: Run in container: cat /etc/passwd
[10524 ms] Start: Run in container: test -d /home/gidev/.vscode-server-insiders
[10620 ms] 
[10620 ms] 
[10620 ms] Exit code 1
[10621 ms] Start: Run in container: test -d /home/gidev/.vscode-remote
[10712 ms] 
[10713 ms] 
[10713 ms] Exit code 1
[10713 ms] Start: Run in container: set -o noclobber ; mkdir -p '/home/gidev/.vscode-server-insiders/data/Machine' && { > '/home/gidev/.vscode-server-insiders/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[10804 ms] 
[10804 ms] 
[10804 ms] Start: Run in container: mkdir -p '/home/gidev/.vscode-server-insiders/data/Machine' && cat >'/home/gidev/.vscode-server-insiders/data/Machine/settings.json' <<'settingsJSON'
[10896 ms] 
[10896 ms] 
[10897 ms] Start: Run in container: test -d /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672
[10992 ms] 
[10993 ms] 
[10993 ms] Exit code 1
[10993 ms] Installing VS Code Server for commit 1bfa086adb9271ff56be11821da686e4cfffb672
[10993 ms] Start: Run in container: mkdir -p /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672_1591093536468
[11100 ms] 
[11100 ms] 
[11101 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672_1591093536468 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 tar --no-same-owner -xz --strip-components 1
[33056 ms] 
[33057 ms] 
[33057 ms] Start: Run in container: mv -n /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672_1591093536468 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672
[33155 ms] 
[33155 ms] 
[33155 ms] Start: Launching Remote-Containers helper.
[33156 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[33163 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-040a639fc146a3385d02802878a3e79e215ae665.js' >/tmp/vscode-remote-containers-040a639fc146a3385d02802878a3e79e215ae665.js
[33258 ms] 
[33258 ms] 
[33258 ms] Start: Run in container: # Copy /Users/markedwards/.gitconfig to /home/gidev/.gitconfig
[33352 ms] 
[33352 ms] 
[33353 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-040a639fc146a3385d02802878a3e79e215ae665.js' >/tmp/vscode-remote-containers-server-040a639fc146a3385d02802878a3e79e215ae665.js
[33568 ms] 
[33568 ms] 
[33568 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --global credential.helper '!f() { /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node /tmp/vscode-remote-containers-040a639fc146a3385d02802878a3e79e215ae665.js $*; }; f' || true
[33569 ms] Start: Run: docker exec -i -u gidev -e REMOTE_CONTAINERS_SOCKETS=["/tmp/vscode-ssh-auth-040a639fc146a3385d02802878a3e79e215ae665.sock"] -e REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-040a639fc146a3385d02802878a3e79e215ae665.sock c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node /tmp/vscode-remote-containers-server-040a639fc146a3385d02802878a3e79e215ae665.js
[33665 ms] 
[33666 ms] 
[33666 ms] Start: Run in container: set -o noclobber ; mkdir -p '/home/gidev/.vscode-server-insiders/data/Machine' && { > '/home/gidev/.vscode-server-insiders/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[33772 ms] 
[33772 ms] 
[33776 ms] Start: Run in container: test -d /home/gidev/.vscode-server-insiders/extensionsCache && ls /home/gidev/.vscode-server-insiders/extensionsCache || true
[33867 ms] 
[33867 ms] 
[33871 ms] Start: Run in container: cd /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672; export VSCODE_AGENT_FOLDER=/home/gidev/.vscode-server-insiders; /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/server.sh --extensions-download-dir /home/gidev/.vscode-server-insiders/extensionsCache --install-extension eamodio.gitlens --install-extension ms-python.python --install-extension Prisma.vscode-graphql --install-extension VisualStudioExptTeam.vscodeintellicode --force
[48011 ms] Installing extensions...
Installing extension 'eamodio.gitlens' v10.2.1...
Extension 'eamodio.gitlens' v10.2.1 was successfully installed.
Installing extension 'ms-python.python' v2020.5.80290...
Extension 'ms-python.python' v2020.5.80290 was successfully installed.
Installing extension 'prisma.vscode-graphql' v0.2.14...
Extension 'prisma.vscode-graphql' v0.2.14 was successfully installed.
Installing extension 'visualstudioexptteam.vscodeintellicode' v1.2.7...
Extension 'visualstudioexptteam.vscodeintellicode' v1.2.7 was successfully installed.
[48011 ms] 
[48012 ms] Start: Run in container: ls /home/gidev/.vscode-server-insiders/extensionsCache
[48115 ms] eamodio.gitlens-10.2.1
ms-python.python-2020.5.80290
prisma.vscode-graphql-0.2.14
visualstudioexptteam.vscodeintellicode-1.2.7
[48115 ms] 
[48115 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 tar c extensionsCache/eamodio.gitlens-10.2.1 extensionsCache/ms-python.python-2020.5.80290
[74447 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink -f /proc/$pid/cwd ; xargs -0 < /proc/$pid/environ ; xargs -0 < /proc/$pid/cmdline ; } ; echo ; done 2>/dev/null
[74643 ms] Start: Run: docker exec -i -u gidev -e SHELL=/bin/sh -e VSCODE_AGENT_FOLDER=/home/gidev/.vscode-server-insiders -w /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/server.sh --disable-user-env-probe --port 0 --extensions-download-dir /home/gidev/.vscode-server-insiders/extensionsCache
[78589 ms] 

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*


[78644 ms] IP Address: 192.168.160.5
[78648 ms] Extension host agent listening on 43263


[78648 ms] Start: Run in container: echo 43263 >/home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/.devport
[78649 ms] [10:26:44] Extension host agent started.
[78747 ms] 
[78747 ms] 
[78751 ms] Forwarding local port 43263 to container port 43263
[78752 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[78753 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[78753 ms] 

-----------------------
Published Ports:
80/tcp -> 0.0.0.0:32875
-----------------------

[78781 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[78789 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[83810 ms] [10:26:49] [::ffff:127.0.0.1][481eb556][ExtensionHostConnection] New connection established.
[83816 ms] [10:26:49] [::ffff:127.0.0.1][481eb556][ExtensionHostConnection] <208> Launched Extension Host Process.
[84792 ms] [10:26:50] [::ffff:127.0.0.1][b1fe9b31][ManagementConnection] New connection established.
[90555 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[114823 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[114829 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[114834 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[114840 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[117574 ms] Start: Run: docker exec -i -u gidev -e VSCODE_REMOTE_CONTAINERS_SESSION=908d8bff-906c-4135-9635-af2be8c33fd11591093524099 c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /home/gidev/.vscode-server-insiders/bin/1bfa086adb9271ff56be11821da686e4cfffb672/node -e 
[143818 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[144586 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[208024 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[208729 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[271976 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[272738 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[335981 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[338042 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[401472 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[402588 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[465434 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[466432 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[529341 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[530448 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[593484 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[595587 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[657834 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[660007 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[722339 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[724441 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[786435 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[788871 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[850824 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[853029 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[915070 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[917035 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[979226 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[981143 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1043406 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1045373 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1108182 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1110263 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1172228 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1174323 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1236439 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1238554 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1300505 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1303059 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1366139 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1368261 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1431189 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1432528 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1495257 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1496632 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1559125 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1560523 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1623372 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1624522 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1687935 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1688822 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1753233 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1753889 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1817938 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1818574 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1882516 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1883083 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[1946792 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[1947319 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2010759 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2011276 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2074821 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2075435 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2139404 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2140256 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2203358 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2204315 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2267915 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2269224 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2333236 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2333695 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2397525 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2397891 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2461920 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2462465 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2526110 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2526744 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2591206 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2591805 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2655200 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2655942 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2719398 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2719981 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2783344 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2784000 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2847294 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2848023 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2911664 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2912276 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[2975596 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[2976311 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3039917 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3040427 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3104014 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3104483 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3168047 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3168471 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3233773 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3234117 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3298710 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3299161 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3362853 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3363228 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3427004 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3427370 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3491508 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3492011 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3555498 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3556052 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3619630 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3620051 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3683530 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3684041 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3747848 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3748419 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3812138 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3812782 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3876419 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3877120 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[3940939 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[3941598 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4005428 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4006054 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4069938 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4070594 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4134636 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4135336 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4199220 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4199726 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4263444 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4264020 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4327489 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4328084 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4391702 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4392682 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4456708 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4457441 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4520600 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4521344 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4584551 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4585168 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4648413 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4648960 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4712347 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4712894 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4776371 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4776856 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4840317 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4840821 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4904195 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4904796 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[4968380 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[4969025 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings
[5032448 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/extensions c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch installed extensions
[5033008 ms] Start: Run: docker exec -i -u gidev -w /home/gidev/.vscode-server-insiders/data/Machine c6b9d1422a2fc8d7feb8200b85d951dc073c7d2a8cee896fe8dcd24281deba81 /bin/sh -c # Watch machine settings

[chrmarti]

Is there anything in the Dev Console? (Help > Toggle Developer Tools)

Is the /tmp/vscode-ssh-auth-*.sock file still there in the container?

What is the output of ssh when it is still working? (To compare with https://github.com/microsoft/vscode-remote-release/issues/2731#issuecomment-612074351.)

[markedwards]

Output of ssh when its still working:

$ ssh -v [email protected]
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to bitbucket.org [18.205.93.0] port 22.
debug1: Connection established.
debug1: identity file /home/gidev/.ssh/id_rsa type -1
debug1: identity file /home/gidev/.ssh/id_rsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_dsa type -1
debug1: identity file /home/gidev/.ssh/id_dsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_ecdsa type -1
debug1: identity file /home/gidev/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/gidev/.ssh/id_ed25519 type -1
debug1: identity file /home/gidev/.ssh/id_ed25519-cert type -1
debug1: identity file /home/gidev/.ssh/id_xmss type -1
debug1: identity file /home/gidev/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version conker_405f5f426f-dirty app-132
debug1: no match: conker_405f5f426f-dirty app-132
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/gidev/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: Will attempt key: id_rsa RSA SHA256:IPnMniwuBpjJXvwFW0v30hJl/uL42ZJXcBpcIp8nXK0 agent
debug1: Will attempt key: /home/gidev/.ssh/id_rsa 
debug1: Will attempt key: /home/gidev/.ssh/id_dsa 
debug1: Will attempt key: /home/gidev/.ssh/id_ecdsa 
debug1: Will attempt key: /home/gidev/.ssh/id_ed25519 
debug1: Will attempt key: /home/gidev/.ssh/id_xmss 
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: id_rsa RSA SHA256:IPnMniwuBpjJXvwFW0v30hJl/uL42ZJXcBpcIp8nXK0 agent
debug1: Server accepts key: id_rsa RSA SHA256:IPnMniwuBpjJXvwFW0v30hJl/uL42ZJXcBpcIp8nXK0 agent
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([18.205.93.0]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
PTY allocation request failed on channel 0
logged in as earth2mark

You can use git or hg to connect to Bitbucket. Shell access is disabled
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to bitbucket.org closed.
Transferred: sent 3000, received 1852 bytes, in 0.0 seconds
Bytes per second: sent 582795.5, received 359779.1
debug1: Exit status 0
$

[markedwards]

The /tmp/vscode-ssh-auth-*.sock is still there. See comment https://github.com/microsoft/vscode-remote-release/issues/2731#issuecomment-612084843.

[markedwards]

Unfortunately the dev console doesn't have anything interesting, as far as I can tell:

All of those log lines occurred prior to the ssh agent starting to fail.

[chrmarti]

Does it help if you restart the ssh-agent on your machine?

[markedwards]

I tried kill -HUP on the ssh-agent process, and that made ssh-agent stop working in all workspaces, not the just the container workspaces. And using Developer: Reload Window no longer worked, I had to restart VSCode to get ssh working at all.

Did you mean for me to try something else?

[chrmarti]

You would have to restart the ssh-agent. I realize that it picks a new socket path on each start, so you would also have to tell it to bind to the old socket path where the connections from the container are forwarded to. This worked for me:

kill <ssh-agent pids>
rm $SSH_AUTH_SOCK
ssh-agent -a $SSH_AUTH_SOCK
ssh-add <github key>

Given that you see other VS Code windows with Remote-Containers still working fine with the existing ssh-agent process, this might not help.

Could you also check the max number of open files in the container? Run: ulimit -n in the container.

[markedwards]

Okay, using your methodology, the ssh agent still works on the SSH workspace, but it does not make the agent work in the container workspaces.

ulimit -n returns 4096 in both the containers and the server. I doubt this as a source, because this exact setup does not produce the issue on OS X Mojave, only on Catalina. And, like I said, this occurs on all container workspaces and not on an SSH workspace, despite everything having the same ulimit.

So, just to recap:

• OS X Catalina attach to container running on remote host: SSH FAILS
• OS X Catalina attach to remote host via SSH: NO ISSUE
• OS X Catalina attach to container running local host: NO ISSUE
• OS X Mojave attach to container running on remote host: NO ISSUE

[chrmarti]

Following both leads since both might be dead-ends:

4096 max files might be low depending on what else you are running. What do you get for lsof | wc -l in the container when SSH starts failing?

How do you connect to the remote Docker daemon? (ssh:// URI, tcp:// URI, SSH tunnel?)

What versions do the remote Docker daemon and the local Docker daemon have? (Shown by docker version.)

[markedwards]

In a container (with SSH is not working):

$ ulimit -n
4096
$ lsof | wc -l
6563

In a container (while SSH is working):

$ ulimit -n
4096
$ lsof | wc -l
7905

In the host (SSH is working):

$ ulimit -n
4096
$ lsof | wc -l
54067

Remote Docker connection is made by setting ssh://{user}@{host} in VSCode docker.host, and using VSCode to attach.

The only place Docker is running is on the remote host, and the version is 19.03.6-ce. So, VSCode connects directly to the remote machine's Docker, and attaches to a container. I'm not running Docker on my local machine (there's no need).

For what its worth, I'm fairly confident this issue can be easily reproduced by following the steps in the issue description, since it reproduces consistently on multiple different machines and hosts on my side, and it happens every single time. And these are standard python:3.8-slim-buster containers, they are not really customised.

Also, apparently the issue reproduces on Mojave (according to a colleague), contrary to my claim above.

[markedwards]

Is there anything I should do to move this forward? I can sense the close-bot knocking at the door :-)

[chrmarti]

I need to investigate further, nothing you can do at the moment. The close-bot only acts when the last comment is from the dev team. (Like this one. :D)

[markedwards]

Okay, thanks for the update. Please ping again if I can help in any way.

[github-actions[bot]]

Hey @chrmarti, this issue might need further attention.

@markedwards, you can help us out by closing this issue if the problem no longer exists, or adding more information.

[markedwards]

This is still an issue as of 1.50.1.