Vscode-remote-release: When building containers via docker-compose and docker.host=ssh://, it fails with a "x.x.x.x not found in known_hosts" error

[terroirman]

• VSCode Version: 1.41.0
• Local OS Version: Win10 10.0.16299
• Remote OS Version: RHEL7 3.10.0-957.el7.x86_64
• Docker Desktop Version : 2.1.0.5 (40693)
• Remote Extension/Connection Type: Docker

Steps to Reproduce:

1. Add "docker.host": "ssh://[email protected]" to workspace settings in local folder
2. Use docker-compose to deploy containers on remote
3. Ensure docker-compose.yml has been modified (in order to trigger a "build" phase)
4. Hit F1 reopen in container

Note :

• If the containers are already built and up to date with the docker-compose.yml, the problem does not occur. Ie.: I can successfully connect to remote container (as long as I don't have to rebuild them via docker-compose)
• If the build process is done via ssh tunnel between the remote host's docker.sock and local machine (ie.: "docker.host": "tcp://localhost:23750) the problem doesn't occur

Does this issue occur when you try this locally?: No
Does this issue occur when you try this locally and all extensions are disabled?: Yes

Detailled output :

[1 ms] Start: Resolving remote
[3 ms] Setting up container for folder or workspace: c:\Users\xxxx\project_xyz

[14 ms] Start: Check Docker is running
[5295 ms] Start: Run: docker-compose -f c:\Users\xxxx\project_xyz\.devcontainer\docker-compose-dev.yml config --services
[8045 ms] Start: Run: docker-compose --project-name project_xyz_devcontainer -f c:\Users\xxxx\project_xyz\.devcontainer\docker-compose-dev.yml up -d --build
[8373 ms] 
[9967 ms] Connected (version 2.0, client OpenSSH_7.4)

[10091 ms] c:\Users\xxxx\AppData\Local\Temp\_MEI81002\paramiko\kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.

[10184 ms] c:\Users\xxxx\AppData\Local\Temp\_MEI81002\paramiko\kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
c:\Users\xxxx\AppData\Local\Temp\_MEI81002\paramiko\kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose\cli\main.py", line 71, in main
  File "compose\cli\main.py", line 124, in perform_command
  File "compose\cli\command.py", line 42, in project_from_options
  File "compose\cli\command.py", line 123, in get_project
  File "compose\cli\command.py", line 94, in get_client
  File "compose\cli\docker_client.py", line 127, in docker_client
  File "site-packages\docker\api\client.py", line 166, in __init__
  File "site-packages\docker\transport\sshconn.py", line 84, in __init__
  File "site-packages\docker\transport\sshconn.py", line 94, in _connect
  File "site-packages\paramiko\client.py", line 407, in connect
  File "site-packages\paramiko\client.py", line 809, in missing_host_key
paramiko.ssh_exception.SSHException: Server 'x.x.x.x' not found in known_hosts
[15144] Failed to execute script docker-compose

[10304 ms] Failed: docker-compose up
[10306 ms] Command failed: C:\Program Files\Docker\Docker\Resources\bin\docker-compose.exe --project-name project_xyz_devcontainer -f c:\Users\xxxx\project_xyz\.devcontainer\docker-compose-dev.yml up -d --build

[LiNuXaDDiKt]

I encounter the same issue

[chrmarti]

Similar to https://github.com/docker/compose/issues/6655

[vscodebot[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[chrmarti]

@terroirman @LiNuXaDDiKt Which version of docker-compose do you have installed? https://github.com/docker/compose/issues/6655#issuecomment-559723271 suggests this might be fixed with the latest version.

[yamlCase]

I am also getting this issue with docker-compose 1.25.0 (chocolatey package) and 1.25.1-rc1 (pip install)). Running the failing command locally will succeed, so I'm not sure we can blame docker-compose.

error: "paramiko.ssh_exception.SSHException: Server 'centos7' not found in known_hosts"

Windows 10

[chrmarti]

@yamlCase Not sure what the difference between the extension running it and you running it in the terminal might be. Did you set DOCKER_HOST to the ssh:// URL?

[yamlCase]

Yes that env is set... however I just realized I didn't put a FQDN and am relying on domain search. Will try again when I get a chance.

[terroirman]

@chrmarti as requested :
docker-compose version 1.24.1, build 4667896b

[terroirman]

Fyi, same error with docker-compose-1.25.0 paramiko-2.7.1 (manually installed with pip)

[terroirman]

@chrmarti , do you need more information on this ? I still see the "needs-more-info" label to this issue, so I want to be sure you have all you need.

[chrmarti]

The error message is cryptic. Is this just saying that the server is not in the known_hosts file? Can you ssh into the machine?

[yamlCase]

@chrmarti I am able to use the VSCode "Remote SSH" extension with success to the same host that fails with the "Remote Containers" extension.

[terroirman]

@chrmarti : of course I can ssh to it, and the host is in my know_host as well. This problem is easy to reproduce.

I'd be glad to help find the root cause of this. Can you point out a few hints on how to see more logs or the src code about this connection process ? I'd like to see how vscode is calling docker-compose for instance.

[rgeraskin]

Same error with docker-compose version 1.25.4, build 8d51620a installed with Docker Desktop @ Win10. But in my case, it happens with --build arg and without it too.

Version: 1.42.1 (system setup)
Commit: c47d83b293181d9be64f27ff093689e8e7aed054
Date: 2020-02-11T14:45:59.656Z
Electron: 6.1.6
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.17763

[chrmarti]

I'd expect Docker Compose to pick up the known hosts the same way ssh does.

It says Server 'x.x.x.x' not found in known_hosts, can you check if 'x.x.x.x' is in $HOME/.ssh/known_hosts? ssh should have added it.

[rgeraskin]

I'd expect it too. Server x.x.x.x is in $HOME/.ssh/known_hosts. Also, I can log in to the server via ssh without prompt.

[rgeraskin]

I've tested the VS Code with few docker-compose providers

• Docker Desktop 2.2.0.42716 (1.25.4, build 8d51620a)
• Chocolatey (1.25.4)
• pip install docker-compose (1.25.4)

And I have no error using the "pip install" docker-compose version. So it is a docker-compose build related problem in my case. I hope it'll help others.

[terroirman]

@roman-geraskin , this is weird, I still get the same error with "pip install" version (I uninstalled docker desktop first). Did you choose "Remote-containers: Rebuild and Reopen in container" to launch it from your local folder ? Because the error won't happen if you only choose "Remote-containers: Reopen in container"

Last result for me with "rebuild and reopen":

paramiko.ssh_exception.SSHException: Server 'x.y.z.a' not found in known_hosts
paramiko.transport._log: EOF in transport thread
[2020-03-02T14:57:06.522Z] [PID 19960] [15286 ms] Failed: docker-compose --verbose up
[2020-03-02T14:57:06.524Z] [PID 19960] [15288 ms] Command failed: C:\Users\xxxxxx\AppData\Local\Programs\Python\Python37\Scripts\docker-compose.exe --project-name test_devcontainer -f c:\Users\xxxxxx\repositories\test\.devcontainer\docker-compose-dev.yml up -d --build

docker-compose for reference:

PS C:\Users\xxxxx> docker-compose -v
docker-compose version 1.25.4, build unknown

[terroirman]

Another thing to note : executing the docker-compose command as reported by vscode manually works correctly (ie.: without "server x.x.x.x not found in known_hosts" error). This is why I still think vscode introduces something in compose/paramiko environment that makes it fail.

@chrmarti : is it possible to run the extension with more logs? It would be nice to see what known_host file paramiko is trying to read from.

[chrmarti]

There is currently no way to enable --verbose for docker-compose. Does that tell you which known_hosts file is used when you run it manually?

And just to be sure: Are you using the same user for running docker-compose manually as for running VS Code?

[terroirman]

It actually --verbose doesn't specify the known_host file, so we can forget this idea. Yes I'm using the same user manually and via vscode.

I got to reproduce the error outside of vscode. For some reason, the problem only occurs only when using the "up" or "down" compose command when the global env var DOCKER_HOST is set to ssh://something.
So I'll open a ticket to the compose project.

I think I just found the cause of this. This is not related to docker-compose after all. @chrmarti, Is it possible that, in the process of building the remote containers, vscode sets the HOME env variable to "/host_mnt/c/Users/user" ?

Paramiko relies on $HOME to find the known_hosts file. Paramiko fails to find the file since that folder doesn't exists but still proceed since known_hosts is optional. It then fails later because of paramiko.client.RejectPolicy for MissingHostKeyPolicy.

[chrmarti]

You're right. We shouldn't do that. Good finding!

[yamlCase]

@terroirman oh great find!

Is there a suggested workaround until the fix is in?

[chrmarti]

@yamlCase No, there is no workaround from what I understand.

[terroirman]

@yamlCase : without being a direct workaround, I use an intermediate ssh tunnel to pass the docker/docker-compose traffic. But I can't wait to get rid of that tunnel when this is solved !

See : https://code.visualstudio.com/docs/remote/troubleshooting#_using-an-ssh-tunnel-to-connect-to-a-remote-docker-host