Buildkit: ssh agent only forwarded from keys on Windows

[lajoie]

Docker Engine: 18.09.2 w/ buildkit enabled
Buildkit Frontend: 1.0.2-experimental
Host OS: Windows 10.0.14393
Container OS: Debian 9.8 Linux
Git-Bash: 2.17.1.windows.2

We're attempting to share our SSH agent socket with a building image in order to support git-ssh.

Doing an ssh-add -l we can see the key is loaded.

Our dockerfile has this directive

RUN ---mount=type=ssh \
    git clone --single-branch --branch "${GIT_BRANCH}" "${GIT_URL}" "${BUILD_HOME}"

The docker build command has this argument

--ssh default

When running the docker build, we get this error:

could not parse ssh: [default]: failed to parse C:/Users/username/AppData/Local/Temp/ssh-l0E48ifEnCIz/agent.8496: ssh: no key found

If we run the git clone locally within git-bash it works. We know that git isn't finding our private key and using it directly from the file system. The key is in a non-standard location and if ssh-agent isn't running or the key is not loaded, the clone fails.

[CCP-Aporia]

Same problem here with a slightly newer Windows and Git Bash version.

Please note that it works when passing the full path to the actual ssh key, e.g.:

$ docker build --ssh default=c:/users/myuser/.ssh/id_rsa .

However, that only works when you know the actual key location on disk, so is not necessarily suitable in all scenarios. Especially TeamCity build agents won't work out of the box as they place any ssh key in a temporary folder.

It seems the problem here is that this check fails to identify the file as a Unix domain socket, and thus treats it as an actual private key file, which it fails to parse.

[tonistiigi]

I tried to look into it. After fixing the https://github.com/moby/buildkit/blob/cb0d51cca32afb196b600ac7e65d0a612c0ea0a9/session/sshforward/sshprovider/agentprovider.go#L135 I'm hit with dial unix C:\Users\Administrator\AppData\Local\Temp\ssh-dD21whKeiVjh\agent.4464: connect: An attempt was made to access a socket in a way forbidden by its access permissions.

ssh-add can connect to it though so I'm not sure what is wrong. This is about the extent of my windows knowledge. If anyone has suggestions, let me know.

As @CCP-Aporia said, workaround is to point to a key.

[lajoie]

@tonistiigi what exactly does providing the full key path do? I assume it bypasses the socket and is the equivalent of doing secret mount of the key. Is that correct?

[tonistiigi]

@lalyos No, it exposes an ssh-agent built into the cli with the specified key loaded. Your private key still never leaves the client. Even on exposing the socket we will not forward the full ssh-agent but the internal one that has limited capabilities and is backed by the local one.

[thaJeztah]

/cc @olljanat @simonferquel

[CCP-Aporia]

@tonistiigi - from an outside perspective I don't see a difference between mapping a SSH key via the --secret or via --ssh. The only reason why I started looking at the --ssh option was because I've the issue that in my build system I don't know the location of the SSH key file (but I do have an SSH agent).

Also, this issue gets even more interesting with Windows 10's built-in OpenSSH agent, which uses a named pipe and does not expose SSH_AUTH_SOCK.

[tonistiigi]

I'd still recommend ssh with a key over a secret in this case. With ssh we never send your key to the daemon. Although we don't persist your secrets either it is safer if they don't ever leave your client at all.

[thaJeztah]

Could we read from a named pipe? Is there some discovery mechanism on Windows to find that pipe?

[olljanat]

To be honest I'm not familiar with this stuff but will try as I was added to cc.

Could we read from a named pipe?

Based on https://github.com/keepassxreboot/keepassxc/pull/1994 I found that there should be named pipe \\.\pipe\openssh-ssh-agent

Is there some discovery mechanism on Windows to find that pipe?

Sure just run PowerShell command:
Get-ChildItem \\.\pipe\ | Format-Table FullName

[lajoie]

sorry for the slow reply, I've been travelling...

From my side, one of the reasons I'd like to have the ssh-agent support is to allow the use of hardware tokens (e.g., yubikey) for our SSH keys. The agent supports that, but the specify-a-key-file models (whether sharing a secret or using the buildkit agent) does not.

[nidkil]

Any progress on this issue?

[MiniXC]

Same issue on linux, sudo DOCKER_BUILDKIT=1 docker build --ssh $SSH_AUTH_SOCK . gives me

could not parse ssh: [/tmp/ssh-kFew4Mw5SPtN/agent.2407]: invalid empty ssh agent socket, make sure SSH_AUTH_SOCK is set

while sudo DOCKER_BUILDKIT=1 docker build --ssh default . results in

could not parse ssh: [default]: invalid empty ssh agent socket, make sure SSH_AUTH_SOCK is set

[lajoie]

This has always worked fine for me on MacOS and Linux. Might want to double-check that you have your key loaded into the SSH agent (ssh-agent -l).

[maximveksler]

Same problem here with a slightly newer Windows and Git Bash version.

Please note that it works when passing the full path to the actual ssh key, e.g.:

$ docker build --ssh default=c:/users/myuser/.ssh/id_rsa .

However, that only works when you know the actual key location on disk, so is not necessarily suitable in all scenarios. Especially TeamCity build agents won't work out of the box as they place any ssh key in a temporary folder.

It seems the problem here is that this check fails to identify the file as a Unix domain socket, and thus treats it as an actual private key file, which it fails to parse.

A bit more portable work around that still works would be:

docker build --ssh default=~/.ssh/id_rsa .

Tested on Git Bash for Windows.

[firecow]

This bug is still present even after latest 2.2.0.0 (42247) update.
docker build --ssh default=~/.ssh/id_rsa . still "works".
Not being able to use the ssh-agent is kind of a pain, and forces big projects to have developers put a non-password protected key in ~/.ssh/id_rsa, or use other hacks.

[dolsem]

I'm having the same issue on Mac OS. Specifying the path to the key works, but ssh-add identities don't.

[mdgilene]

Any update on this? This is a real issue for my team causing us to resort to dirty workarounds.

[r4j4h]

Running into the same issue on Linux. Pathing to non-passworded keys works but ssh-add is ignored :(

[tonistiigi]

This is a windows-only issue. If you run into trouble in other OS it is something else. Debug with ssh-add -L if identities are visible and open new issue or ask in #buildkit if you can't figure it out. If you do receive the same permission denied error on accessing the ssh-agent socket then your user that is invoking docker/buildx/buildctl does not have access to the ssh-agent you are trying to forward.

For windows users: this issue has less to do with buildkit and more with how running ssh-agent socket is accessible under windows permission model. We are currently lacking maintainers who use Windows themselves and/or are familiar with these internals. If you are a windows dev please see if you can help out. Basically you just need to show how a go program can communicate with ssh-agent without getting the permission error.

[r4j4h]

FWIW I was able to proceed in my case adding --ssh default=$SSH_AUTH_SOCK in the docker build command. Maybe that will help the aforementioned MacOS users issue.

[nicks]

I played around with this a bit on my Windows box. The behavior I was seeing was that the ModeSocket bit simply isn't on the ssh agent socket. Might be related to https://github.com/golang/go/issues/33357, which sounds similar

will post a PR that worked for me