Docker-stacks: Docker BuildKit SSH fails to download repository

[earthastronaut]

What docker image you are using?

Tried with
jupyter/scipy-notebook
jupyter/base-notebook

And I imagine it effects all the images

What complete docker command do you run to launch the container (omitting sensitive values)?

DOCKER_BUILDKIT=1 docker --log-level debug build \
    --progress=plain --ssh default -t docker_build_secret_test:latest .

With Dockerfile:

# syntax = docker/dockerfile:1.0-experimental
# https://docs.docker.com/develop/develop-images/build_enhancements/
FROM jupyter/base-notebook:latest

USER root

# Install ssh
RUN apt-get update && apt-get install -y openssh-client git
RUN mkdir -p -m 0600 ${HOME}/.ssh && ssh-keyscan github.com >> ${HOME}/.ssh/known_hosts \
    && fix-permissions ${HOME}

USER ${NB_USER}

# Download repositories
RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stacks.git

What steps do you take once the container is running to reproduce the issue?

Container fails to build.

What do you expect to happen?

I'm trying to use the Docker BuildKit ssh feature to download repositories via SSH. It works with a python base image and I expected it to work with the Jupyter base images but it does not.

Python base image which it works with this Dockerfile:

# syntax = docker/dockerfile:1.0-experimental
# https://docs.docker.com/develop/develop-images/build_enhancements/

FROM python:3.9-buster

# Download public key for github.com
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# Download repositories
RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stacks.git

What actually happens?

It errors with the following error message. I debugged that I needed the fix-permissions step to use the ${HOME}/.ssh directory. I think it's something to do with the --mount=type=ssh and the permission to create/access/write to certain directories within the image but I'm struggling to troubleshoot this.

...
#11 [4/4] RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stack...
#11 0.393 Cloning into 'docker-stacks'...
#11 0.674 Warning: Permanently added the RSA host key for IP address '140.82.113.4' to the list of known hosts.
#11 0.845 [email protected]: Permission denied (publickey).
#11 0.846 fatal: Could not read from remote repository.
#11 0.846 
#11 0.846 Please make sure you have the correct access rights
#11 0.846 and the repository exists.
#11 ERROR: executor failed running [/bin/bash -o pipefail -c git clone [email protected]:jupyter/docker-stacks.git]: runc did not terminate sucessfully
------
 > [4/4] RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stacks.git:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/bash -o pipefail -c git clone [email protected]:jupyter/docker-stacks.git]: runc did not terminate sucessfully

[Bidek56]

Why would you post this question here instead of SO?
This seem to be a Docker issue instead of an image issue. Are you using WSL?
I see related issues on SO

[earthastronaut]

Well I don't think it's a docker issue because it works on Dockerfiles with different docker base images. I do think it's something to do specifically with the jupyter base image and how the environment is built. it is not a bug with the jupyter image per say, this is a feature request that the jupyter base image should work with buildkit ssh. I posted here because I believe it's specific to the jupyter base images and I wanted to have the contributors, like yourself, involved.

That said, your point is fair to prefer it on SO instead of here. There are some questions like you found though none also related to jupyter base image that would fix this issue. If you would prefer to address it in SO then definitely close it here and I'll post it there. Thanks!

[mathbunnyru]

@earthastronaut I think I know the problem in your example.

Actually, it's right in the logs you sent:
#11 0.202 /bin/bash: git: command not found

Basically, git is not installed in jupyter images.
So, if you want your example to work, just install git like this: apt-get install --yes git (under root user).

I guess git is present in python:3.9-buster, so it works just fine.

[earthastronaut]

Shoot, I was sloppy there @mathbunnyru. My apologies, that was the wrong error message. I did fix the missing git requirement by adding an apt-get install. I update the error message in the ticket to show the one where it's failing to have access rights to github.

For historical reference this is the error message with the missing git error I posted originally:

...
#11 [4/4] RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stack...
#11 0.202 /bin/bash: git: command not found
#11 ERROR: executor failed running [/bin/bash -o pipefail -c git clone [email protected]:jupyter/docker-stacks.git]: runc did not terminate sucessfully
------
 > [4/4] RUN --mount=type=ssh git clone [email protected]:jupyter/docker-stacks.git:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/bash -o pipefail -c git clone [email protected]:jupyter/docker-stacks.git]: runc did not terminate sucessfully

[romainx]

Hello,

In fact git is not installed in base-notebook but installed in minimal-notebook and all downstream images.

https://github.com/jupyter/docker-stacks/blob/95ccda3619d03bfd6a2233b715ecc19d56966e05/minimal-notebook/Dockerfile#L11-L15

So the solution if you do not want to install it may be to switch to minimal-notebook or one of the downstream images.

$ docker run -it --rm jupyter/base-notebook git --version
# [FATAL tini (6)] exec git failed: No such file or directory

$ docker run -it --rm jupyter/scipy-notebook git --version
# git version 2.25.1

$ docker run -it --rm jupyter/minimal-notebook git --version
# git version 2.25.1

I'm closing the issue, do not hesitate if it's not clear.
Best.

[mathbunnyru]

@romainx after my comment Dylan fixed the description and the issue is not in the git not installed anymore.
It's something to do with permissions in jupyter images and definitely worth checking.

I can also confirm that I have the same issue when running the example above.
So, please reopen the issue, it's totally legit.

[romainx]

@mathbunnyru, @earthastronaut sorry my bad, my answer was related to git installation that is actually installed in most of the images.
I reopen it. If you can have a look It would be great 😄

[mathbunnyru]

Ok, after spending quite a lot of time, I think I know what the issue is.

This is the thing - when you're running python:3.9-buster, you're using root.
But in jupyter dockers the user is non-root.
And here's the SO question for using ssh forwarding for non-root user:
https://stackoverflow.com/questions/60959367/ssh-forwarding-during-build-only-works-for-root

And finally, I was able to fix the bug just by adding ,uid=1000 in the last line.

So the final versions looks like this:

# syntax = docker/dockerfile:1.0-experimental
# https://docs.docker.com/develop/develop-images/build_enhancements/
FROM jupyter/base-notebook:latest

USER root

# Install ssh
RUN apt-get update && apt-get install -y openssh-client git
RUN mkdir -p -m 0600 ${HOME}/.ssh && ssh-keyscan github.com >> ${HOME}/.ssh/known_hosts \
    && fix-permissions ${HOME}

USER ${NB_USER}

# Download repositories
RUN --mount=type=ssh,uid=1000 git clone [email protected]:jupyter/docker-stacks.git

Everything works fine.

The value 1000 is used by default by docker-stacks, that's why I chose it here. I do not know how no to hardcode this, but I'm not familiar with syntax RUN --mount=type=ssh, so worth searching if you don't like hardcoding.

[earthastronaut]

Thank you @mathbunnyru ! That's brilliant. I checked too and there does not appear to be a way to use the NB_UID variable instead of hardcoding it. But hardcoding is not the worse. I'm not trying to change that value on the image.

[romainx]

@mathbunnyru thank you for this work, impressive!
@earthastronaut if this solution is OK for you could you please close the issue?