K3d: [Enhancement] Using "local" images from the host docker

[ashb]

It appears that since this is some form of "DinD" that the docker images built locally from the host are not visible to run in the k3s cluster.

Is there anywork around to this?

[zeerorg]

Currently no way yet. But latest k3s ssupports custom containerd config so we could possibly create a local docker registry pushing to which should be trivial. cc @iwilltry42

[iwilltry42]

A big YES to this feature request!
That's also the most important feature for local development that I'm missing right now.

[Megzo]

As a quick and temporary solution I would suggest to add a -v mount option for k3d, which will be directly passed to the Docker daemon. k3s could then read local images from the /var/lib/rancher/k3s/agent/images directory so you could something like this:

docker save myimage -o $HOME/images/myimage.tar
k3d create -v $HOME/images:/var/lib/rancher/k3s/agent/images

[zeerorg]

There is already a volume mount feature in the latest update. Checkout k3d create --help

[iwilltry42]

Just to give an update on this: while you can certainly preload the images like @Megzo mentioned, I'd like to have the images shared directly without the need for docker save and containerd import.
I'm digging into image saving formats and paths now to get a grasp on how to do this :+1:
Any suggestions are welcome :)

[runningman84]

What about running a registry within k3s? Microk8s offers a similar feature out of the box:
https://itnext.io/working-with-image-registries-and-containerd-in-kubernetes-63c311b86368

[zeerorg]

@runningman84 , I have the exact same idea. if @iwilltry42 has some time he can look into it. I think the best experience would be creating a local docker registry and forwarding port (5000) so that the user can push their images to this registry and the k3s container should be able to pull images with same prefix.

For example:
I push an image: localhost:5000/newfeature:latest from my command line.
The yaml file refers to: localhost:5000/newfeature:latest and k3s container is able to pull the same image.

[goffinf]

+1 being able to configure a local registry is a MUST HAVE

[ashb]

Local registry would be a work-around for me -- ideally would be sharing the same images as the docker host (i.e. by volume mounting in something under /var/lib/docker - but that may not be possible)

[kajanth]

could kaniko be used to solve this issue as we can build an image without the dependency on docker daemon

[silasb]

kaniko is an interesting approach, but I don't see it being much better than docker build ... && docker save ....

Both these solutions require you to store your code in multiple places. With kaniko you'd store it:

• raw source
• kaniko container
• kaniko export artifact (tarball)
• k3s containerd

I like @iwilltry42 solution of trying to somehow allow Docker file format to work with containerd file formwat. Or what I think he's trying to get to.

Ultimately, I think https://github.com/moby/moby/issues/38043 needs to get merged in before we can easily achieve having docker images shared.

[iwilltry42]

So I just started working on this.
Unfortunately it's not that easy and a registry might be the best option for now.
Problems that I faced: we don't have ctr in k3s and the available crictl doesn't have functionality to import images. Then, we cannot easily connect to the containerd.sock inside k3d and use the containerd client, since (you guess it), it's hidden inside the container (or locked by access rights in the local overlayfs of docker).
There's the preloadImages function in k3s that we might ask to expose in a newer version though...

UPDATE 1: I don't think that my original idea to simply share files, won't work, since the image storage formats differ too much and also people might use different storage drivers in docker, which would create the need to support all of them.

I see those two options for now:

1. have a registry running inside k3d or connect it to a running one
2. somehow leverage the preloadImage functionality from k3s from the outside

[kajanth]

so we do this for our gitlab pipeline with k3s by mounting the docker volume

script:
stage: script
image: registry.com/image-dind:v1.0.3
script:
- /usr/local/bin/dind -- dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 &>docker.log &
- k3s server --disable-agent --no-deploy traefik &>master.log &
- sleep 30
- k3s agent --server https://127.0.0.1:6443 --token-file /var/lib/rancher/k3s/server/node-token --docker &>minion.log &
- cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

[iwilltry42]

Thanks for the input @kajanth.
Unfortunately the --docker option doesn't work here, since docker is not included in the k3s image that we use here (containerd only). That's why we have the image storage mismatch between host (docker) and k3d (containerd).
But certainly, creating a new image for k3d with e.g. dind and then mounting the dirs could work as well :+1:

[silasb]

Came across another tool that might be interesting: https://github.com/containers/skopeo which supports OCI export. I haven't tried anything yet, but we might be able to mount the k3s OCI directory on the host and export Docker images via skopeo --insecure-policy copy docker://redis:2.8 oci:k3s-oci-images/redis:2.8

[iwilltry42]

On first glance this looks pretty cool @silasb!
Also, the guys maintaining k3s are thinking about integrating ctr in k3s, so we could leverage its power to import images :+1:

[iwilltry42]

Just to give you an update on this: I created a working version of an import-image command based on docker save and ctr image import that will work as soon as ctr is included in upstream k3s (https://github.com/rancher/k3s/issues/590)

[iwilltry42]

You can check out this release: https://github.com/rancher/k3d/releases/tag/v1.3.0-dev.0
Please read the Release Notes to see how it works.

[silasb]

@iwilltry42 this is awesome. I see that you are importing the images from the host (by saving to a tar) and then moving them to each worker via ctr image import tarball. It'd be nice to also support piping a tarball directly to import-image. I don't love programming for hypotheticals, but I could see case where you might be exporting an image from a remote docker instance (maybe on another computer you are building an image by using https://github.com/genuinetools/img).

[iwilltry42]

@silasb good idea, I can certainly see this as a valid use-case :+1:
I think it'd be a good thing to add using a --tar flag (or we automatically check for the .tar file extension, but that might introduce complexity of mixed statements).
I will work on this in a follow-up PR after #83 got merged.

[iwilltry42]

Since we now have docs for connecting k3d to a locally running insecure registry (https://github.com/rancher/k3d/blob/master/docs/examples.md#connect-with-a-local-insecure-registry) and the new import-images command, I'll go ahead and close this issue, since the basic issue is solved.
We'll still keep an open project to enhance the usage with docker: https://github.com/rancher/k3d/projects/3

[wimo7083]

https://github.com/rancher/k3d/blob/master/docs/registries.md
insecure registry was moved to a subdirectory.

[dahankzter]

Where is this docs @iwilltry42 or has the handling of this been replaced by something else? The website just has an empty toc entry.

[iwilltry42]

@wimo7083 & @dahankzter I just ported the docs over for v3: https://k3d.io/usage/guides/registries/
The built-in managed registry is not yet available and will probably follow in a post v3.0.0 release :+1:
Edit: also, in v3, the import-images command is k3d load images IMAGE -c CLUSTER :+1:

[foobarbecue]

seems it's now k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]