K3d: [FEATURE] Standard way for K3d to communicate about a local registry

[nicks]

KEP

• https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md

---

Original Issue Text

Many Kubernetes clusters support a local registry, to make pushing images directly to the cluster fast. But every cluster documents it with manual shell scripts:

• https://kind.sigs.k8s.io/docs/user/local-registry/
• https://microk8s.io/docs/registry-built-in
• https://github.com/rancher/k3d/blob/master/docs/registries.md#using-a-local-registry
• https://docs.openshift.com/container-platform/4.3/registry/accessing-the-registry.html#registry-accessing-directly_accessing-the-registry

Because every cluster is slightly different, this makes it hard for tools to automate this work for users.

The big idea

Clusters should use annotations on the kube-system namespace to document how to configure a registry

Detailed design

I propose 3 new "standard" annotations:

x-k8s.io/registry: the host of the registry (hostname and port), as seen from outside of the cluster
x-k8s.io/registry-from-cluster: the host of the registry (hostname and port), as seen from inside the cluster
x-k8s.io/registry-help: The URL of documentation where users can read more about how to set a registry up.

Tools that would benefit from local registries should read this annotation and display it to the user

What this means for K3d

All K3d clusters would have a registry-help annotation pointing users to https://github.com/rancher/k3d/blob/master/docs/registries.md#using-a-local-registry

We'd also update the example script at https://github.com/rancher/k3d/blob/master/docs/registries.md#using-a-local-registry to add the x-k8s.io/registry and x-k8s.io/registry-from-cluster annotation during setup.

Prior Art

This proposal is based on what we've been recommending to people in Tilt, with a lot of success: https://docs.tilt.dev/choosing_clusters.html#custom-clusters

update 5/4 - changed the proposal to use the kube-system namespace instead of node annotations

[nicks]

I also filed this feature request here https://github.com/kubernetes-sigs/kind/issues/1543 and here https://github.com/ubuntu/microk8s/issues/1173, and would love other ideas on places I should post about this

[iwilltry42]

Hi @nicks, thanks for opening this issue :+1:
This is really an interesting approach. However, there are still some open questions about whether using node annotations is a good idea here when talking about a cluster-wide attribute.
I see that there are already some comments concerning this in the issue on kind, so I'll just hook in there to see what comes out of that.
We're currently not pulling in that many k8s dependencies, so I hope to get this in as lightweight as possible :grin: E.g. a configmap could be created by using auto-deploy manifests.

[nicks]

ya, that's fair. I'll follow up on the Kind ticket and make sure this is up-to-date with specifics

[nicks]

Just to keep you all in the loop, sig-cluster-lifecycle has suggested I open a KEP for this to kick the tires more on the proposal.

But I also recognize that K3d falls slightly outside the KEP process. Are the K3d maintainers ok with that? Ya, I'll be sure to check in with you all if it starts to veer towards something that would require more k8s deps.

[iwilltry42]

Hi @nicks thanks for the efforts here :+1:
You're right, that the KEPs don't affect us too much, but I guess it would make sense to use a standard also in k3d, if there is one :+1:

[nicks]

Yay! Thanks @iwilltry42

KEP PR: https://github.com/kubernetes/enhancements/pull/1757
KEP tracking issue: https://github.com/kubernetes/enhancements/pull/1755

[nicks]

one question that came up in another thread is if it's worth standardizing on how we document other image-loading options, e.g.,

imageLoadCommand: "k3d import-images $IMAGE_REF"

but maybe that makes more sense in a future KEP

[nicks]

Hi @iwilltry42! I'm doing a last call for any major objections or things I overlooked before we merge the initial spec

https://github.com/kubernetes/enhancements/pull/1757
https://github.com/kubernetes/enhancements/blob/c5b6b632811c21ababa9e3565766b2d70614feec/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md#design-details

If you're OK with this proposal, a thumbs-up on the PR would be a help a lot! Thanks!

[iwilltry42]

Thanks for your efforts @nicks :) You have my :+1:

[nicks]

The KEP has been marked implementable! I suspect it will be straightforward to implement, but let me know if there's any way I can assist!

[iwilltry42]

Very cool @nicks ! Congrats for pushing this through :partying_face:
However, the k3d-managed registry didn't land in v3 yet and will probably make it into v3.1.0, so we'll apply this then :+1:

[iwilltry42]

Hi @nicks :wave:
This is in it's final steps for the v4.0.0 release.
I've read your great KEP, and it's sure, that this specifies exactly one local registry, right?
So k3d supports connecting a cluster to multiple (local) registries. In the current solution draft, it would simply choose the first registry it sees for a cluster as the primary registry, that will be advertised using the new spec.

[nicks]

Ya, that sounds reasonable! The goal of the Config is to advertise to tools what registry they should push to...so choosing the first registry it sees is fine.

[iwilltry42]

441 will land in the next v4.0.0-beta.0 release

[nicks]

Hi @iwilltry42 ! Thank you for landing this. I tried it out and I think there's a minor issue.

$ k3d cluster create --create-registry
$ kubectl local-registry get
host: 0.0.0.0:45659
hostFromContainerRuntime: k3d-k3s-default-registry:5000
help: https://k3d.io/usage/guides/registries/#using-a-local-registry

"0.0.0.0" isn't a valid host here...

would it be better to file a separate bug? I can play around a bit to figure out what it should be...

[iwilltry42]

@nicks are you on a Mac or Windows?
There I guess we need to go via the docker IP again. Or just localhost..

[nicks]

linux!

[iwilltry42]

And it doesn't work for you in Linux with 0.0.0.0? I know it's not a "real" IP, but should work anyway in many cases on Linux.
In any case, we should open a new issue to tackle this 👍

[nicks]

Nope. Ya, let me file a new issue with the details.

[nicks]

filed as https://github.com/rancher/k3d/issues/449