Kind: Better UX for using local registry

[h7kanna]

What would you like to be added:

Add a special node to the Kind cluster with 'registry' role. This node can run docker registry without the user having to run any extra scripts to configure Kind cluster

Why is this needed:

This can be an alternative to 'kind load docker-image' and naturally integrates with 'docker push' CI workflows.

Also, the registry storage can be mounted from the host and kind load can be avoided for easier setup.

[BenTheElder]

We're still designing this, there are some tradeoffs that depend on how the networking changes to fix https://github.com/kubernetes-sigs/kind/issues/148

The obvious "registry" role route has some downsides, including that you'll need to push to each cluster.

[diafour]

I choose not to run a local registry in-cluster. It is not great for testing against multiple versions and/or clusters.
So I run local registry as a standalone docker container with mounted volume in a home directory. Kind cluster is started with a small script, that does two things after the cluster is ready: it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP.

That is an adopted solution back from kubeadm-dind-cluster days: https://github.com/kubernetes-retired/kubeadm-dind-cluster/issues/56#issuecomment-387463386

[BenTheElder]

thanks for the input!

... it adds localhost:5000 as a trusted registry to /etc/containerd/config.toml ...

you can do this at boot time now, xref: https://kind.sigs.k8s.io/docs/user/local-registry/

... and creates a NodePort Service and a DaemonSet to proxying from that NodePort to docker host IP. ...

That's a neat approach, thanks :-)

[diafour]

Yeah, that config.toml override was for 0.5 release. Thanks for the link!

[diafour]

Hooray! I've just realized that endpoint should not be the same as mirror name and simplified my setup: no proxying with DaemonSet required, just containerdConfigPatches:
https://gist.github.com/diafour/13cef191b7cf39543393d310dd6353a0

The workflow is like this:

docker build -t "localhost:5000/name:tag" .
docker push localhost:5000/name:tag

kind: Pod
...
    image: localhost:5000/name:tag

cluster-15.sh create
kubectl replace ... 

There are create and delete commands to address #148 ;)

@BenTheElder Thanks again!

[nicks]

Hi! I work on Tilt! We care so much about good local registry support that if you're not using a local registry, we emit a warning that you _should_ be.

We currently point people at this script to set it up:
https://github.com/windmilleng/kind-local/blob/master/kind-with-registry.sh
but we'd love to recommend a solution built into Kind.

I think the main things that are important to us (from a UX perspective) are (in order):
1) The ability for tooling to auto-detect the registry from the cluster
2) The ability to delete the cluster and create a new one without deleting the registry

So I think our ideal CLI UX would be something like:

kind create registry [--name optional-name]
kind create cluster --with-registry default

where --with-registry default annotated the Kubernetes nodes with the URL of the registry (as the kind-with-registry.sh script above does)

[BenTheElder]

still on my radar! we've mostly landed the host reboot work which includes using a docker network w/ embeded DNS, but had to spend time on a number of other critical fixes.

we still need some follow up to that, but once we settle down the new networking a bit we should have a better idea what we can build the registry integration against.

[BenTheElder]

I think instead of a "create registry" w/ ordering issues (and presumably failing if it already exists etc.) the ux from k3d is closer to what people need, ie. --enable-registry when creating the cluster. (which enables a shared local registry). https://github.com/rancher/k3d/blob/master/docs/registries.md#using-the-k3d-registry

[BenTheElder]

(and I've just filed what I hope is the last major fix related to the DNS stuff, we neglected to update the NO_PROXY logic to handle it properly)