Kind: Support insecure-registries for container runtime running inside of kind container

[font]

The host that is running kind to set up kind clusters may want to create container images to be pulled by the container runtime (docker/containerd daemons) running inside of the kind-<name>-control-plane containers e.g. kind-1-control-plane. To simplify this, it would be great to have a way to easily configure the container runtime running inside the kind containers with insecure-registries in order to pull images from the host's insecure registry. This would simplify the local registry setup on the host to not require TLS.

For now, I have used the following workaround:

1. Run local insecure registry on host:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

1. Create daemon.json inside of the kind container via docker exec:

docker exec kind-1-control-plane bash -c "cat <<EOF > /etc/docker/daemon.json
{
    "insecure-registries": ["172.17.0.1:5000"]
}
EOF

1. Send SIGHUP to docker daemon in kind container in order to reload config:

docker exec kind-1-control-plane bash -c 'kill -s SIGHUP $(pgrep dockerd)'

This works for now and then any container image to be pulled needs to be specified like so:

docker pull 172.17.0.1:5000/<imagename>:<tag>

[BenTheElder]

SGTM, looks like both cri-o and containerd support this as well so if we want to use those inside the container in the future this can still be supported.

We can add a config option to specify a list of insecure registries and write it through to the daemon config before we start the daemon.

One thought though, specifically for the case of using a registry running on the host where kind is running, probably we can avoid the user needing to know what IP kind will see the host as, otherwise this config will be brittle / non-portable.

https://dev.to/bufferings/access-host-from-a-docker-container-4099 looks like an option for that. perhaps we can have config like:

insecureLocalRegistryPort: 5000

and then images can be at host.docker.internal:5000/foo-image ?

cc @munnerz

This also seems related to #28

[padiazg]

Please, take in account also that there is the possibility of using a private registry with self signed certificates, and to use this you need also put the corresponding CA certificate in place.
Ex: /etc/docker/certs.d/<registry-ip>/ca.crt

Also take in account that we can use a private registry as a proxy, and that must be configured in daemon.json too.
Ex:

{
    "insecure-registries": [
       "<registry-ip>"
     ],
     "registry-mirrors": [
       "https://<registry-mirror-ip>"
     ]
}

We are using two private registries, both use self signed certificates (mostly to avoid using :5000 in the image label), one for our own created images and the other as proxy due that we are in a restricted network.

[BenTheElder]

I think certs can be injected using #62
Edit: technically the config could be too, but note that we may switch to containerd on the nodes.

[BenTheElder]

We're injecting a dockerd systemd dropin for proxy settings now, I think we can look at something similar for insecure registries. Something like kind config containing a list of these registries -> write dropins on the nodes. See also https://github.com/kubernetes-sigs/kind/issues/340

[dims]

fyi, federation folks are using this right now https://github.com/kubernetes-sigs/federation-v2/blob/master/scripts/create-clusters.sh

[h0tbird]

Alternatively you can also do something like this:

➜  ~ cat test.sh
#!/bin/bash

TEMP_DIR=$(mktemp -d /tmp/cluster-api.XXXX)

cat << EOF > ${TEMP_DIR}/kind-config.json
kind: Config
apiVersion: kind.sigs.k8s.io/v1alpha2
nodes:
- role: control-plane
  extraMounts:
    - containerPath: /etc/docker/daemon.json
      hostPath: ${TEMP_DIR}/docker-daemon.json
      readOnly: true
EOF

cat << EOF > ${TEMP_DIR}/docker-daemon.json
{
  "insecure-registries": ["http://172.17.0.1:5000"]
}
EOF

kind create cluster --config ${TEMP_DIR}/kind-config.json

[BenTheElder]

note that overwriting the entire daemon.json is not ideal as we move off the docker-shim: https://github.com/kubernetes-sigs/kind/pull/425#discussion_r271485022

I think we will need a first class option in kind to configure insecure registries. this should be easier to add to v1alpha3 config now.

[fspaniol]

Hi,

I'm trying to add a registry as insecure but it seems that my control-plane does not have the docker binary.. :(

This is what I'm executing:

docker exec -it kind-control-plane bash
root@kind-control-plane:/# docker
bash: docker: command not found
root@kind-control-plane:/#

kind version
v0.4.0

Am I doing something wrong? :)

Cheers,
Fernando

[BenTheElder]

@fspaniol the control plane switched to containerd since this issue was first open :sweat_smile:

[font]

Updated the title to be more generic. :smile:

[fspaniol]

Ah haha, that explains it.

Btw, my use case was that I was trying to follow the tutorial from kubebuilder using kind and I was using a private registry to push my images and when a pod tried to fetch any image, it was getting the x509 issue. The solution I found was to deploy a registry within kind and now every works fine :)

The guide can be found here.

ps: thanks so much for kind, it makes kubernetes usage so much easier <3

[BenTheElder]

@font thanks :-)

@fspaniol Thanks for the feedback, I appreciate it and I'm sure others will find those links very useful.

---

If anyone's interested in this issue, ideally I'd like to find a way to patch .toml files similar to kustomizing kubernetes yaml, that way we can just add the insecure registries we need on top of whatever existing config we have composably. I suspect people are typically writing this config file by hand currently...

[TrentonAdams]

kind started using containerd and none of the solutions here work anymore, how do I go about adding an insecure registry now?

[BenTheElder]

@TrentonAdams the guide mentioned in https://github.com/kubernetes-sigs/kind/issues/110#issuecomment-517433525 is one option for now.

[TrentonAdams]

@BenTheElder Thanks.

[crossdot]

I've got an external insecure registry and deploying it within kind is not an option for me. Is there a way to bring it to work?

[fspaniol]

Maybe load the images manually? kind load docker-image

[BenTheElder]

+1 to side loading the images, it's the most robust and portable option for now.

Tracking https://github.com/containerd/containerd/pull/3574 for a better way to customize containerd config targeted for the next minor release.

[BenTheElder]

xref: https://github.com/containerd/containerd/pull/3702 for being able to use upstream builds, we're up to 1.2.9 from newer ubuntu packaging but will likely need this or our own builds to get 1.3 in a reasonable time frame.

[fedepaol]

In case somebody is interested, I managed to get a (hacky) solution in kubevirt CI, with the registry as a docker container on the same level of kind nodes.

I get that by injecting the container address in the nodes and by setting the registry as insecure in the containerd configuration file.

PR here: https://github.com/kubevirt/kubevirtci/pull/159

[BenTheElder]

The built in merging in v1.3.0 turned out to not be suitable for this use case, but for kind v0.6 I'd like to ship our own config patch merging instead and use that to configure registries as the first use case... https://github.com/kubernetes-sigs/kind/issues/1070

[BenTheElder]

So this will not be the _best_ in v0.6.0, I'm working on a design for better UX, however:
In v0.6.0* we have containerdConfigPatches that can be used to _patch_ the config with the insecure registry setting like:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches: 
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry:5000"]
    endpoint = ["http://registry:5000"]

I'll write up a guide & script around this for a working approach that does not involve clobbering the existing config.

In future releases we can offer a more integrated experience for this.

* will probably release tomorrow after I have time to write good release notes... sometime before kubecon is out ;-)

[BenTheElder]

moving to v0.7.0 because that's possibly the timeframe for making this better, but this is basically in v0.6.0

[BenTheElder]

https://github.com/kubernetes-sigs/kind/pull/1107 adds a tutorial for this

[BenTheElder]

this is pretty much supported, if not the most elegant. will follow up further in https://github.com/kubernetes-sigs/kind/issues/602

[brightzheng100]

Good writeup @BenTheElder

Two errors there:

1. The injecting registry host is wrong:

# for node in $(kind get nodes); do
for node in $(kind get nodes --name ${CLUSTER_NAME}); do
  docker exec "${node}" sh -c "echo $(docker inspect --format '{{.NetworkSettings.IPAddress }}' "${REGISTRY_CONTAINER_NAME}") registry >> /etc/hosts"
done

1. There is a small typo: registry:500/image:foo should be registry:5000/image:foo

BTW, how to contribute to docs?

[aojea]

@brightzheng100 you can submit a PR, the docs files are in the kind repo
https://github.com/kubernetes-sigs/kind/blob/master/site/content/docs/user/local-registry.md

[brightzheng100]

yup, just submitted as #1119

[BenTheElder]

Thanks for catching these and the PR

On Thu, Nov 21, 2019, 00:36 Bright Zheng [e-mail] wrote:

yup, just submitted as #1119
https://github.com/kubernetes-sigs/kind/pull/1119

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/110?email_source=notifications&email_token=AAHADK5G43GFITYY3D6DEF3QUZCBBA5CNFSM4GDZ5OUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEZM2OY#issuecomment-556977467,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK3RET3X6VJJKTZDWITQUZCBBANCNFSM4GDZ5OUA
.

[yuzp1996]

The host that is running kind to set up kind clusters may want to create container images to be pulled by the container runtime (docker/containerd daemons) running inside of the kind-<name>-control-plane containers e.g. kind-1-control-plane. To simplify this, it would be great to have a way to easily configure the container runtime running inside the kind containers with insecure-registries in order to pull images from the host's insecure registry. This would simplify the local registry setup on the host to not require TLS.

For now, I have used the following workaround:

1. Run local insecure registry on host:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

1. Create daemon.json inside of the kind container via docker exec:

docker exec kind-1-control-plane bash -c "cat <<EOF > /etc/docker/daemon.json
{
    "insecure-registries": ["172.17.0.1:5000"]
}
EOF

1. Send SIGHUP to docker daemon in kind container in order to reload config:

docker exec kind-1-control-plane bash -c 'kill -s SIGHUP $(pgrep dockerd)'

This works for now and then any container image to be pulled needs to be specified like so:

docker pull 172.17.0.1:5000/<imagename>:<tag>

Hi I follow your step but find there is no pid like dockerd and so how can I restart docker to reload the daemon.json.
I find all the pid in the kind node container but can not find any pid that I can kill.
Can you give me some suggestions?

Thank you very much

[BenTheElder]

These steps are outdated.
Please see the containerdConfigPatches mechanism used here instead https://kind.sigs.k8s.io/docs/user/private-registries/

[FredericLeroy]

It concerns private registry, not insecure registry, isn't it ?

[BenTheElder]

The same mechanisms / patch type are used to configure all registries.

On Thu, Jun 25, 2020, 01:13 FredericLeroy [e-mail] wrote:

It concerns private registry, not insecure registry, isn't it ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/110#issuecomment-649352204,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK6UHTBEYVDTHAYYGOTRYMBLJANCNFSM4GDZ5OUA
.

[yuzp1996]

These steps are outdated.
Please see the containerdConfigPatches mechanism used here instead https://kind.sigs.k8s.io/docs/user/private-registries/

Thank you very much

I accpect your advice in my helm chart

    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    containerdConfigPatches:
    - |-
    {{- range .Values.insecureRegistry }}
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ . }}"]
        endpoint = ["http://{{ . }}"]
    {{- end }}

and it work well there is no more error when pull image from insecure registry

Thanks again