kindnetd image is not present in the v1.16.3 node image

Created on 25 Nov 2019 · 13Comments · Source: kubernetes-sigs/kind

What happened:

The kindnetd image is not present on the docker images and the CNI fails

What you expected to happen:

Have the CNI kindnetd image inside the nodes images so it works in airgapped environment

How to reproduce it (as minimally and precisely as possible):

Compile the master version a create a cluster ./kind create cluster

Once it boots check the images inside the container

docker exec -it kind-control-plane crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
cfa147aceaae4       74308f46cf2d3       10 seconds ago      Running             kube-controller-manager   0                   d56de2f644749
92b1fef768cd8       6750aac993260       10 seconds ago      Running             kube-apiserver            0                   2f5bb293fdc41
91f83cf886dff       b2756210eeabf       10 seconds ago      Running             etcd                      0                   3f0fc0e1eab8b
4417271451444       0f7390a0dfb32       11 seconds ago      Running             kube-scheduler            0                   4dc7e7aac03c8

kindnetd is not running and the image is not present

docker exec -it kind-control-plane crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
k8s.gcr.io/coredns                   1.6.2               bf261d1579144       44.2MB
k8s.gcr.io/etcd                      3.3.15-0            b2756210eeabf       248MB
k8s.gcr.io/kube-apiserver            v1.16.3             6750aac993260       185MB
k8s.gcr.io/kube-controller-manager   v1.16.3             74308f46cf2d3       128MB
k8s.gcr.io/kube-proxy                v1.16.3             2388adcec48cf       103MB
k8s.gcr.io/kube-scheduler            v1.16.3             0f7390a0dfb32       105MB

Anything else we need to know?:

After pulling the image crictl doesn't show the tag version 🤔, may be a crictl bug???

docker exec -it kind-control-plane crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
k8s.gcr.io/kube-controller-manager   v1.16.3             74308f46cf2d3       128MB
k8s.gcr.io/kube-proxy                v1.16.3             2388adcec48cf       103MB
docker.io/kindest/kindnetd           <none>              aa67fec7d7ef7       32.4MB
k8s.gcr.io/coredns                   1.6.2               bf261d1579144       44.2MB
k8s.gcr.io/etcd                      3.3.15-0            b2756210eeabf       248MB
k8s.gcr.io/kube-apiserver            v1.16.3             6750aac993260       185MB
k8s.gcr.io/kube-scheduler            v1.16.3             0f7390a0dfb32       105MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca19       746kB

Environment:

kind version: (use kind version): kind v0.7.0-alpha go1.13 linux/amd64
Kubernetes version: (use kubectl version):
Docker version: (use docker info):
OS (e.g. from /etc/os-release):

kinbug prioritimportant-soon

Source

aojea

👀3 👍1

All 13 comments

this shouldn't be the case
/assign

BenTheElder on 25 Nov 2019

After pulling the image crictl doesn't show the tag version thinking, may be a crictl bug???

BenTheElder on 25 Nov 2019

it's probably only pulling by the sha, because we specify it.

however the preloading being broken is likely orthogonal

BenTheElder on 25 Nov 2019

👍1

seems that the problem is in the ctr import command, sideloading the image doesn't work too

./kind load docker-image  kindest/kindnetd:0.5.3@sha256:bc1833b3da442bb639008dd5a62861a0419d3f64b58fce6fb38b749105232555 -v7
Image: "kindest/kindnetd:0.5.3@sha256:bc1833b3da442bb639008dd5a62861a0419d3f64b58fce6fb38b749105232555" with ID "sha256:aa67fec7d7ef71445da9a84e9bc88afca2538e9a0aebcba6ef9509b7cf313d17" not present on node "kind-control-plane"

and after that

root@kind-control-plane:/# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
22b2491c262cb       bf261d1579144       43 minutes ago      Running             coredns                   0                   f86bbfac7d586
1a49c7dc5acea       bf261d1579144       43 minutes ago      Running             coredns                   0                   b6d2a362f8515
7afeb402f5088       2388adcec48cf       43 minutes ago      Running             kube-proxy                0                   b0a2ff5149c4f
cfa147aceaae4       74308f46cf2d3       44 minutes ago      Running             kube-controller-manager   0                   d56de2f644749
92b1fef768cd8       6750aac993260       44 minutes ago      Running             kube-apiserver            0                   2f5bb293fdc41
91f83cf886dff       b2756210eeabf       44 minutes ago      Running             etcd                      0                   3f0fc0e1eab8b
4417271451444       0f7390a0dfb32       44 minutes ago      Running             kube-scheduler            0                   4dc7e7aac03c8
root@kind-control-plane:/# crictl ps | grep kindn

aojea on 25 Nov 2019

~the hash is wrong.~

no it isn't, I was looking at the architecture specific one.

BenTheElder on 26 Nov 2019

👍1

I remove my previous configuration file :
rm /root/.kube -rf

and it works.

TomatoAres on 2 Dec 2019

@TomatoAres I think you're commenting on the wrong issue? :upside_down_face:
This issue is not about kubeconfigs.

BenTheElder on 3 Dec 2019

So the root issue is that docker save with an image ID does not include tags in the output.

We can fix that ourselves with a little work. We have some similar code in the preloading of kubernetes built images already.

BenTheElder on 3 Dec 2019

working on new images with the fix for kindnetd, will cut a small release with these

BenTheElder on 4 Dec 2019

https://github.com/kubernetes-sigs/kind/issues/1140 will cover the kind load docker-image issue, which is due to docker save.

BenTheElder on 4 Dec 2019

pushed:

v1.16.3@sha256:64ecb9181c2d38e86ec637978c1ed6fa6e63a17c7e397e5f5c1521a2a1ca6859
v1.15.6@sha256:943b9fe5ff4dfb2e4ef585f195210f71bdf7caea67b05a1e2987283032cc03aa
v1.14.9@sha256:14d4883aed89a588f408b7aa4ae2fc78b31d459e226a1c08f3ef027cd0608806
v1.13.12@sha256:1341d0af02c724ad61082a523b9ef920a57eb8d94de47d2fdd006f596b3079f4
v1.12.10@sha256:78f71c96a51beb311ac20d4f070e343b76ec888c1d16975801f197a123982c44
v1.11.10@sha256:b2a23c6a9675fc7b116c8274d66dfd970bc44dc20248686383c56340c8aac597

BenTheElder on 4 Dec 2019

need to push again after https://github.com/kubernetes-sigs/kind/pull/1141 :|

BenTheElder on 4 Dec 2019

pushed:

v1.16.3@sha256:70ce6ce09bee5c34ab14aec2b84d6edb260473a60638b1b095470a3a0f95ebec
v1.15.6@sha256:18c4ab6b61c991c249d29df778e651f443ac4bcd4e6bdd37e0c83c0d33eaae78
v1.14.9@sha256:bdd3731588fa3ce8f66c7c22f25351362428964b6bca13048659f68b9e665b72
v1.13.12@sha256:1fe072c080ee129a2a440956a65925ab3bbd1227cf154e2ade145b8e59a584ad
v1.12.10@sha256:c5aeca1433e3230e6c1a96b5e1cd79c90139fd80242189b370a3248a05d77118
v1.11.10@sha256:8ebe805201da0a988ee9bbcc2de2ac0031f9264ac24cf2a598774f1e7b324fe1