Kubeadm: kinder: switch to a newer CNI

[neolit123]

we pinned the Calico (CNI) version to avoid downloading the latest version.
our old version of calico is using deprecated APIs (i think it was CRD v1beta1 and it will be removed in 1.22), so we should switch to the latest.

the latest Calico install includes an operator and have two manifests:

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

https://docs.projectcalico.org/getting-started/kubernetes/quickstart

the last time i checked this it did not work for me, maybe because i was running a single CP node.
we are currently pinning calico:
https://github.com/kubernetes/kubeadm/blob/05f9b96cf83d73ed2177815487eba90517b66708/kinder/pkg/data/calicomanifests.go

we can switch to Cillium:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml

this would allow us to better pin the CNI version and worked fine for me.

[fabriziopandini]

Why not switching to kindnet? we are using it in CAPD now https://github.com/kubernetes-sigs/cluster-api/blob/master/test/e2e/data/cni/kindnet/kindnet.yaml

[neolit123]

I have not tried it. Is it production ready?

One downside is that it is not something that users commonly use in
production. So i cannot tell them 'Try kindnet. We know it works in our CI'.

[neolit123]

got Calico working, but only with ClusterConfiguration->networking->podSubnet: 192.168.0.0/16
Calico at one point supported custom CIDR allocation and this was not needed...

i'm realizing that CNI plugins have become huge and slow to start due to their operators and other bits:

quay.io/tigera/operator              v1.15.1              73b255331773        2 weeks ago         47.5MB
calico/node                          v3.18.1              50b52cdadbcf        2 weeks ago         172MB
calico/typha                         v3.18.1              cdfbea6501a9        2 weeks ago         52.8MB
calico/pod2daemon-flexvol            v3.18.1              3994c62982cc        2 weeks ago         21.7MB
calico/cni                           v3.18.1              21fdaa2fccee        2 weeks ago         131MB
calico/kube-controllers              v3.18.1              a274b7fd57f5        2 weeks ago         53.4MB

quay.io/cilium/operator-generic      v1.9.4               0cd130536240        7 weeks ago         51.6MB
quay.io/cilium/cilium                v1.9.4               aaf366dbd941        7 weeks ago         434MB

so it it feels like prepull in the nodeimage must be done here...

then again, we don't really need these operators for the CNI even, since we don't manage CNI upgrade.

compared to:

aojea/kindnetd                       latest               9be511e8f0bb        13 months ago       130MB

this is just what we need, so switching to kindnet like you recommended @fabriziopandini SGTM.
cc @aojea

not sure what to update this note here in the docs to...

Currently Calico is the only CNI plugin that the kubeadm project performs e2e tests against. If you find an issue related to a CNI plugin you should log a ticket in its respective issue tracker instead of the kubeadm or kubernetes issue trackers.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

based on the last survey Calico is by far the most popular CNI, but since we will not be testing against it any more, we have to be honest and adjust that note that we are testing with a custom CNI.

[aojea]

this is just what we need, so switching to kindnet like you recommended @fabriziopandini SGTM.
cc @aojea

the good thing of kubernetes is that CNI must just work, so it is relatively safe to use the one that you wants
The image that you show is from my own repo, you should use the one from kind
https://github.com/kubernetes-sigs/kind/blob/bcfa8dedde8fc0b9608fefb14cce40ae3c057c88/pkg/build/nodeimage/const_cni.go#L23

[SataQiu]

/cc

[fabriziopandini]

A possible middle ground is to use kindnet for the majority of our tests (where the focus is on other things), and eventually add dedicated tests validating kubeadm working with CNI providers.
However, this makes really sense if we get help for the CNI maintainers in keeping those tests up to date...

[aojea]

However, this makes really sense if we get help for the CNI maintainers in keeping those tests up to date...

most CNI plugins use KIND in their CIs, that is indirectly validating kubeadm and the CNI ... I can't provide you an official list but ovn-kubernetes, anthrea, calico, ... use KIND

[fabriziopandini]

Thanks @aojea !
It makes really sense to me if CNI projects are maintaining their own validation against kind/kubeadm

[BenTheElder]

AIUI cillium does some testing on kind as well.

I think the major kindnetd limitations currently are:

• still requires basic CNI binaries (lo, portmap, ptp, ...) to be preinstalled on the node (most other CNI manifests copy this in from the image, we don't currently since we control the node as well)
• requires patching to pass CIDRs (more reliable than trying to detect from the cluster ...) except the node pod CIDR. This is typically true of other CNI though
• no network policy implementation
• not tested on "real" clusters, no workarounds to handle large numbers of nodes, limited attention paid to node add/remove

Most of these are fixable without much work if necessary but may be fine. The passing data one is comparable to calico I think.

It's not hard to port that here or if you don't need the subnets to be configurable then you can just set a fixed value like the current kinder PR.

[BenTheElder]

FWIW kindnetd is aiming to be the simplest workable CNI option. It's very similar to what kube-up and pre-cillium GKE configure on the CNI json side, but the daemonset bundles in IP masquerade and does the node to node pod routes with netlink instead of programming a cloud network or similar.

It would probably be relatively fine on a real small cluster with equal cost node to node links (like presumably most home labs) but so far we haven't tried to actively support that as we're not trying to "compete" with the production offerings.

We could do better offering a way to clean up after it without deleting the host. IIRC we did at least start tagging the routes (but haven't registered the tag with the kernel ...) and the iptables rules are named appropriately.

It could also do with a bit more care with logging etc. but it's been relatively robust in CI and we've been taking care to make sure it starts quickly and remains lightweight for small clusters (for large clusters it probably would need a better solution than periodically listing nodes in each pod in the daemonset).