Kind: provide an option to bring up a cluster without a cni installed.

[mauilion]

This option will enable the user to deploy their choice of cni in advance of kind supporting a variety of cni implementations.

[BenTheElder]

/assign
/kind feature

[lwr20]

+1. It'd be awesome to be able to put kind into Calico's CI, and starting with CNI configured but not installed makes it easy to install dev builds or master builds.

I'd imagine the same would be true for pretty much all the CNI plugins.

[neolit123]

we should bring the CNI topic on monday.

my proposal ATM is the following:

• add optional skip of the CNI phase in kind.
• let the user call kubectl apply their preferred plugin.

related topic is enabling offline support for a set of CNI plugins.
my proposal there is:

• we could bundle images and manifests for the top 3 - flannel, weave, callico (as per the kubeadm survey)
  or decide to only bundle weave and force the rest of the users to do the alternative in the previous proposal (skip, kubeclt).

[BenTheElder]

-1 on shipping all 3 unless they are small, our images are big.

I've been working on a rough design for alternate images, and looking at other options for pre-loading / caching (eg a local image mirror).

The additional thing to skipping the overlay apply step in kind is we also need to skip the node readiness check / --wait will be unsupported, but this is likely an acceptable tradeoff for applying arbitrary CNI.

The other reason to consider doing this is EG Cillium, which we simply can't ship by default currently because it relies on kernel modules and we need to support docker for mac / windows + docker-ce on random distros.

[neolit123]

-1 on shipping all 3 unless they are small, our images are big.

i think they are roughly 5-10 MB.

The additional thing to skipping the overlay apply step in kind is we also need to skip the node readiness check / --wait will be unsupported, but this is likely an acceptable tradeoff for applying arbitrary CNI.

that is true.

The other reason to consider doing this is EG Cillium, which we simply can't ship by default currently because it relies on kernel modules and we need to support docker for mac / windows + docker-ce on random distros.

flannel is a step ahead on the Windows front.
but the functionally is lacking documentation and windows images and node selectors are missing in their official manifests.

in fact i just had to patch their 0.11.0 manifest a couple of days ago.

[BenTheElder]

er docker for windows != windows images, still linux images. (we only support linux containers for now.... windows nodes may be interesting someday)

[neolit123]

ah, my confusion. windows containers are out of scope for now.
also quite crazy due to the image sizes.

[Arvinderpal]

Please correct me if I'm wrong here, but the primary purpose of kind today is to support running of CNCF conformance tests on kind created clusters. CNCF tests however appear to be network plugin agnostic, so having in built support for many CNIs does not address this primary goal.

Having said that, as a plugin developer, I would love to use kind for e2e testing of my plugin. Here, IMO, the burden of the actual integration should be placed on the plugin developer (me). Kind just needs to kept flexible enough to support all CNI compatible plugins and not limit support to only the most popular.

My recommendation would be to keep kind as lean as possible (bare minimum required for CNCF tests) but provide the appropriate hooks for someone who wishes to do more.

[BenTheElder]

So let me clarify some things then :upside_down_face:

Please correct me if I'm wrong here, but the primary purpose of kind today is to support running of CNCF conformance tests on kind created clusters

It's _initially_ meant for testing Kubernetes proper, initially we only ran the conformance tests. The scope is expanded at this point to include testing things on top of Kubernetes.

CNCF tests however appear to be network plugin agnostic, so having in built support for many CNIs does not address this primary goal.

To be clear again, we're talking about tests run by Kubernetes the project, not the CNCF.

Having said that, as a plugin developer, I would love to use kind for e2e testing of my plugin. Here, IMO, the burden of the actual integration should be placed on the plugin developer (me). Kind just needs to kept flexible enough to support all CNI compatible plugins and not limit support to only the most popular.

Yes and no. The burden should be on you to test your plugin, but for everyone who is _not_ testing a CNI plugin there should be no burden to get a cluster up with a default configuration.

For that reason when we build an image it includes Kubernetes binaries, images, ... etcd. (and haproxy so we can run HA clusters for testing kubeadm IE part of the Kubernetes project) and soon a CNI. The minimum viable components to boot a standard cluster.

Necessarily we must ship one of the CNIs ourselves for this purpose. Which one we ship is dictated by running on our user platforms without any additional work, so we can't EG ship cillium as the default yet.

We can additionally support users hooking an alternate CNI, but we won't actively ship those. We don't have enough momentum to version and support many of these ourselves, which is why the discussion about "most popular" came up.

My recommendation would be to keep kind as lean as possible (bare minimum required for CNCF tests) but provide the appropriate hooks for someone who wishes to do more.

Bare minimum yes, but again, "CNCF tests" is not really a thing. The conformance tests currently are a subset of the end to end tests that the Kubernetes project develops for testing clusters, and which ones are in the subset is also defined by a work group out of Kubernetes / SIG API Machinery.

[BenTheElder]

The mechanism by which the default is shipped should probably work for building a custom image with a different default CNI ideally, but our "official" usage will need to pick _something_.

So far weave seems to work widely without kernel modules etc. so we're using that.

[Arvinderpal]

Thanks for the clarifications! :) What you said sounds perfect!
I incorrectly used "conformance tests" and "CNCF tests" interchangeably but really meant to say conformance tests everywhere.

[BenTheElder]

Understood, thanks!
A lot of these things are confusing so I just wanted to make sure we were 100% on the same page :sweat_smile:

I'm also planning to remove that note from the README / homepage today, we're using kind for much more than just testing kubernetes with the conformance tests, some rough scoping has been added to https://kind.sigs.k8s.io/docs/contributing/project-scope/ but needs more fleshing out.

[aojea]

Regarding the different CNI plugins it's important to mention that some of them can fail because inside the container we don't have the folder /proc/sys/net/bridge.
Seems that weave made it optional since 1.9.0 https://github.com/weaveworks/weave/issues/2789 and don't have any problem with it.
I was investigating more about this topic and I've found this issue https://github.com/lxc/lxd/issues/5193 that explains the problem, however, seems that the kernel community doesn't like the solution proposed https://lkml.org/lkml/2018/12/13/305

[neolit123]

Regarding the different CNI plugins it's important to mention that some of them can fail because inside the container we don't have the folder /proc/sys/net/bridge.

if the container shares a kernel with the host and if br_netfilter is loaded this will affect the container.
same applies to net.bridge.bridge-nf-call-iptables=1

do you happen to know which CNI plugins will fail for the lack of this folder inside the container?

so in terms of weave, calico and flannel the part of bringing up the cluster works.
which is good, but we still need to run conformance tests on flannel and calico to verify they are good.

kube-system   calico-node-bj4qm                            2/2     Running   0          106s
kube-system   calico-node-npkmf                            2/2     Running   0          111s
kube-system   coredns-ff8f79c87-5m66b                      1/1     Running   0          112s
kube-system   coredns-ff8f79c87-c2fgn                      1/1     Running   0          112s
kube-system   etcd-kind-control-plane                      1/1     Running   0          44s
kube-system   kube-apiserver-kind-control-plane            1/1     Running   0          56s
kube-system   kube-controller-manager-kind-control-plane   1/1     Running   0          49s
kube-system   kube-proxy-crx87                             1/1     Running   0          106s
kube-system   kube-proxy-hpm9z                             1/1     Running   0          111s
kube-system   kube-scheduler-kind-control-plane            1/1     Running   0  

kube-system   coredns-ff8f79c87-b8995                      1/1     Running   0          2m30s
kube-system   coredns-ff8f79c87-hb9p5                      1/1     Running   0          2m30s
kube-system   etcd-kind-control-plane                      1/1     Running   0          103s
kube-system   kube-apiserver-kind-control-plane            1/1     Running   0          97s
kube-system   kube-controller-manager-kind-control-plane   1/1     Running   0          105s
kube-system   kube-flannel-ds-amd64-4n857                  1/1     Running   0          2m26s
kube-system   kube-flannel-ds-amd64-bvxgk                  1/1     Running   0          2m30s
kube-system   kube-proxy-7rjvt                             1/1     Running   0          2m30s
kube-system   kube-proxy-plmr7                             1/1     Running   0          2m26s
kube-system   kube-scheduler-kind-control-plane            1/1     Running   0          100s

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                NAMES
784edd83b81d        kindest/node:latest   "/usr/local/bin/en..."   3 minutes ago       Up 3 minutes                                             kind-worker
2d532ebac16f        kindest/node:latest   "/usr/local/bin/en..."   3 minutes ago       Up 3 minutes        35269/tcp, 0.0.0.0:35269->6443/tcp   kind-control-plane

$ docker exec -ti kind-control-plane /bin/sh -c "ls /proc/sys/net/bridge"
ls: cannot access '/proc/sys/net/bridge': No such file or directory

[aojea]

do you happen to know which CNI plugins will fail for the lack of this folder inside the container?

I just hit the problem yesterday with one "containerization" of the neutron OpenStack service that's using an openvswitch agent and checks for this value. I think that you are right and we'll find out later with the conformance testing, I just thought that it was important to mention to avoid other people spending time troubleshooting the problem :sweat_smile:

if the container shares a kernel with the host and if br_netfilter is loaded this will affect the container.
same applies to net.bridge.bridge-nf-call-iptables=1

great, it seems that can be solved from the CNI plugin side making that check optional and "trusting" the operator but, just curiosity, do you think that it's worth exposing the /proc/sys/net/bridge inside the container?

[neolit123]

great, it seems that can be solved from the CNI plugin side making that check optional and "trusting" the operator but, just curiosity, do you think that it's worth exposing the /proc/sys/net/bridge inside the container?

given the top 3 most popular plugins (survey results) already work, i'd say we consider this only once support for another plugin is requested that faces the problem.

[aojea]

I'm able to launch kubernetes IPv6 single and multinode clusters but it turns out that weave doesn't support IPv6.
I've checked other CNI plugins:

• [ ] [Flannel doesn't support IPv6](https://github.com/coreos/flannel/issues/248)
• [x] Calico supports IPv6

what will be the best option moving forward, default CNI to Calico or implement a configurable CNI option that allows us to use IPv6?

[BenTheElder]

I'm pretty sure calico's manifests at least do not support ARM, and it requires installing an etcd to the cluster. 🤔

CNI can be hacked around by modifying what is in the image in the short term, at boot we read the manifest etc. from the node image.

[BenTheElder]

Btw https://github.com/kubernetes-sigs/kind/issues/205#issuecomment-466597250

[neolit123]

the calico CNI supports ARM64, but not ARM(32).

with the golang ease of mutli-arch, in theory it's easier to add support for another arch, compared to adding ipv6 support.

[BenTheElder]

@neolit123 I see the binaries but not the images / manifest. I hope we wouldn't need to ship those ourselves. Their guide for Kubernetes still requires amd64.

[BenTheElder]

Hmm this is worth referencing from a purely CNI on arm (not k8s) POV: https://community.arm.com/tools/b/blog/posts/understanding-and-deploying-overlay-networks

[neolit123]

they are lacking a schema v2 mutli-arch docker manifest linked from their k8s manifest.

the multi-arch images do exist: e.g. calico/node:v3.3.3-arm64.

[lwr20]

I'm pretty sure calico's manifests at least do not support ARM, and it requires installing an etcd to the cluster.

No need for a separate etcd if you use the Calico manifest that runs using the kubernetes api as a datastore (stores its data in CRDs). At present that also means using host-local IPAM, but that's getting fixed soon...

[neolit123]

@lwr20

https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

seems to point to images such as calico/node:version, but the image doesn't seem to be schemav2 with multi-arch images.

but images such as calico/node:version-arm64 do exist.

are there plans to support multi-arch manifests anytime soon?

[lwr20]

I'm told that calico images went schema v2 from calico v3.4

So https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml should be multi-arch.

And I think it is:

$ docker manifest inspect calico/node:v3.5.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1580,
         "digest": "sha256:f60d1ebb18f15ac6798f9060006c7ac443f3e10b976cc8b67221cbdd4ce6c599",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2205,
         "digest": "sha256:277428813d8bd36bcaf4e2cc95488a3d4e2fbb43cedde04ce623de8214b3b63b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2205,
         "digest": "sha256:a2023a7161ef4cd10e1c6a7f8ce9bd9fb8aefa73a771382da5ead211ee837972",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

[brauner]

If you need this feature, please reply to
https://lkml.org/lkml/2018/11/7/681
and point out that you need it. We kernel people are way more likely to merge stuff like that if users reply and point out that they need it and ask for progress. :)

[aojea]

@BenTheElder I think we can close this issue since https://github.com/kubernetes-sigs/kind/pull/530 was merged

[BenTheElder]

Agreed. This is implemented.
See Cluster.networking.disableDefaultCNI in the config 👍