Kubeadm: e2e tests using common CNI plugins

[caseydavenport]

Following on from this comment: https://github.com/kubernetes/kubeadm/issues/190#issuecomment-290297030

The e2e tests currently run a kubeadm cluster with Weave Net installed as the CNI plugin.

@pipejakob suggested that we should test a cross section of CNI plugins in the tests so we can spot when changes to kubeadm/kubernetes break one but not others.

Some other commonly used network providers:

• [x] Calico
• [x] Flannel
• [ ] Canal

In the original issue I suggested it might be good to test the upstream CNI plugins (e.g bridge). I think given we're planning on doing in-tree testing of 3rd party providers, this becomes less immediately important given they are not commonly used on kubeadm clusters. Once kubenet is no longer compiled into the kubelet (planned for v1.7) we should add testing using the kubenet CNI replacement.

• [ ] kubenet

[cmluciano]

I have been working on a reliable way of performing these e2e bring-ups with several cni plugins on openstack.

I would love to contribute this work upstream in support of this issue.

Open questions from my side

• Are the kubeadm e2e tests sufficient to verify the correctness of any particular cni plugin?
• Should they test only the latest tagged release of the cni plugin?
• Can kubeadm install unreleased k8s binaries based on a commit or nightly release?

[ozdanborne]

I am going to get started adding Calico as an option to kubernetes-anywhere.

I noticed there is a PR open to add Flannel as well: https://github.com/kubernetes/kubernetes-anywhere/pull/390

[pipejakob]

@cmluciano I'll do my best to answer your questions:

Are the kubeadm e2e tests sufficient to verify the correctness of any particular cni plugin?

Probably not, but a great start. The kubeadm e2e tests just initialize a Kubernetes cluster using kubeadm, and then run the Kubernetes Conformance tests against it. These are aimed at testing the correctness of a cluster, but I don't think are guaranteed to trigger every networking edge case (especially the more interesting/hairy ones like failure recovery and network partitions).

Should they test only the latest tagged release of the cni plugin?

If there's value in different variants, go for it. At the minimum, I think we want to test kubeadm with the "latest known good" version of each CNI provider, but if you're asking if it's okay to also set up e2e tests for every commit of the CNI provider as well, that doesn't sound like a bad idea. I'm sure there are plenty of meaningful combinations to test routinely; we'll just need to figure out how to surface the variants on testgrid.

Can kubeadm install unreleased k8s binaries based on a commit or nightly release?

It _can_, but it's not particularly easy to. The kubeadm e2e jobs already use CI builds of kubelet/kubectl/kubeadm that they're testing, but the control plane components (kube-apiserver, kube-controller-manager, etc.) are run as static pods, which means we need Docker images for them all. To the best of my knowledge, we only build these Docker images at release time, not every commit (although that includes alpha/beta/rc releases). It's certainly possible to start a CI/nightly job for building control plane Docker images, and kubeadm also supports the environment variable KUBE_REPO_PREFIX to override the default location for these images in case they're coming from a non-standard location, but I don't think anyone has done this yet.

[luxas]

I think we need an owner for this for v1.7

cc @timothysc @caseydavenport @bboreham @tomdee

[cmluciano]

@ozdanborne Would you like me to take over the calico work?

[luxas]

@cmluciano Feel free to from my side at least.
This is very related to https://github.com/kubernetes/kubeadm/issues/190, you can find some more information about how to do it in practice there

[ozdanborne]

@cmluciano That would be awesome, I've got a lot on my plate atm so happy to hand this over.

I added the Calico manifests as well as the options to apply them but the scripts just wouldn't apply them. I tried rebasing off the flannel PR's and still no dice. That's as far as I got. Here's the WIP:

https://github.com/kubernetes/kubernetes-anywhere/compare/master...ozdanborne:calico

[klizhentas]

Update on the issue:

I think
https://github.com/kubernetes/kubernetes-anywhere/pull/390
and
https://github.com/kubernetes/test-infra/pull/2771
are good to go, I've manually tested both backwards compatibility and with flannel and weave using new flag, clusters start up fine.

This week I'm going to add more Jenkins jobs to the kubernetes-anywhere repository.

@cmluciano if you will be working on Calico, take a look at PRs above.

[cmluciano]

ack, I'll take a look

[cmluciano]

/assign

[luxas]

ping @cmluciano @caseydavenport any movement on this issue?

[cmluciano]

I was waiting on an update to https://github.com/kubernetes/kubernetes-anywhere/pull/390 , but it looks like this work has stalled.

[luxas]

@cmluciano Any chance you could take up the lead on this due to https://github.com/kubernetes/kubernetes-anywhere/pull/390#issuecomment-320702184?

[cmluciano]

I have to finish out some SIG-Network features and then I can address the comments in there.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[timothysc]

Closing this issue it looks like all the necessary elements have been merged.