Kubeadm: Add an option to use CoreDNS instead of KubeDNS in v1.9

[luxas]

This would be alpha (feature-gated) in v1.9
As discussed in the SIG meeting and on Slack

[luxas]

cc @johnbelamaric
I can't assign you this officially, but please work on this as agreed anyway ;)

[johnbelamaric]

Thanks @luxas. In fact @rajansandeep has it ready: https://github.com/kubernetes/kubernetes/pull/52501

[luxas]

Thanks @rajansandeep for https://github.com/kubernetes/kubernetes/pull/52501, great work so far!
The next steps are to:

• add support for CoreDNS in kubeadm alpha phase addons
• add support to upgrade to using CoreDNS. That is, I have a v1.8 cluster and want to upgrade to v1.9 with CoreDNS instead. kubeadm upgrade plan should support --feature-gates and support showing CoreDNS and it's version instead of kube-dns
• add e2e tests for this

[rajansandeep]

@luxas the e2e tests, you mean the ones which are associated with kube-up?

[luxas]

@rajansandeep No (but that would probably not hurt), I was thinking about adding coverage for using CoreDNS here: https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all#kubeadm-gce

[fturib]

What we expect to do is:

Ensure that all usual e2e tests are evaluated against a k8s cluster that is configured with CoreDNS as the default DNS Server.

I took a day to investigate how the e2e testing was organized.
I think I found the steps for add the e2e for coreDNS - in the 3 steps below (will need 2 PRs)

1- Ability to deploy coredns in k8s for e2e:
In kubernetes/kubernetes-anywhere/phase3

• introduce a new addon for ‘coredns’, similar to addon kube-dns
• Modify the all.jsonnet to import either kube-dns, either core-dns (depending a new parameter phase3.dns to declare - see similar parameter phase3.cni). Ensure to continue importing kube-dns if parameter does not exist.
• update docs (Kconfig)

_see sample with calico addon https://github.com/kubernetes/kubernetes-anywhere/pull/405)_

2- Add a new job in PROWL for testing e2e with a k8s deployment that uses coredns as DNS service
In kubernetes/test-infra/blob/master/jobs/config.json, create a new job “ci-kubernetes-e2e-kubeadm-gce-dns-coredns” by copying the def of job ci-kubernetes-e2e-kubeadm-gce, and add a parameter in the job "--kubernetes-anywhere-dns=coredns" that will be interpreted in phase3 all.jsonnet above.

3- Update the Testgrid to include that job into the “sig-cluster-lifecycle-all”
Update configuration in this file : kubernetes/test-infra/blob/master/testgrid/config/config.yaml

_See similar addition for calico in https://github.com/kubernetes/test-infra/pull/4981)_

@luxas : can you validate it is what you expect for the e2e of CoreDNS within k8s ?

[luxas]

Yes, that is what I expect @fturib. Thanks for investigating and contributing to that!

[fturib]

@luxas: @rajansandeep and myself are pushing the PR for:

PR https://github.com/kubernetes/kubernetes/pull/55952

• add support for CoreDNS in kubeadm alpha phase addons
• add support to upgrade to using CoreDNS

PR test-infra : https://github.com/kubernetes/test-infra/pull/5601
PR kubernetes-anywhere : https://github.com/kubernetes/kubernetes-anywhere/pull/486

• e2e testing for coredns

IMPORTANT:

The kubeadm upgrade was expected for BETA only, as the migration of the kube-dns configuration into a CoreDNS configuration an heavy work.
@rajansandeep did an upgrade that will start CoreDNS with the default configuration.

Question:

does it worth to embed right now that upgrade w/o migration ? How should we inform the user that the configuration is not upgraded ?

[luxas]

does it worth to embed right now that upgrade w/o migration

I don't see why that would be that hard to do. As commented in the PR:

• Run the func installing CoreDNS as ususal
• Check that CoreDNS starts correctly, and remove the kube-dns Deployment

Voilá :)

[johnbelamaric]

Francois is talking about the translation of the kube-dns ConfigMap into a Corefile that will implement the same features. I believe you can control the "stub zones" and federation configurations via that ConfigMap.

It's not hard but we are short on time.

[luxas]

Francois is talking about the translation of the kube-dns ConfigMap into a Corefile that will implement the same features. I believe you can control the "stub zones" and federation configurations via that ConfigMap.

Ah, ok. kubeadm clusters aren't using that feature, so no worries. You don't have to care about that in this case :smile:. Doing that migration generally sounds like a perfect thing for v1.10 and beta, but that's not needed for me right now.
Makes sense?

[johnbelamaric]

Yep, thanks.