Kubeadm: configure coredns anti affinity

[jowenn]

What keywords did you search in kubeadm issues before filing this one?

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST

Versions

kubeadm version:
kubeadm version: &version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:14:41Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

rpm -qa:
kubelet-1.11.0-0.x86_64
kubectl-1.11.0-0.x86_64
kubernetes-cni-0.6.0-0.x86_64
kubeadm-1.11.0-0.x86_64

docker images:
k8s.gcr.io/kube-controller-manager-amd64 v1.11.0 55b70b420785 2 weeks ago 155 MB
k8s.gcr.io/kube-scheduler-amd64 v1.11.0 0e4a34a3b0e6 2 weeks ago 56.8 MB
k8s.gcr.io/kube-apiserver-amd64 v1.11.0 214c48e87f58 2 weeks ago 187 MB
k8s.gcr.io/kube-proxy-amd64 v1.11.0 1d3d7afd77d1 2 weeks ago 97.8 MB
k8s.gcr.io/coredns 1.1.3 b3b94275d97c 7 weeks ago 45.6 MB
k8s.gcr.io/etcd-amd64 3.2.18 b8df3b177be2 3 months ago 219 MB

What happened?

In a three node cluster after restarting the first node (master0) both coredns pods got scheduled on the same node, although the third one would have had enough resources.

NAME READY STATUS RESTARTS AGE IP NODE
coredns-78fcdf6894-frwxk 1/1 Running 1 12h 10.15.224.4 master1
coredns-78fcdf6894-ls7nq 1/1 Running 1 12h 10.15.224.3 master1

What you expected to happen?

kubeadm should generate a coredns deployment with pod anti affinity rules. So the pod distribution looks something like

coredns-78fcdf6894-frwxk 1/1 Running 1 12h 10.15.224.4 master1
coredns-78fcdf6894-ls7nq 1/1 Running 1 12h 10.15.224.3 master2

[neolit123]

kubeadm should generate a coredns deployment with pod anti affinity rules

if someone thinks that there are concerns in doing that, please comment.
any takers for sending a PR?

reference:
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

/cc @kubernetes/sig-cluster-lifecycle-feature-requests

[neolit123]

@aojea is working on this issue.
/active

[xlgao-zju]

Hi, @neolit123 I find out this issue too. I created a three-master cluster, and the two coredns pods are on the same node. Anti affinity is a good option to spread the pods, but I have a concern about doing this. I read this form https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

Note: Inter-pod affinity and anti-affinity require substantial amount of processing which can slow down scheduling in large clusters significantly. We do not recommend using them in clusters larger than several hundred nodes.

I am wondering if it is worth slowing down scheduling?

[neolit123]

that might a problem. thanks for pointing that out @xlgao-zju.
i don't see a PR from @aojea, yet.

so i guess we need feedback if this is something that we don't want to enable by default from kubeadm.
/assign @luxas
/assign @timothysc

[fturib]

@chrisohaver : what is your opinion, I think we already added and removed that option.

[chrisohaver]

Yes - this came up a while back (~3-6 months ago?). Ultimately the decision was to not include anti-affinity for the scaling issue mentioned above. At the time there was a parallel PR that attempted to fix the anti-affinity scaling issue, but after merging it was found that it did not resolve the problem effectively enough. For the brief time that we thought the issue was resolved, we had pod anti-affinity in the coredns/deployment manifest.

[chrisohaver]

related:
coredns/deployment#60
kubernetes/kubernetes#54164
kubernetes/kubernetes#57683
kubernetes/kubernetes#59357

[neolit123]

thanks for the valuable info @chrisohaver and @fturib. :100:

@aojea please hold on sending a PR for this for now.

for small clusters this is nice to have and there has to be a switch for it, when kubeadm is setting up coredns, but this should be OFF by default so that we don't get reports about scaling issues.

deferring to @luxas and @timothysc on the possibility of switches/knobs for this.
i think we should pretty much expose the default manifest + Corefile and let the users grade on whatever rules they want to use.

[xlgao-zju]

i think we should pretty much expose the default manifest + Corefile and let the users grade on whatever rules they want to use.

@neolit123 Agree. If we can expose the manifest yaml, it would be very convenient for users to modify it.

[timothysc]

@MrHohn - originally I requested anti-affinity, then it was removed by the coreDNS folks. I'm not certain why this was done.

/cc @kubernetes/sig-network-bugs

[chrisohaver]

removed by the coreDNS folks. I'm not certain why this was done.

performance reasons. kube-dns did the same.

[MrHohn]

Yep as @chrisohaver said it is mainly for performance reason. Ref the PR that reverted it: https://github.com/kubernetes/kubernetes/pull/60642.

[neolit123]

closing in favor of the tracking issue for customizing kubeadm addons:
https://github.com/kubernetes/kubeadm/issues/1091

/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

closing in favor of the tracking issue for customizing kubeadm addons:
https://github.com/kubernetes/kubeadm/issues/1091

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.