Kops: kube-dns pods cannot be scheduled on master instances

[georgebuckerfield]

$ kops version
Version 1.6.1

Kubernetes version: 1.6.4

Currently, it looks like the two DNS deployments (kube-dns and kube-dns-autoscaler) are unable to be scheduled on a master instance, due to their manifests missing the tolerations in the pod spec. If I reduce the number of nodes to zero (or start a cluster with just one low-powered node) I see the following in /var/log/kube-scheduler on the master:

I0613 22:27:21.514486       6 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-1321724180-6zk7d", UID:"f564bde4-5083-11e7-aada-0aa377c71964", APIVersion:"v1", ResourceVersion:"2281", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' No nodes are available that match all of the following predicates:: PodToleratesNodeTaints (1).

Adding the following tolerations allows the pods to schedule on the master(s):

tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: NoSchedule

Is this intentional? Happy to submit a PR with the changes if not.

[kenden]

Related https://github.com/kubernetes/kops/pull/2705
Maybe more tolerations could be added to kube-dns in that PR

[chrislovecnm]

This is as designed, as the masters typically should not be running applications.

[georgebuckerfield]

@chrislovecnm thanks for clarifying. I guess I was assuming DNS would count as a system resource that should always get scheduled.

Thanks for the work you guys do on kops, it's made adoption of Kubernetes a very smooth experience!

[rlees85]

please be aware this decision prevents scaling nodes down to 0 (which is a shame for dev environments only really used from 9-5). I tried looking around but is there any way of stopping Kops deploying kube-dns so I can just deploy my own modified version? Failing that I'll try using replace. Would be really nice to have a KOPS option to say "allow kube-dns on master" but I fully understand that probably isn't possible.