Aks-engine: Upgrade from 1.11 to 1.12 results in both kube-dns and coredns

[jackfrancis]

Needs validation, but I built a 1.11 cluster then upgraded to 1.12 and observed that kube-dns componentry was not replaced by coredns, but we simply added coredns to the cluster.

[angrox]

Have the same issue upgrading from AKS 1.11.5 (acsengineversion v0.25.2-aks and v0.26.0-aks) to AKS 1.12.4 (v0.28.1-aks). Selectors are set correctly - only the kube-dns deployment is not deleted.
When deleted manually name resolution over CoreDNS works.

[mboersma]

Verified that this bug happens using current master of aks-engine:

...
INFO[0705] Successfully deleted VM k8s-agentpool1-88093162-vmss000000 in VMSS k8s-agentpool1-88093162-vmss 
INFO[0705] Completed upgrading VMSS k8s-agentpool1-88093162-vmss 
INFO[0705] Completed upgrading all VMSS                 
INFO[0705] Cluster upgraded successfully to Kubernetes version 1.12.4 
$ k get deploy --all-namespaces
NAMESPACE     NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns                1         1         1            1           7m15s
kube-system   heapster               1         1         1            1           25m
kube-system   kube-dns               1         1         1            1           24m
kube-system   kubernetes-dashboard   1         1         1            1           24m
kube-system   metrics-server         1         1         1            1           24m
kube-system   tiller-deploy          1         1         1            1           25m

[jackfrancis]

is the kube-dns spec present in /etc/kubernetes/addons after upgrade?

[mboersma]

It is not:

azureuser@k8s-master-88093162-0:~$ ls /etc/kubernetes/addons/
audit-policy.yaml                     coredns.yaml                         kube-proxy-daemonset.yaml
azure-cloud-provider-deployment.yaml  ip-masq-agent.yaml                   kubernetes-dashboard-deployment.yaml
azure-cni-networkmonitor.yaml         keyvault-flexvolume-installer.yaml   kube-tiller-deployment.yaml
azure-storage-classes.yaml            kube-heapster-deployment.yaml
blobfuse-flexvolume-installer.yaml    kube-metrics-server-deployment.yaml

[jackfrancis]

10-4. So, I think the immediate next step is to publish a simple bash script that will manually clean up the kube-dns artifacts. And validate that container resolution is in good working order w/ coredns.

That will help folks now running upgraded 1.12 clusters w/ both kube-dns and coredns.

[oscarwest]

1.11.5 -> 1.12.5
The kube-dns service loadbalances between both the kube-dns and core-dns pods.
We set the kube-dns-autoscaler to 0 followed by the kube-dns deployment to 0. After verifying DNS was still working with core-dns we cleaned up kube-dns by deleting the kube-dns autoscaler and kube-dns deployment.

[jackfrancis]

Thanks @oscarwest for sharing that known-working guidance. @mboersma Can we craft a working bash script that executes the above remediation?

[Thubo]

Here is what we did to cleanup after upgrading 1.11.5 -> 1.12.5

1) Identify existing ressources

# kubectl --namespace kube-system get all,cm,secret | grep -e 'kube-dns\|kubedns' | awk '{print $1}'
[...]
service/kube-dns
deployment.apps/kube-dns-autoscaler
deployment.apps/kube-dns-v20
[...]
configmap/kube-dns-autoscaler
configmap/kubedns-kubecfg
secret/kube-dns-autoscaler-token-tkwsp
secret/kube-dns-token-lgw45

2) Scale Deployments to 0

# kubectl --namespace kube-system scale deployment kube-dns-autoscaler kube-dns-v20 --replicas=0
deployment.extensions/kube-dns-autoscaler scaled
deployment.extensions/kube-dns-v20 scaled

3) Test if everything is still working as expected

4) Delete everything no longer required

# kubectl  --namespace kube-system delete deployment kube-dns-autoscaler kube-dns-v20
deployment.extensions "kube-dns-autoscaler" deleted
deployment.extensions "kube-dns-v20" deleted

# kubectl  --namespace kube-system delete cm kube-dns kube-dns-autoscaler kubedns-kubecfg
configmap "kube-dns" deleted
configmap "kubedns-kubecfg" deleted
configmap "kube-dns-autoscaler" deleted

# kubectl --namespace kube-system delete serviceaccount kube-dns kube-dns-autoscaler
serviceaccount "kube-dns" deleted
serviceaccount "kube-dns-autoscaler" deleted

# kubectl --namespace kube-system delete clusterrolebinding system:kube-dns-autoscaler system:kube-dns
clusterrolebinding.rbac.authorization.k8s.io "system:kube-dns-autoscaler" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:kube-dns" deleted

# kubectl --namespace kube-system delete clusterrole system:kube-dns-autoscaler system:kube-dns
clusterrole.rbac.authorization.k8s.io "system:kube-dns-autoscaler" deleted
clusterrole.rbac.authorization.k8s.io "system:kube-dns" deleted

Edit: As @Fydon pointed out, we can also delete the clusterroles and the clusterrolebindings (see https://github.com/Azure/aks-engine/issues/398#issuecomment-472326873)

[jackfrancis]

Thanks for providing that clear guidance of a known-working remediation @Thubo!

[Fydon]

@Thubo Should the following also be deleted?

# kubectl --namespace kube-system delete serviceaccount kube-dns kube-dns-autoscaler
serviceaccount "kube-dns" deleted
serviceaccount "kube-dns-autoscaler" deleted

# kubectl --namespace kube-system delete clusterrolebinding system:kube-dns-autoscaler
clusterrolebinding.rbac.authorization.k8s.io "system:kube-dns-autoscaler" deleted

# kubectl --namespace kube-system delete clusterrole system:kube-dns-autoscaler
clusterrole.rbac.authorization.k8s.io "system:kube-dns-autoscaler" deleted

I found the secrets were recreated when just deleting them, but deleting the service accounts also removed their secrets.

[Thubo]

Hi @Fydon - thanks for pointing that out: I simply missed those entities.
We applied your suggestion in our clusters (we also deleted the kube-dns role and binding) and everything is still working fine.
I will update my post above accordingly to have everything in one place.

Thanks a lot :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.