Containers-roadmap: [EKS] CoreDNS v1.7.0 fails to start after upgrading cluster to Kubernetes 1.18

[gitfool]

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I upgraded an existing EKS cluster from Kubernetes 1.17 to Kubernetes 1.18 and followed the steps for updating an Amazon EKS cluster Kubernetes version. I updated the VPC CNI and KubeProxy images without incident, however the CoreDNS image fails to start with the following in the logs:

plugin/kubernetes: /etc/coredns/Corefile:6 - Error during parsing: unknown property 'upstream'
stream closed 

coredns config map:

apiVersion: v1
kind: ConfigMap
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

Are you currently working around this issue?

Yes, by reverting to CoreDNS v1.6.6.

[ueokande]

upstream option is no longer supported in CoreDNS v1.7.0.

plugin/kubernetes: Remove already-deprecated options resyncperiod and upstream (https://github.com/coredns/coredns/pull/3737)
https://coredns.io/2020/06/15/coredns-1.7.0-release/

I cannot find an upgrade guide about CoreDNS' ConfigMap in AWS's document.

[gitfool]

@ueokande good to know, thanks! I deleted the upstream option and it works.

[iamsudip]

Yes. An upgrade guide will be helpful. I went through this to figure out the correct config: coredns/deploy.sh

[ueokande]

see also https://github.com/awsdocs/amazon-eks-user-guide/issues/212

[mikestef9]

Thanks for reporting. We have updated our user guide with instructions to remove the upstream directive as part of upgrading to CoreDNS 1.7.0

https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

[Ghazgkull]

@mikestef9 Thank you for this, but the documentation only includes manual steps for fixing the configmap (kubectl edit...). How are customers who manage infrastructure via automation intended to handle this?

[ghdiska]

you can run a kubectl patch as one of your iaac step

[Ghazgkull]

Yes. I'm working on that now. Unfortunately, the change required here is a surgical edit inside the content of a multi-line string value. So it's not a straightforward patch operation.

[Ghazgkull]

Here's a working version of a bit of bash script which fetches the configmap, strips out the "upstream" line, and patches the converted string back to the configmap. Note that it requires kubectl, jq, and sed.

UPDATED_COREFILE="$(kubectl get configmap -n kube-system coredns -o json | jq .data.Corefile | sed 's/\\n[ ]*upstream\\n/\\n/g')"
kubectl patch configmap -n kube-system coredns -p "{\"data\": { \"Corefile\": $UPDATED_COREFILE }}"

[ghdiska]

You don't need jq, can filter output with jsonpath . Something like -o=jsonpath="{$.data.Corefile)

[Ghazgkull]

You don't need jq, can filter output with jsonpath . Something like -o=jsonpath="{$.data.Corefile)

You're probably right, but again it's not that simple. :) Your suggestion outputs a multiline string, which won't feed into kubectl patch the way you want it to. If you get a working example without jq, I would be interested though.

[ghdiska]

this example works for me on a test config map ( sorry i can't use it on a k8s 1.18 ) , but i think if you can use jq is more clear

UPDATED_COREFILE="$(kubectl get cm/coredns -n kube-system -o=jsonpath='"{.data.Corefile}"' | sed -z "s/\n/\\\n/g" | sed 's/\\n[ ]*upstream\\n/\\n/g')"