Kops: Upgrade from KubeDNS to CoreDNS with nodeLocalDNS is failing cluster spec validation

[pshanoop]

1. What kops version are you running? The command kops version, will display
this information.
Version 1.16.0 (git-4b0e62b82)
I tried with Version 1.17.0 (git-a17511e6dd) too.

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T11:36:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
kops edit cluster
5. What happened after the commands executed?
Updating cluster spec with

spec:
  kubeDNS:
    provider: CoreDNS
    nodeLocalDNS:
      enabled: true

Reopens the yaml with this errors on it

# Found fields that are not recognized
# ...
#         rbac.authorization.k8s.io/v1alpha1: "true"
#     kubeDNS:
# +     nodeLocalDNS:
# +       enabled: true
#       provider: CoreDNS
#     kubeProxy:
# ...
#.
````
**6. What did you expect to happen?**

CoreDNS installed with node local cache enabled.

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2019-12-12T16:08:07Z"
  generation: 12
  name: k-staging.companydomain
spec:
  api:
    loadBalancer:
      crossZoneLoadBalancing: true
      sslCertificate: arn:aws:acm:**************************
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://kops-state-s3/k-staging.companydomain
  dnsZone: ZONEID
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-eu-central-1a
      name: a
    - instanceGroup: master-eu-central-1b
      name: b
    - instanceGroup: master-eu-central-1c
      name: c
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-eu-central-1a
      name: a
    - instanceGroup: master-eu-central-1b
      name: b
    - instanceGroup: master-eu-central-1c
      name: c
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    authorizationMode: RBAC
    oidcClientID: kubernetes
    oidcGroupsClaim: groups
    oidcIssuerURL: https://auth.k-staging.companydomain/dex
    oidcUsernameClaim: email
    runtimeConfig:
      rbac.authorization.k8s.io/v1alpha1: "true"
  kubeProxy:
    metricsBindAddress: 0.0.0.0
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.16.9
  masterInternalName: api.internal.k-staging.companydomain
  masterPublicName: api.k-staging.companydomain
  networkCIDR: 10.2.0.0/16
  networkID: vpc-0000000000000
  networking:
    canal: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 10.2.4.0/22
    id: subnet-0000000000000000
    name: eu-central-1a
    type: Private
    zone: eu-central-1a
  - cidr: 10.2.8.0/22
    id: subnet-0000000000000000
    name: eu-central-1b
    type: Private
    zone: eu-central-1b
  - cidr: 10.2.12.0/22
    id: subnet-0000000000000000
    name: eu-central-1c
    type: Private
    zone: eu-central-1c
  - cidr: 10.2.16.0/22
    id: subnet-0000000000000000
    name: utility-eu-central-1a
    type: Utility
    zone: eu-central-1a
  - cidr: 10.2.20.0/22
    id: subnet-0000000000000000
    name: utility-eu-central-1b
    type: Utility
    zone: eu-central-1b
  - cidr: 10.2.24.0/22
    id: subnet-0000000000000000
    name: utility-eu-central-1c
    type: Utility
    zone: eu-central-1c
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2019-12-12T16:08:07Z"
  generation: 1
  labels:
    kops.k8s.io/cluster: k-staging.companydomain
  name: master-eu-central-1a
spec:
  additionalSecurityGroups:
  - sg-0ca124fde7d94f621
  image: kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-central-1a
  role: Master
  subnets:
  - eu-central-1a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2019-12-12T16:08:07Z"
  generation: 1
  labels:
    kops.k8s.io/cluster: k-staging.companydomain
  name: master-eu-central-1b
spec:
  additionalSecurityGroups:
  - sg-0ca124fde7d94f621
  image: kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-central-1b
  role: Master
  subnets:
  - eu-central-1b

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2019-12-12T16:08:07Z"
  generation: 1
  labels:
    kops.k8s.io/cluster: k-staging.companydomain
  name: master-eu-central-1c
spec:
  additionalSecurityGroups:
  - sg-0ca124fde7d94f621
  image: kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-central-1c
  role: Master
  subnets:
  - eu-central-1c

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2020-05-14T16:22:59Z"
  generation: 4
  labels:
    kops.k8s.io/cluster: k-staging.companydomain
  name: nodes-infra
spec:
  additionalSecurityGroups:
  - sg-0ca124fde7d94f621
  cloudLabels:
    Owner: DevOps
    instanceGroup: nodes-infra
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/k-staging.companydomain: ""
  image: kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: m4.xlarge
  maxSize: 2
  minSize: 0
  nodeLabels:
    beta.kubernetes.io/fluentd-ds-ready: "true"
    kops.k8s.io/instancegroup: nodes-infra
    companydomain/node-importance: high
    companydomain/node-type: infra
  role: Node
  subnets:
  - eu-central-1a
  - eu-central-1b
  - eu-central-1c
  taints:
  - dedicated=infra:NoSchedule

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2019-12-12T16:08:07Z"
  generation: 2
  labels:
    kops.k8s.io/cluster: k-staging.companydomain
  name: nodes-spot
spec:
  additionalSecurityGroups:
  - sg-0ca124fde7d94f621
  cloudLabels:
    Owner: DevOps
    instanceGroup: nodes-spot
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/k-staging.companydomain: ""
    spot: "true"
  image: kope.io/k8s-1.16-debian-stretch-amd64-hvm-ebs-2020-01-17
  machineType: m4.xlarge
  maxPrice: "0.15"
  maxSize: 15
  minSize: 1
  nodeLabels:
    beta.kubernetes.io/fluentd-ds-ready: "true"
    kops.k8s.io/instancegroup: nodes-spot
    companydomain/node-importance: low
    companydomain/node-launchpad: "true"
    companydomain/node-type: spot
  role: Node
  subnets:
  - eu-central-1a
  - eu-central-1b
  - eu-central-1c

**8. Please run the commands with most verbose logging by adding the -v 10 flag.

I0612 12:51:09.129547 2072636 factory.go:68] state store s3://kops-state
I0612 12:51:09.129608 2072636 s3context.go:325] unable to read /sys/devices/virtual/dmi/id/product_uuid, assuming not running on EC2: open /sys/devices/virtual/dmi/id/product_uuid: permission denied
I0612 12:51:09.129618 2072636 s3context.go:170] defaulting region to "us-east-1"
I0612 12:51:10.228231 2072636 s3context.go:210] found bucket in region "eu-central-1"
I0612 12:51:10.228809 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/config"
I0612 12:51:10.544216 2072636 s3fs.go:257] Listing objects in S3 bucket "kops-state" with prefix "k-staging.companydomain/instancegroup/"
I0612 12:51:10.588127 2072636 s3fs.go:285] Listed files in s3://kops-state/k-staging.companydomain/instancegroup: [s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1a s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1b s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1c s3://kops-state/k-staging.companydomain/instancegroup/nodes-infra s3://kops-state/k-staging.companydomain/instancegroup/nodes-spot]
I0612 12:51:10.588151 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1a"
I0612 12:51:10.642840 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1b"
I0612 12:51:10.695040 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/instancegroup/master-eu-central-1c"
I0612 12:51:10.733904 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/instancegroup/nodes-infra"
I0612 12:51:10.783954 2072636 s3fs.go:220] Reading file "s3://kops-state/k-staging.companydomain/instancegroup/nodes-spot"
I0612 12:51:10.840641 2072636 editor.go:128] Opening file with editor [vim /tmp/kops-edit-82tvjyaml]
I0612 12:51:32.260616 2072636 editor.go:128] Opening file with editor [vim /tmp/kops-edit-zjpezyaml]
A copy of your changes has been stored to "/tmp/kops-edit-zjpezyaml"

Edit cancelled, no valid changes were saved.

9. Anything else do we need to know?

I tried after updating cluster to CoreDNS then enable nodeLocalDNS. But this also failed with same error.

[pshanoop]

Doing further search, It seems nodeLocalDNS is missing in KubeDNSConfig type
https://pkg.go.dev/k8s.io/kops/pkg/apis/kops?tab=doc#KubeDNSConfig

[olemarkus]

It is another instance of #9151. Support for this is comes in kops 1.18, which is currently in beta.

[pshanoop]

Until then is there anyway to enable nodeLocalDNS ?

[olemarkus]

You can just deploy the DS as per https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/ as long as you are not using kubeproxy ipvs mode or cilium

[erickfaustino]

I've stumble on this while doing some research about the same problem. In my case, my cluster is running with IPVS, so we need to set the following variable to customize nodelocaldns as suggested in https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

export KOPS_FEATURE_FLAGS="+ExperimentalClusterDNS"

Before it I was seeing kops cluster spec validation errors. This flags has the purpose of bypass it.

Maybe someone find it useful.