K3s: CoreDNS initial forwarders specification

[coldpenguin]

This may or may not be related to #1527

Version:
k3s version v1.18.2+k3s1 (698e444a)

K3s arguments:
(ps -ef |grep k3s, looks to be what I remember)
/usr/local/bin/k3s server --no-deploy servicelb --no-deploy traefik

Describe the bug
I have edited the configmap for coredns:
kubectl -n kube-system edit cm coredns
To change the forwarders from being the /local to pod/ forwarders to be my local nameserver,

-        forward . /etc/resolv.conf
+        forward . 10.1.0.2

This is because I have local machines that I wish k3s to interact with (git repo etc. ) which are not in the google DNS servers.
It would appear that on a reboot of the master k3s server (there are 3 nodes currently, no HA in this cluster) that the configmap is reset to the contents of /var/lib/rancher/k3s/server/manifests/coredns.yaml

why this is a pain
On occasion it is not possible for k3s to get to the google DNS servers, and at this point I am left with error messages which appear to be the same as #1527, in that coredns just fails completely.
(it should be noted though that this might be a pointer to the fault in #1527 rather than #1527 necessarily being the fix to this).

Even when google is working, when I have pods which try and pull from a fully qualified registry running within k3s (registry.my-domain-name rather than registry.registry.svc as I have external build tools too) because my router does not support hair-pinning this fails (as google returns my external IP address, which the router can't route internally from an internal address, don't know why this seems to be so difficult for so many routers).

To Reproduce
1) Edit configmap for coredns
2) Restart k3s (when I have noticed it, it is usually a reboot then a systemctl restart k3s)

Expected behavior
My expectation is that the configmap is CREATED upon deployment of k3s from the manifest file.
Then when edited using the kubectl command (not through the POD as this is read-only) it should remain as it is edited, even through a reboot/restart of the cluster

Actual behavior
It would appear that the configmap is being reset back to 'defaults' written when the k3s system was deployed through the file /var/lib/rancher/k3s/server/manifests/coredns.yaml
Either the system should check whether the configmap exists and not patch it, OR, it should be clearly documented somewhere if the intended behaviour is to edit this manifest file upon deployment (which I couldn't find when trying to debug why it kept resetting).
(from memory Openshift does not allow apply to be done directly to configmaps, so I guess k3 here is doing a diff then patch)

Additional context / logs
Several restarts of coredns pod (i.e. delete pod) with 'wrong' forwarder (/etc/resolv.conf)

[INFO] plugin/reload: Running configuration MD5 = 4665410bf21c8b272fcfd562c482cb82
   ______                ____  _   _______
  / ____/___  ________  / __ \/ | / / ___/      ~ CoreDNS-1.6.3
 / /   / __ \/ ___/ _ \/ / / /  |/ /\__ \       ~ linux/arm64, go1.12.9, 37b9550
/ /___/ /_/ / /  /  __/ /_/ / /|  /___/ / 
\____/\____/_/   \___/_____/_/ |_//____/  
E0603 09:58:38.572021       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused
E0603 09:58:38.572021       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: connect: connection refused

after restarting coredns pod (i.e. delete pod) with forwarder set correctly, the coredns starts correctly immediately with no errors reported

--- edit
I don't think this has any relevance to the main bug (resetting configmap), but the cluster is aarch64 Raspbian on 3 rpi 4bs)

[brandond]

Yes. There are a couple open issues about it. The one that best represents the current state is probably #1797.

Your best bet at the moment is to copy coredns.yaml to something unique (coredns-local.yaml, for example), then add --disable=coredns to the command-line options, and restart k3s. One of the downsides to disabling coredns is that the nodeHosts configmap entry won't get updated with host file entries for new nodes, so if you're adding or removing nodes from the cluster you'll need to configure it manually.

[huiser]

Workaround we use:

iptables -t nat -A PREROUTING -p udp -d 8.8.8.8  --dport 53 -j DNAT --to 11.22.33.44

11.22.33.44 as an example for our internal DNS resolver. We are running K3S on RHEL 7 VM's.

[coldpenguin]

Thank you for the workaround @huiser
@brandond if you think this is close enough to #1797 then this can be closed.

Where I am coming from here, is that I personally would expect upgrades/updates (/just restarting the service) to patch rather than overwrite configmaps.
I think that while both of these workarounds would work, neither of them really cover the failure of expectation.

[AaronJan]

Update

I just reinstalled my cluster using --resolv-conf (tried to change the k3s.service script, didn't work), now everything works!

systemd-resolved (I'm using Debian) puts a stub DNS server in the /etc/resolv.conf file, like this:

nameserver 127.0.0.53

But there's a "real" resolv.conf file in /run/systemd/resolve/resolv.conf (check it first), so I just set my local DNS server in /etc/netplan/50-cloud-init.yaml - BTW, do not use multiple nameservers, because CoreDNS will randomly pick one for every query by default, If you want to change this behavior, then you have to change the Corefile (we try to avoid that remember?) - at last, use --resolv-conf /run/systemd/resolve/resolv.conf to install K3s.

I think we could put some common pitfalls for Debian/Ubuntu in FAQ. 🤣

---

Same issue.

I have to disable Traefik (--disable traefik) and create a customized one a couple days ago, because I want to use my own self-signed SSL certificate for Ingress, and any change made to the /var/lib/rancher/k3s/server/manifests/traefik.yaml file will gets reset when restarting k3s service.

Yesterday, I want the built-in CoreDNS forward all non-cluster DNS queries to a separate local CoreDNS server (I have some local services too), I tried the --resolv-conf flag, point resolv.conf to another file like this:

nameserver 192.168.1.60

Then I start a Busybox pod to test it, not working (don't know why, nslookup works fine on all nodes).

But like what @coldpenguin does, using kubectl -n kube-system edit configmap coredns, change the forward . /etc/resolv.conf to forward . 192.168.1.60 works, but nonpersistent.

The iptables method seems a little bit hacky, so I might have to disable CoreDNS too...

Play with K3s has been very painful for me... 😭

[CuBiC3D]

I had some debugging going on because of the strange behaviour of coredns using 8.8.8.8 as default forwarder. I'm running locally and need the local DNS to resolve. Actually setting the --resolv-conf to /etc/resolv.conf on k3os did the trick. If I inspect the container's volumes with crictl, I see that the resolv.conf has now the same content has the host's one.