Microk8s: Failed to enable dns

[sagotsky]

I'm on Arch. I was able to install snap and the microk8s snap ok. I can enable registry, but when I try to enable dns I get this.

❯ sudo microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
systemctl: /snap/microk8s/1247/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)
systemctl: symbol lookup error: /usr/lib/systemd/libsystemd-shared-245.so: undefined symbol: p11_kit_uri_get_slot_info, version LIBP11_KIT_1.0
Failed to enable dns

It looks like my systemd libs are mismatched with the libp11-kit libs in the container. I have arch's p11-kit and libp11-kit packages both installed and up to date.

Please run microk8s.inspect and attach the generated tarball to this issue.

[sagotsky]

inspection-report-20200316_092834.tar.gz

[Ranguna]

I'm also getting this error on the same distro.

[jrobsonchase]

I'm blocked on this as well. Any workaround or are we just waiting for a fix in the snap?

[sagotsky]

I've heard kind works, but haven't had a chance to try it yet.

[Ranguna]

@sagotsky kind uses minikube which is not native and it's a single node solution. microk8s on the other hand is native and you can deploy more than one node.

[jrobsonchase]

kind uses minikube

Not sure how you came to that conclusion - it's Kubernetes in Docker - all it needs is a working docker daemon, not a full minikube vm.

FWIW, I am using kind now for local development with reasonable success.

But this is a microk8s issue thread :wink:

[nandanrao]

Also getting this error on arch with up-to-date libp11 packages.

[bench]

same for me, also getting this error on an updated arch machine

[Ranguna]

Not sure how you came to that conclusion - it's Kubernetes in Docker - all it needs is a working docker daemon, not a full minikube vm.

You are right, I'm not sure how I got that wrong.

FWIW, I am using kind now for local development with reasonable success.

Nice, I just started using k3s and it's working great as well :+1:

[vyshkov]

I have same issue on the Archlinux

[bletvaska]

Same issue for me on Fedora 32 (installed yesterday). Same issue for v1.18 and v1.17 as well. On Fedora 31 no such issue.

But when I check status, DNS is enabled.

[rganowski]

Same issue for me on Fedora 32 (installed yesterday). Same issue for v1.18 and v1.17 as well. On Fedora 31 no such issue.

But when I check status, DNS is enabled.

Exact same.

In 1.19/edge apiserver doesn't start, reporting the issue in tarfile::inspection-report/snap.microk8s.daemon-apiserver-kicker/systemctl.log

[bmoller]

Likewise seeing this on Arch Linux.

I think unfortunately there's not a lot the microk8s team can do to fix this due to it being a snap. If you check snapcraft.yaml this snap is built off core which is Ubuntu 16.04-based. There is a snap based on Ubuntu 18.04 (core18), but I'm guessing it would take a fair bit of effort to move this snap to it, and I don't know if it would resolve the problem.

The core snaps provide the majority of the system libraries, and I can only imagine that trying to replace them in this snap is a major effort. Similarly, this is a classic snap with system-level interaction, so there's always going to be linking from your host system into the snap.

I'm going to try building the snap locally with snapcraft base on core18, but I'm not a Snap expert and I have no idea if it will work.

TLDR it's the nature of snaps. I would love to see this distributed outside of snaps, but I also get the reason it's not.

[bmoller]

Update:

I was able to build the snap with a base of core18 but the problem was not resolved. The error message did change slightly, to this:

systemctl: /snap/microk8s/x1/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)
/usr/bin/systemd-tty-ask-password-agent: /snap/microk8s/x1/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)

Based on systemctl and systemd-tty-ask-password-agent appearing in there I installed polkit and wrote custom rules to allow passwordless management of the microk8s services, but still no dice. Curiously, 1.15/stable does work, so I may be way off in regards to the base core being the problem.

[bmoller]

OK, the problem lies here:

https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L70

When microk8s is invoking sudo, it's modifying the linker paths to only look in its container. Previously, with versions <= 1.15 the scripts just invoked sudo directly without any wrapper function or environment modification. You can see where it changed the kubelet restart at the end of this file in this commit:

https://github.com/ubuntu/microk8s/blob/36f79886a074167a0972a19cc14b134bef5a1bd8/microk8s-resources/actions/enable.dns.sh

So, the workaround is to simply perform these steps manually:

1. Copy the microk8s DNS manifest from /var/lib/snapd/snap/microk8s/current/actions/dns.yaml to a temporary location
2. Edit the manifest, replacing all occurrences of $ARCH with your native architecture (most likely amd64
3. Apply the manifest with microk8s kubectl or your own kubectl binary with these args: microk8s kubectl --kubeconfig=/var/snap/microk8s/current/credentials/client.config apply -f dns.yaml
4. Edit the kubelet arguments file located at /var/snap/microk8s/current/args/kubelet, adding these lines:

--cluster-domain=cluster.local
--cluster-dns=10.152.183.10

1. Restart kubelet via systemd: sudo systemctl restart snap.microk8s.daemon-kubelet.service
2. Congratulate yourself for doing microk8s' work for it!

However, you'll still get microk8s status reporting that DNS is not enabled. I found why, but it's 3 AM and I'll open a bug report after some sleep.

[vyshkov]

@bmoller, do you think it's possible for the snap team to fix it?

[ktsakalozos]

Thank you for digging into this.

Would any one be interested in unzipping the snap on the [1] PR, installing it with sudo snap install ./microk8s.snap --classic --dangerous and provide feedback if it fixes the issue at hand?

[1] https://github.com/ubuntu/microk8s/pull/1170/checks?check_run_id=636209186

[ymyzk]

In addition to the error of enable dns, I found a similar error is happening when running microk8s join:

$ sudo microk8s join 172.xx.yy.zz:25000/xxxxxxxxxxxxxxxxxx

WARNING: can't open config file: /usr/lib/ssl/openssl.cnf
systemctl: /snap/microk8s/1379/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)
systemctl: symbol lookup error: /usr/lib/systemd/libsystemd-shared-245.so: undefined symbol: p11_kit_uri_get_slot_info, version LIBP11_KIT_1.0
Traceback (most recent call last):
  File "/snap/microk8s/1379/scripts/cluster/join.py", line 398, in <module>
    update_flannel(info["etcd"], master_ip, master_port, token)
  File "/snap/microk8s/1379/scripts/cluster/join.py", line 143, in update_flannel
    subprocess.check_call("systemctl restart snap.microk8s.daemon-flanneld.service".split())
  File "/snap/microk8s/1379/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['systemctl', 'restart', 'snap.microk8s.daemon-flanneld.service']' returned non-zero exit status 127

Arch Linux + microk8s v1.18.2

[ktsakalozos]

Please give me feedback on the microk8s enable dns path and then I (or anyone else wishes) will change all the references of systemctl to snapctl.

[ymyzk]

Okay, I tried the snap of the PR and the microk8s enable dns path seems to be working:

$ sudo snap install ./microk8s.snap --classic --dangerous
microk8s v1.18.2 installed 

$ sudo microk8s enable dns
Enabling DNS                                                  
Applying manifest                                                                                                           serviceaccount/coredns created
configmap/coredns created                                                                                                   deployment.apps/coredns created    
service/kube-dns created                                                                                                    clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created                                                                Restarting kubelet                                            
DNS is enabled  

[bmoller]

Yes, I see the same, but it _must_ be executed with sudo.

Maybe it's semantics, but can it be worked to handle the elevation in the scripting? The docs all indicate that as long as I'm am member of microk8s I should be able to interact with the platform.

Reference examples in docs:
https://microk8s.io/docs/

[bmoller]

This is growing outside the scope of this issue, but the apiserver-kicker "daemon" is also doing this at [1] and spamming the system journal.

Should I open a new issue? This seems pretty simple to fix these; I can craft a PR if it helps expedite.

[1] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/apiservice-kicker#L38-L42

[ktsakalozos]

@ymyzk, @bmoller, everyone, I think I fixed the systemctl references, could you unzip the artifact of build [1], install with snap install ./microk8s.snap --classic --dangerous and provide me with some feedback if it works for you? Thank you.

[1] https://github.com/ubuntu/microk8s/pull/1170/checks?check_run_id=639349717

[ymyzk]

Thanks for updates, enable dns is not working this time:

$ microk8s enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
error: error running snapctl: cannot restart without a context
Failed to enable dns

sudo microk8s enable dns was also the same and I'm in the microk8s group.

[ktsakalozos]

Thank you @ymyzk. There is a new snap at https://github.com/ubuntu/microk8s/pull/1170/checks?check_run_id=640405337 . This one is better tested.

[ymyzk]

So far, I confirmed that microk8s enable dns, microk8s disable dns, and microk8s join work :tada: I'll report if I find an issue.

[ktsakalozos]

The PR with the fix is merged. The snap on the edge channel should be available within the next few hours. Try it with:

sudo snap install microk8s --classic --channel=edge

The Patch should reach stable with the 1.18.3 release.

Thank you all for helping in fixing this.

[danberindei]

FWIW I'm getting the same error with microk8s enable dns on Fedora 32 with microk8s 18.9:

$
systemctl: /snap/microk8s/1702/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)
systemctl: symbol lookup error: /usr/lib/systemd/libsystemd-shared-245.so: undefined symbol: p11_kit_uri_get_slot_info, version LIBP11_KIT_1.0

I had to use 18.9 because the latest (19.2) wouldn't start the node and I couldn't figure out why. I worked around this issue in 18.9 by restarting microk8s completely (although I know I probably could have restarted just the affected service, I didn't bother finding out the service name).

[thibgc]

I'd like to join the previous commenter in reporting I'm also getting this with v1.18.9 on both Arch Linux and Fedora 32:

systemctl: /snap/microk8s/1702/usr/lib/x86_64-linux-gnu/libp11-kit.so.0: no version information available (required by /usr/lib/systemd/libsystemd-shared-245.so)
systemctl: symbol lookup error: /usr/lib/systemd/libsystemd-shared-245.so: undefined symbol: p11_kit_uri_get_slot_info, version LIBP11_KIT_1.0

but v1.18.9 works fine on CentOS 7.6, CentOS 8, Ubuntu 20.04 or Debian 10.

[thibgc]

@danberindei My guess is this won't get much attention because the issue was previously closed. I will open a brand new issue referencing this one.

[danberindei]

Thanks @thibgc, I didn't want to open another issue because I was able to work around it (I think) by running "sudo microk8s". I'm not 100% sure, because then I hit some networking problems and I gave up, installing Ubuntu on a desktop machine instead.