Microk8s: Installation timeout v1.19/Raspberry PI 4/Ubuntu 20.20 snapshot

[ita1024]

Only timeouts are observed:

# snap install microk8s --classic
error: cannot perform the following tasks:
- Run configure hook of "microk8s" snap if present (run hook "configure": 
-----
+ '[' '!' -e /var/snap/microk8s/1669/args/containerd ']'
+ grep -e '\-\-docker unix://${SNAP_DATA}/docker.sock' /var/snap/microk8s/1669/args/kubelet
++ produce_certs
++ export OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
++ OPENSSL_CONF=/snap/microk8s/current/etc/ssl/openssl.cnf
++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
++ '[' -f /var/snap/microk8s/1669/certs/serviceaccount.key ']'
++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
++ '[' -f /var/snap/microk8s/1669/certs/ca.key ']'
++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
++ '[' -f /var/snap/microk8s/1669/certs/server.key ']'
++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
++ '[' -f /var/snap/microk8s/1669/certs/front-proxy-ca.key ']'
++ for key in serviceaccount.key ca.key server.key front-proxy-ca.key front-proxy-client.key
++ '[' -f /var/snap/microk8s/1669/certs/front-proxy-client.key ']'
++ '[' -f /var/snap/microk8s/1669/certs/ca.crt ']'
++ '[' -f /var/snap/microk8s/1669/certs/front-proxy-ca.crt ']'
++ render_csr_conf
+++ get_ips
++++ /snap/microk8s/1669/bin/hostname -I
+++ local 'IP_ADDR=192.168.0.73 '
+++ [[ -z 192.168.0.73  ]]
+++ /snap/microk8s/1669/sbin/ifconfig cni0
+++ echo '192.168.0.73 '
++ local 'IP_ADDRESSES=192.168.0.73 '
++ cp /var/snap/microk8s/1669/certs/csr.conf.template /var/snap/microk8s/1669/certs/csr.conf.rendered
++ '[' '192.168.0.73 ' == 127.0.0.1 ']'
++ '[' '192.168.0.73 ' == none ']'
++ local ips= sep=
++ local -i i=3
+++ echo '192.168.0.73 '
++ for IP_ADDR in $(echo "$IP_ADDRESSES")
++ ips+='IP.3 = 192.168.0.73'
++ sep='\n'
++ /snap/microk8s/1669/bin/sed -i 's/#MOREIPS/IP.3 = 192.168.0.73/g' /var/snap/microk8s/1669/certs/csr.conf.rendered
++ '[' -f /var/snap/microk8s/1669/certs/csr.conf ']'
++ local force
++ /snap/microk8s/1669/usr/bin/cmp -s /var/snap/microk8s/1669/certs/csr.conf.rendered /var/snap/microk8s/1669/certs/csr.conf
++ force=false
++ false
++ '[' '!' -f /var/snap/microk8s/1669/certs/front-proxy-client.crt ']'
+++ /snap/microk8s/1669/usr/bin/openssl x509 -noout -issuer
++ '[' 'issuer= /CN=front-proxy-ca' == 'issuer=CN = 127.0.0.1' ']'
++ echo 0
+ '[' 0 == 1 ']'
+ '[' -e /var/snap/microk8s/1669/args/containerd-template.toml ']'
+ grep -e 'stream_server_address = ""' /var/snap/microk8s/1669/args/containerd-template.toml
+ grep -e '\-\-allow-privileged' /var/snap/microk8s/1669/args/kubelet
+ '[' -f /root/snap/microk8s/common/istio-auth.lock ']'
+ '[' -f /root/snap/microk8s/common/istio-auth.lock ']'
+ need_api_restart=false
+ '[' -f /var/snap/microk8s/1669/credentials/kubelet.config ']'
+ '[' -f /var/snap/microk8s/1669/credentials/proxy.config ']'
+ '[' -f /var/snap/microk8s/1669/credentials/scheduler.config ']'
+ '[' -f /var/snap/microk8s/1669/credentials/controller.config ']'
+ for dir in "${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_DATA}/var/lock"
+ chmod -R ug+rwX /var/snap/microk8s/1669/credentials/ /var/snap/microk8s/1669/certs/ /var/snap/microk8s/1669/args/ /var/snap/microk8s/1669/var/lock
+ chmod -R o-rwX /var/snap/microk8s/1669/credentials/ /var/snap/microk8s/1669/certs/ /var/snap/microk8s/1669/args/ /var/snap/microk8s/1669/var/lock
+ getent group microk8s
+ getent group microk8s
+ chgrp microk8s -R /var/snap/microk8s/1669/credentials/ /var/snap/microk8s/1669/certs/ /var/snap/microk8s/1669/args/ /var/snap/microk8s/1669/var/lock/ /var/snap/microk8s/1669/var/kubernetes/backend/
+ '[' -e /var/snap/microk8s/1669/opt/cni/bin/flanneld ']'
+ '[' -f /var/snap/microk8s/1669/args/flanneld ']'
+ grep -e etcd.socket:2379 /var/snap/microk8s/1669/args/etcd
+ grep -e basic-auth-file /var/snap/microk8s/1669/args/kube-apiserver
+ grep '\-\-enable\-v2' /var/snap/microk8s/1669/args/etcd
--enable-v2=true
+ false
+ '[' -L /var/snap/microk8s/1669/bin/cilium ']'
+ '[' -e /var/snap/microk8s/1669/var/lock/clustered.lock ']'
+ '[' -e /var/snap/microk8s/1669/args/flannel-template.conflist ']'
+ grep -e cniVersion /var/snap/microk8s/1669/args/flannel-template.conflist
    "cniVersion": "0.3.1",
+ '[' '!' -f /var/snap/microk8s/1669/args/cluster-agent ']'
+ grep -e '\-\-timeout' /var/snap/microk8s/1669/args/cluster-agent
--timeout 240
+ mkdir -p /var/snap/microk8s/1669/juju/share/juju /var/snap/microk8s/1669/juju-home
+ chmod -R ug+rwX /var/snap/microk8s/1669/juju /var/snap/microk8s/1669/juju-home
+ chmod -R o-rwX /var/snap/microk8s/1669/juju /var/snap/microk8s/1669/juju-home
+ getent group microk8s
+ chgrp microk8s -R /var/snap/microk8s/1669/juju /var/snap/microk8s/1669/juju-home
+ grep -e '\-\-ip-masq' /var/snap/microk8s/1669/args/flanneld
--ip-masq=true
+ grep -e '\-\-cluster-cidr=10.152.183.0/24' /var/snap/microk8s/1669/args/kube-proxy
+ '[' -e /var/snap/microk8s/1669/args/cni-network/cni.yaml ']'
+ '[' -e /var/snap/microk8s/1669/var/lock/ha-cluster ']'
+ echo 'Setting up the CNI'
Setting up the CNI
++ date +%s
+ start_timer=1603402337
+ timeout=120
+ KUBECTL='/snap/microk8s/1669/kubectl --kubeconfig=/var/snap/microk8s/1669/credentials/client.config'
+ sleep 5
++ date +%s
+ now=1603402461
+ [[ 1603402461 > 1603402457 ]]
+ break
+ /snap/microk8s/1669/kubectl --kubeconfig=/var/snap/microk8s/1669/credentials/client.config apply -f /var/snap/microk8s/1669/args/cni-network/cni.yaml
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

The system was installed from a snapshot (20 October) and updated afterwards.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.10
Release:        20.10
Codename:       groovy
$ uname -a
Linux k8s3 5.8.0-1006-raspi #9-Ubuntu SMP PREEMPT Fri Oct 16 12:55:30 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

microk8s inspect cannot run because the installation fails and ufw is by default inactive. While version 1.18 can be successfully installed, I am interested in features found in version 1.19.

[nnewc]

Also see this behavior on Ubuntu Server 20.10

[roc26002w]

I have same issue too

but reinstall choice custom version 1.18 can success installed.

[pyro-tec]

Same issue here... found a workarround though:
Interestingly, installing 1.18 and then refreshing to 1.19 seems to work!

[ita1024]

@roc26002w I am not interested in version 1.18 because that version does not provide high availability. This does not solve the problem.
@pyro-tec Installing 1.18 and refreshing to 1.19 does not work for me. This does not solve the problem.

[sorennielsen]

Same issue here.
First updated Ubuntu from 20.04 to 20.10 and then tried to completely remove old version first with sudo snap remove microk8s --purge and then installing version 1.19 which then fails with "The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?"

[sorennielsen]

I can confirm that installing Microk8s in old version and then refreshing to newest version does makes it seem to work (even though I do not know how to verify that it ends up in state equal to a fresh install of 1.19.)

Commands used:
sudo snap install microk8s --classic --channel=1.18/stable
=> microk8s (1.18/stable) v1.18.9 from Canonical✓ installed

then:
sudo snap refresh microk8s --channel 1.19/stable
=> microk8s (1.19/stable) v1.19.0 from Canonical✓ refreshed

then:
sudo microk8s.status
=> microk8s is running

[christian-draeger]

This just took me a hole day and made me play around with minikube instead 😅 Glad I found the issue now.
Can confirm it's failing in version 19. Installing version 18 is working.
(raspberry pi 4b 8gb - Ubuntu Desktop 20.10 fresh install)

[roc26002w]

Now I can install 1.19/stable from snap install microk8s --classic --channel=1.19/stable and it work

[ita1024]

@roc26002w Are you on a Raspberry Pi 4 and using Ubuntu Server 20.10? The installation is still failing in the exact same manner for me. Please post the installation image that you used for installing Ubuntu 20.10 Server on a Raspberry Pi 4 to obtain a successful microk8 1.19 installation.

[roc26002w]

@ita1024 oh, my mistake
I'm not used Raspberry Pi 4 and Ubuntu Server 20.10
I'm using my dev VM and Install official Ubuntu Server 20.10

[ktsakalozos]

The 1.19 release waits up until 2 mins for the API server to start. This may become an issue on slow machines. We are considering to lift this requirement. An an AMD64 architecture, could you please try installing MicroK8s with:

sudo snap install microk8s --classic --channel=latest/edge/slow-start

We appreciate your feedback.

[ita1024]

I do not have any sAMD64 architecture for testing slow starts, and on Raspberry PI 4 the following outputs are observed:

$ uname -a
Linux k8s1 5.8.0-1006-raspi #9-Ubuntu SMP PREEMPT Fri Oct 16 12:55:30 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

# snap install microk8s --classic --channel=latest/edge/slow-start
error: requested a non-existing branch on latest/edge for snap "microk8s": slow-start

[jcbettinelli]

Same problem here:
$ uname -a Linux pi-node06 5.8.0-1006-raspi #9-Ubuntu SMP PREEMPT Fri Oct 16 12:55:30 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
$ sudo snap install microk8s --classic --channel=latest/edge/slow-start error: requested a non-existing branch on latest/edge for snap "microk8s": slow-start

+ /snap/microk8s/1669/kubectl --kubeconfig=/var/snap/microk8s/1669/credentials/client.config apply -f /var/snap/microk8s/1669/args/cni-network/cni.yaml The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

With 1.18:
$ sudo snap install microk8s --classic --channel=1.18/stable microk8s (1.18/stable) v1.18.9 from Canonical✓ installed

[nderraugh]

I can also reproduce this on a clean latest. Doing a refresh from 1.18/stable up to 1.19/stable works as a workaround for me too. I'm also on raspi 4, but Linux ubuntu0 5.8.0-1006-raspi #9-Ubuntu SMP PREEMPT Fri Oct 16 12:55:30 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux.

[ita1024]

The installation finally works:

# snap install microk8s --classic --channel=latest/edge
microk8s (edge) v1.19.3 from Canonical✓ installed

Afterwards, it takes around five minutes to reach the running state:

# microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.
...
# microk8s status
microk8s is running

After a reboot, microk8s seems to reach that state in less than a minute.

[krk628]

1.19 installs on Ubuntu fail here:

/snap/microk8s/1752/kubectl --kubeconfig=/var/snap/microk8s/1752/credentials/client.config apply -f /var/snap/microk8s/1752/args/cni-network/cni.yaml

It's failing because microk8s cannot deploy calico from the default manifest which tries to autodetect the BGP endpoint for the node you are installing. I think on my RPi I could get it to work by specifying the interface. Does anyone know how to customize the cni.yaml manifest? Basically I'd want to put:

value: "interface=eth0"

instead of

value: "first-found"

Installing 1.18 then upgrading sucks as you get 1.19 with etcd rather than dqlite...so no HA.

[rockaut]

Any update to this? Still having troubles with 1.19 and infact a stable install which is 1.20 at this day already

[ktsakalozos]

@rockaut @krk628 the CNI manifest is in /var/snap/microk8s/current/args/cni-network/cni.yaml.

Apologies for the very late response.