Microk8s: v1.16.1 (rev 936) broken

[octopus-prime]

mike@workstation:~$ sudo snap install microk8s --channel=edge --classic
[sudo] Passwort für mike: 
microk8s (edge) v1.16.1 aus Canonical✓ installiert
mike@workstation:~$ microk8s.enable dns ingress registry
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling Ingress
error when retrieving current configuration of:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "ingress", Namespace: ""
Object: &{map["apiVersion":"v1" "kind":"Namespace" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"ingress"]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/api/v1/namespaces/ingress: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount"
Name: "nginx-ingress-microk8s-serviceaccount", Namespace: "ingress"
Object: &{map["apiVersion":"v1" "kind":"ServiceAccount" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-ingress-microk8s-serviceaccount" "namespace":"ingress"]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/api/v1/namespaces/ingress/serviceaccounts/nginx-ingress-microk8s-serviceaccount: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole"
Name: "nginx-ingress-microk8s-clusterrole", Namespace: ""
Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"ClusterRole" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-ingress-microk8s-clusterrole"] "rules":[map["apiGroups":[""] "resources":["nodes" "services"] "verbs":["get" "list" "watch"]] map["apiGroups":[""] "resources":["configmaps" "endpoints" "nodes" "pods" "secrets"] "verbs":["list" "watch"]] map["apiGroups":[""] "resources":["events"] "verbs":["create" "patch"]] map["apiGroups":["extensions" "networking.k8s.io"] "resources":["ingresses"] "verbs":["get" "list" "watch"]] map["apiGroups":["extensions" "networking.k8s.io"] "resources":["ingresses/status"] "verbs":["update"]]]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/clusterroles/nginx-ingress-microk8s-clusterrole: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "rbac.authorization.k8s.io/v1, Resource=roles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=Role"
Name: "nginx-ingress-microk8s-role", Namespace: "ingress"
Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"Role" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-ingress-microk8s-role" "namespace":"ingress"] "rules":[map["apiGroups":[""] "resources":["configmaps" "endpoints" "pods" "secrets"] "verbs":["get"]] map["apiGroups":[""] "resourceNames":["ingress-controller-leader-nginx"] "resources":["configmaps"] "verbs":["create" "update"]]]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/namespaces/ingress/roles/nginx-ingress-microk8s-role: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding"
Name: "nginx-ingress-microk8s", Namespace: ""
Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"ClusterRoleBinding" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-ingress-microk8s"] "roleRef":map["apiGroup":"rbac.authorization.k8s.io" "kind":"ClusterRole" "name":"nginx-ingress-microk8s-clusterrole"] "subjects":[map["kind":"ServiceAccount" "name":"nginx-ingress-microk8s-serviceaccount" "namespace":"ingress"]]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/nginx-ingress-microk8s: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "rbac.authorization.k8s.io/v1, Resource=rolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=RoleBinding"
Name: "nginx-ingress-microk8s", Namespace: "ingress"
Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"RoleBinding" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-ingress-microk8s" "namespace":"ingress"] "roleRef":map["apiGroup":"rbac.authorization.k8s.io" "kind":"Role" "name":"nginx-ingress-microk8s-role"] "subjects":[map["kind":"ServiceAccount" "name":"nginx-ingress-microk8s-serviceaccount"]]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/apis/rbac.authorization.k8s.io/v1/namespaces/ingress/rolebindings/nginx-ingress-microk8s: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "/v1, Resource=configmaps", GroupVersionKind: "/v1, Kind=ConfigMap"
Name: "nginx-load-balancer-microk8s-conf", Namespace: "ingress"
Object: &{map["apiVersion":"v1" "kind":"ConfigMap" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"nginx-load-balancer-microk8s-conf" "namespace":"ingress"]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/api/v1/namespaces/ingress/configmaps/nginx-load-balancer-microk8s-conf: dial tcp 127.0.0.1:16443: connect: connection refused
error when retrieving current configuration of:
Resource: "apps/v1, Resource=daemonsets", GroupVersionKind: "apps/v1, Kind=DaemonSet"
Name: "nginx-ingress-microk8s-controller", Namespace: "ingress"
Object: &{map["apiVersion":"apps/v1" "kind":"DaemonSet" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "labels":map["microk8s-application":"nginx-ingress-microk8s"] "name":"nginx-ingress-microk8s-controller" "namespace":"ingress"] "spec":map["selector":map["matchLabels":map["name":"nginx-ingress-microk8s"]] "template":map["metadata":map["labels":map["name":"nginx-ingress-microk8s"]] "spec":map["containers":[map["args":["/nginx-ingress-controller" "--configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf" "--publish-status-address=127.0.0.1"] "env":[map["name":"POD_NAME" "valueFrom":map["fieldRef":map["fieldPath":"metadata.name"]]] map["name":"POD_NAMESPACE" "valueFrom":map["fieldRef":map["fieldPath":"metadata.namespace"]]]] "image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller-amd64:0.25.1" "livenessProbe":map["httpGet":map["path":"/healthz" "port":'\u280e' "scheme":"HTTP"] "initialDelaySeconds":'\x1e' "timeoutSeconds":'\x05'] "name":"nginx-ingress-microk8s" "ports":[map["containerPort":'P'] map["containerPort":'\u01bb']]]] "dnsPolicy":"ClusterFirstWithHostNet" "hostNetwork":%!q(bool=true) "serviceAccountName":"nginx-ingress-microk8s-serviceaccount" "terminationGracePeriodSeconds":'<']]]]}
from server for: "/home/mike/snap/microk8s/936/tmp/temp.ingress.yaml": Get https://127.0.0.1:16443/apis/apps/v1/namespaces/ingress/daemonsets/nginx-ingress-microk8s-controller: dial tcp 127.0.0.1:16443: connect: connection refused
Failed to enable ingress

[balchua]

Hi @octopus-prime I've seen this happen when the cluster is not fully ready.

Usually i do microk8s.status --wait-ready before enabling addons.

[ktsakalozos]

@octopus-prime I cannot reproduce this. Could you please attach the microk8s.inspect tarball? In addition to what @balchua suggested could you try to enable each addon separately? The dns addon is a bit special because it has to restart services. It might be that there is a delay in services coming up and the triggering of the next addon.

[rzr]

Well I guess it wont harm to wait in enabler tool too, no ?
I have myself observed a possible regression with ingresses.

Update: I just discoverd that CPU needs SSE 4.2 instruction set, I can file a bug if it helps
(v1.15.4 is not affected but v1.16.0 is)

[ktsakalozos]

@rzr we recently added the microk8s.status --wait-ready in the microk8s.enable wrapper https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/microk8s-enable.wrapper#L38 . It might worth making the microk8s.enable ingress addon blocking (ie wait for services to come up). Any tips on how to reproduce this race condition?

[octopus-prime]

Running microk8s.enable separately for each add-on works. Thanks!

[rzr]

is it same issue as:
https://github.com/ubuntu/microk8s/issues/705
?

[ktsakalozos]

Yes it seems to be the same only in rev 936 we have some mitigation: https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/microk8s-enable.wrapper#L38

Unfortunately I cannot reproduce this anymore.

[octopus-prime]

Unfortunately I cannot reproduce this anymore.

Revision 940 works like a charm:

mike@workstation:~/idea/demo$ sudo snap install microk8s --channel=edge --classic
microk8s (edge) v1.16.1 aus Canonical✓ installiert
mike@workstation:~/idea/demo$ microk8s.enable dns ingress registry helm
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
Enabling Ingress
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled
Enabling the private registry
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
The registry is enabled
Enabling Helm
Fetching helm version v2.14.3.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 25.3M  100 25.3M    0     0  7474k      0  0:00:03  0:00:03 --:--:-- 7476k
Helm is enabled
mike@workstation:~/idea/demo$ snap list | grep microk8s
microk8s              v1.16.1                     940   edge          canonical*   classic
mike@workstation:~/idea/demo$ 

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.