microk8s is not running. microk8s.inspect showing no error

Created on 2 Jan 2020 · 33Comments · Source: ubuntu/microk8s

Please run microk8s.inspect and attach the generated tarball to this issue.

wtf@k8s-master:~$ microk8s.inspect
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster

Building the report tarball
Report tarball is at /var/snap/microk8s/1107/inspection-report-20200102_011315.tar.gz

inspection-report-20200102_011315.tar.gz

wtf@k8s-master:~$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.

We appreciate your feedback. Thank you for using microk8s.

Source

ibigbug

👍1

Most helpful comment

Solved it for me @gavinB-orange - had to remove my previously installed kubectl and then it microk8s started working!

rm -rf /usr/local/bin/kubectl

antsankov on 4 Jun 2020

🎉3 🚀2 ❤2

All 33 comments

Your apiserver is complaining about an invalid bearer token.

Jan 02 01:13:06 k8s-master.syd.home microk8s.daemon-apiserver[4971]: E0102 01:13:06.280497    4971 authentication.go:104] Unable to authenticate the request due to an error: invalid bearer token
Jan 02 01:13:06 k8s-master.syd.home microk8s.daemon-apiserver[4971]: E0102 01:13:06.453439    4971 authentication.go:104] Unable to authenticate the request due to an error: invalid bearer token

Was this a fresh installation?

balchua on 2 Jan 2020

@balchua no it's not. I rebooted the machine after it's been running for a while

ibigbug on 2 Jan 2020

Thanks @ibigbug can you try to restart microk8s? microk8s.stop then microk8s.start to see if it resolve the issue?

balchua on 2 Jan 2020

@balchua not seem working

wtf@k8s-master:~$ microk8s.stop
[sudo] password for wtf:
Stopped.
wtf@k8s-master:~$ microk8s.start
Started.
Enabling pod scheduling
wtf@k8s-master:~$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.
wtf@k8s-master:~$

ibigbug on 2 Jan 2020

pod status

admin@k8s-master:~$ kubectl get po -n kube-system
NAME                                              READY   STATUS        RESTARTS   AGE
coredns-9b8997588-kldbv                           0/1     Pending       0          2d14h
coredns-9b8997588-xllr9                           0/1     Terminating   0          14d
dashboard-metrics-scraper-687667bb6c-kg6zd        1/1     Terminating   0          14d
dashboard-metrics-scraper-687667bb6c-sqdj4        0/1     Pending       0          2d13h
filebeat-p6nfk                                    1/1     Running       0          14d
filebeat-w55z9                                    1/1     Running       1          14d
heapster-v1.5.2-5c58f64f8b-4dfw2                  4/4     Terminating   0          14d
heapster-v1.5.2-5c58f64f8b-v5699                  0/4     Pending       0          2d13h
hostpath-provisioner-7b9cb5cdb4-f7jh7             1/1     Terminating   0          14d
hostpath-provisioner-7b9cb5cdb4-wgmwq             0/1     Pending       0          2d13h
kubernetes-dashboard-5c848cc544-4rlxr             1/1     Terminating   1          14d
kubernetes-dashboard-5c848cc544-j2vzv             0/1     Pending       0          2d14h
metricbeat-55f4fc45cb-5whm2                       1/1     Terminating   1          14d
metricbeat-55f4fc45cb-l49zf                       0/1     Pending       0          2d14h
metricbeat-cw92z                                  1/1     Running       0          14d
metricbeat-kkq8s                                  1/1     Running       1          14d
monitoring-influxdb-grafana-v4-6d599df6bf-lzqtw   2/2     Terminating   2          14d
monitoring-influxdb-grafana-v4-6d599df6bf-pfsdx   0/2     Pending       0          2d14h

ibigbug on 5 Jan 2020

Are you running multi nodes?

balchua on 5 Jan 2020

yes 1 master + 1 follower

ibigbug on 5 Jan 2020

Can you go to the worker/follower node and do a microk8s.stop and microk8s.start?

balchua on 5 Jan 2020

it doesn't actually allow me:

admin@k8s-node1:~$ microk8s.stop
This MicroK8s deployment is acting as a node in a cluster. Please use the microk8s.stop on the master.

ibigbug on 5 Jan 2020

Is it possible to make it a single node cluster to see if it is still running? I think you may need to do microk8s.leave or microk8s.remove-node something like that.

balchua on 5 Jan 2020

still not working. maybe I'll just reinstall..

ibigbug on 5 Jan 2020

You may want to pin it to a particular channel ex. 1.16 stable.

balchua on 5 Jan 2020

@ibigbug I see that the kubelets cannot register with the apiserver. The last time they registered with the API server was on the 22nd of Dec. The error you have looks like this:

Jan 02 01:13:06 k8s-master.syd.home microk8s.daemon-kubelet[9551]: E0102 01:13:06.297471    9551 kubelet.go:2263] node "k8s-master.syd.home" not found
Jan 02 01:13:06 k8s-master.syd.home microk8s.daemon-kubelet[9551]: E0102 01:13:06.399105    9551 kubelet.go:2263] node "k8s-master.syd.home" not found
Jan 02 01:13:06 k8s-master.syd.home microk8s.daemon-kubelet[9551]: E0102 01:13:06.487449    9551 reflector.go:156] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Unauthorized

Any idea what might have changed around then?

ktsakalozos on 7 Jan 2020

if it's saying node not found, might be due to the reboot of VM?

ibigbug on 8 Jan 2020

I also had the same issue. It happens with v1.17 only (other versions like v1.16, v1.15, v1.14) are ok. Also, it happens with v1.17 after I try to install kubeflow using
microk8s.enable kubeflow
which basically throws an error about Juju. To resolve that I did install Juju and lxd and did
juju bootstrap
after this
microk8s.status
gives me microk8s not running.

I reproduced this behaviour in another machine as well.

pankajxyz on 9 Jan 2020

👎1 👍1

Same error.
Running single node.
microk8s version:
installed: v1.17.2 (1173) 179MB classic

$ microk8s.start
Started.
Enabling pod scheduling
$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.
$ microk8s.inspect 
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/1173/inspection-report-20200202_114517.tar.gz

inspection-report-20200202_114517.tar.gz

TribalNightOwl on 2 Feb 2020

Either removing and re-installing fixed the issue or the version:
installed: v1.17.0 (1109) 179MB classic

$ snap remove microk8s 
microk8s removed

$ microk8s.status
bash: /snap/bin/microk8s.status: No such file or directory


$ sudo snap install microk8s --classic --channel=1.17/stable

microk8s (1.17/stable) v1.17.0 from Canonical✓ installed


$ microk8s.start
Started.
Enabling pod scheduling
node/blushy already uncordoned


$ microk8s.status
microk8s is running
addons:
cilium: disabled
dashboard: disabled
dns: disabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

TribalNightOwl on 2 Feb 2020

After several delete and re-installs, I narrowed it down to microk8s dying the moment I try to change the context to use.

I enabled DNS, then created two namespaces, then two contexts, I checked the status of microk8s after each command and it was running.

$ kubectl get namespaces 
NAME                 STATUS   AGE
default              Active   52s
jenkinsmaster-dev    Active   5s
jenkinsmaster-prod   Active   5s
kube-node-lease      Active   66s
kube-public          Active   66s
kube-system          Active   67s


$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s
    namespace: jenkinsmaster-dev
    user: admin
  name: jenkinsmaster-dev
- context:
    cluster: microk8s
    namespace: jenkinsmaster-prod
    user: admin
  name: jenkinsmaster-prod
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    password: bCtlMTl6dUhSVXlFb1hVRXpYcWs0QUpzbFc4dFpPd2hsb3U4UVA0UFo0VT0K
    username: admin


$ kubectl config current-context 
microk8s

After I did:

$ kubectl config use-context jenkinsmaster-dev 
Switched to context "jenkinsmaster-dev".


$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.

TribalNightOwl on 2 Feb 2020

@TribalNightOwl thanks for the info. When you added the context, did u add it to the file /var/snap/microk8s/current/credentials/client.config?
And the kubectl you are using is an alias?
Thanks again.

balchua on 2 Feb 2020

@TribalNightOwl your context jenkinsmaster-dev and jenkinsmaster-prod is pointing to a non existing cluster microk8s.
It should be microk8s-cluster.

balchua on 3 Feb 2020

@TribalNightOwl thanks for the info. When you added the context, did u add it to the file /var/snap/microk8s/current/credentials/client.config?

No, I just used these commands:

microk8s.kubectl config set-context jenkinsmaster-dev --namespace=jenkinsmaster-dev   --cluster=microk8s   --user=admin

microk8s.kubectl config set-context jenkinsmaster-prod --namespace=jenkinsmaster-prod   --cluster=microk8s   --user=admin

And the kubectl you are using is an alias?

yes:

alias kubectl='microk8s.kubectl'

TribalNightOwl on 3 Feb 2020

@TribalNightOwl your context jenkinsmaster-dev and jenkinsmaster-prod is pointing to a non existing cluster microk8s.
It should be microk8s-cluster.

I will try again and report back. Although I would argue that microk8s shouldn't stop running (and refuse to start) due to this.

TribalNightOwl on 3 Feb 2020

$ snap install microk8s --classic
microk8s v1.17.2 from Canonical✓ installed

$ microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
[sudo] password for hugo: 
DNS is enabled

$ kubectl apply -f namespaces.yaml 
namespace/jenkinsmaster-dev created
namespace/jenkinsmaster-prod created


$ kubectl get namespaces 
NAME                 STATUS   AGE
default              Active   94s
jenkinsmaster-dev    Active   4s
jenkinsmaster-prod   Active   4s
kube-node-lease      Active   107s
kube-public          Active   107s
kube-system          Active   108s

$ microk8s.kubectl config set-context jenkinsmaster-dev --namespace=jenkinsmaster-dev \
>   --cluster=microk8s-cluster \
>   --user=admin
Context "jenkinsmaster-dev" created.


$ microk8s.kubectl config set-context jenkinsmaster-prod --namespace=jenkinsmaster-prod \
>   --cluster=microk8s-cluster \
>   --user=admin
Context "jenkinsmaster-prod" created.


$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    namespace: jenkinsmaster-dev
    user: admin
  name: jenkinsmaster-dev
- context:
    cluster: microk8s-cluster
    namespace: jenkinsmaster-prod
    user: admin
  name: jenkinsmaster-prod
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    password: ZytkS1o5NVZhZWRTU0t3NnNReFhHaHpRcHRaaUxkaG1XNWFBTXFPbVNNaz0K
    username: admin

$ kubectl config use-context jenkinsmaster-dev 
Switched to context "jenkinsmaster-dev".

$ microk8s.status
microk8s is running
addons:
cilium: disabled
dashboard: disabled
dns: enabled
fluentd: disabled
gpu: disabled
helm3: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

BINGO! it didn't die this time.

$ kubectl config current-context 
jenkinsmaster-dev

TribalNightOwl on 3 Feb 2020

@TribalNightOwl microk8s is not actually dying. The status command uses the kubeconfig settings to verify the cluster's health. So if the kubeconfig is misconfigured it will not be able to gather kubernetes resources, hence it will say not running

The message can be misleading though.

balchua on 3 Feb 2020

👍1

$ microk8s.kubectl config set-context jenkinsmaster-dev --namespace=jenkinsmaster-fail   --cluster=microk8s   --user=admin

$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s
    namespace: jenkinsmaster-fail
    user: admin
  name: jenkinsmaster-dev
- context:
    cluster: microk8s-cluster
    namespace: jenkinsmaster-prod
    user: admin
  name: jenkinsmaster-prod
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: jenkinsmaster-dev
kind: Config
preferences: {}
users:
- name: admin
  user:
    password: ZytkS1o5NVZhZWRTU0t3NnNReFhHaHpRcHRaaUxkaG1XNWFBTXFPbVNNaz0K
    username: admin


$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.

sudo vi /var/snap/microk8s/current/credentials/client.config

Manually deleted this section:

- context:
    cluster: microk8s
    namespace: jenkinsmaster-fail
    user: admin
  name: jenkinsmaster-dev


$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    namespace: jenkinsmaster-prod
    user: admin
  name: jenkinsmaster-prod
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: jenkinsmaster-dev
kind: Config
preferences: {}
users:
- name: admin
  user:
    password: ZytkS1o5NVZhZWRTU0t3NnNReFhHaHpRcHRaaUxkaG1XNWFBTXFPbVNNaz0K
    username: admin

$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.

TribalNightOwl on 3 Feb 2020

Hold on, I got it:

My previous was still pointing to something non-existent.

I did:

 kubectl config use-context jenkinsmaster-prod
Switched to context "jenkinsmaster-prod".

$ microk8s.status
microk8s is running
addons:
cilium: disabled
dashboard: disabled
dns: enabled
fluentd: disabled
gpu: disabled
helm3: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

That completely proves your previous comment, thanks!

TribalNightOwl on 3 Feb 2020

How about changing the message:

Currently:

microk8s is not running. Use microk8s.inspect for a deeper inspection.

New:

microk8s is not running. Verify your config is valid and use microk8s.inspect for a deeper inspection.

Or:

microk8s is not running in cluster $CLUSTERNAME. Use microk8s.inspect for a deeper inspection.

Something that would make the user think about having a misconfigured client and not necessarily microk8s dying.

TribalNightOwl on 3 Feb 2020

👍2

How about changing the message

We could also detect such problems and suggest a fix in microk8s.inspect https://github.com/ubuntu/microk8s/blob/master/scripts/inspect.sh#L106

ktsakalozos on 3 Feb 2020

On my system I found that the problem went away after I updated the rather too old kubectl installed in /usr/local/bin on my system. I had assumed that microk8s would exclusively use it's own kubectl, but apparently not.