Microk8s: storage enabled but no persistent volume found in cluster
v1.12.2
inspection-report-20181115_174033.tar.gz
I have enabled storage microk8s.enable storage however, microk8s.kubectl get pv shows "No resources found". As a result, all the PVCs are in pending state. Please advise. THanks!
khteh
👍2
All 19 comments
Hi @khteh ,
microk8s.enable storage will set a default storage class as you can see here (get pv is expected to be empty):
> microk8s.kubectl get sc
NAME PROVISIONER AGE
microk8s-hostpath (default) microk8s.io/hostpath 62s
Can you make sure that microk8s.kubectl get all -n kube-system reports back a deployment.apps/hostpath-provisioner and a corresponding pod? For some reason the microk8s.inspect report you attached does not include the kubectl get all output as if your api server is not running.
Here is a PVC we use during our testing https://github.com/ubuntu/microk8s/blob/master/tests/templates/pvc.yaml
Would would you like to share a minimal PVC that is failing for you?
Thank you for using microk8s and for opening this issue.
ktsakalozos
on 15 Nov 2018
👍2
Hi @ktsakalozos,
I will try the "microk8s.kubectl get all -n kube-system" command tomorrow when I get back to the office. I was able to build StatefuleSet which consists of a Service, one pod, and one PVC. I see all these but their status are in pending state waiting for the default PV to be dynamically brought up which is not for some reason. Just in case that the API Server is not running for some reason, what can I do about it? How can I find out the reason of such situation? This is my PVC in my StatefulSet:
`
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Gi
`
You are welcome. microk8s is much better than https://github.com/kubernetes/kubernetes.
khteh
on 15 Nov 2018
@khteh, first restart microk8s to make sure all services are up:
sudo snap disable microk8s
sudo snap enable microk8s
Wait a few seconds so that all services have initialized and try a microk8s.kubectl get all --all-namespaces to get the status of all pods services deployments running on your microk8s.
Do also a microk8s.kubectl get no to see if your node has registered with the apiserver.
microk8s.inspect will run through all the services and tell you if anyone is failing.
Could you share the full set of manifests you are applying or at least a complete minimal example of a setup that is failing?
Finally, you should know that microk8s is using most of the upstream kubernetes binaries found in https://github.com/kubernetes/kubernetes. And I agree with your point that kubernetes by itself is hard to setup; this is exactly what microk8s solves.
ktsakalozos
on 15 Nov 2018
❤1
inspection-report-20181115_174033.tar.gz
k get all --all-namespaces:
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/default-http-backend-587b7d64b5-87kfk 1/1 Running 1 15h
default pod/mysql-0 0/1 Pending 0 90s
default pod/nginx-ingress-microk8s-controller-slrtw 1/1 Running 1 15h
istio-system pod/grafana-689f8f94c4-72nk4 1/1 Running 1 15h
istio-system pod/istio-citadel-74fc7fd7c-2vrzv 0/1 Error 0 15h
istio-system pod/istio-cleanup-secrets-9rzhf 0/1 Completed 0 15h
istio-system pod/istio-egressgateway-6984cd8bd7-zmms8 0/1 Completed 105 15h
istio-system pod/istio-galley-6c65f59796-lg785 0/1 ContainerCreating 0 15h
istio-system pod/istio-ingressgateway-66c647cdb6-sth6p 0/1 Error 104 15h
istio-system pod/istio-pilot-c7c9568f7-z5hhw 0/2 ContainerCreating 0 15h
istio-system pod/istio-policy-6b6db7547-stlt2 0/2 Completed 0 15h
istio-system pod/istio-sidecar-injector-5b7d457c87-fglvx 0/1 ContainerCreating 0 15h
istio-system pod/istio-statsd-prom-bridge-67bbcc746c-v2qzd 1/1 Running 2 15h
istio-system pod/istio-telemetry-64859fb894-r5px9 2/2 Running 2 15h
istio-system pod/istio-tracing-6445d6dbbf-4g5tv 0/1 Running 180 15h
istio-system pod/prometheus-6967c997cf-2xql9 0/1 Running 1 15h
istio-system pod/servicegraph-c8747cf8d-6mhvw 0/1 CrashLoopBackOff 301 15h
kube-system pod/heapster-v1.5.2-74f7978789-n5mqn 0/4 Error 0 15h
kube-system pod/hostpath-provisioner-7d7c578f6b-9wz4x 0/1 ContainerCannotRun 147 15h
kube-system pod/kube-dns-67b548dcff-r4c2l 2/3 Running 350 15h
kube-system pod/kubernetes-dashboard-67d4c89764-4skz7 0/1 Error 168 15h
kube-system pod/monitoring-influxdb-grafana-v4-8467db6558-xwdtw 0/2 Completed 0 15h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/default-http-backend ClusterIP 10.152.183.78 <none> 80/TCP 15h
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 21h
istio-system service/grafana ClusterIP 10.152.183.11 <none> 3000/TCP 15h
istio-system service/istio-citadel ClusterIP 10.152.183.53 <none> 8060/TCP,9093/TCP 15h
istio-system service/istio-egressgateway ClusterIP 10.152.183.81 <none> 80/TCP,443/TCP 15h
istio-system service/istio-galley ClusterIP 10.152.183.150 <none> 443/TCP,9093/TCP 15h
istio-system service/istio-ingressgateway LoadBalancer 10.152.183.109 <pending> 80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:32243/TCP,8060:30102/TCP,15030:32333/TCP,15031:32628/TCP 15h
istio-system service/istio-pilot ClusterIP 10.152.183.112 <none> 15010/TCP,15011/TCP,8080/TCP,9093/TCP 15h
istio-system service/istio-policy ClusterIP 10.152.183.22 <none> 9091/TCP,15004/TCP,9093/TCP 15h
istio-system service/istio-sidecar-injector ClusterIP 10.152.183.77 <none> 443/TCP 15h
istio-system service/istio-statsd-prom-bridge ClusterIP 10.152.183.164 <none> 9102/TCP,9125/UDP 15h
istio-system service/istio-telemetry ClusterIP 10.152.183.120 <none> 9091/TCP,15004/TCP,9093/TCP,42422/TCP 15h
istio-system service/jaeger-agent ClusterIP None <none> 5775/UDP,6831/UDP,6832/UDP 15h
istio-system service/jaeger-collector ClusterIP 10.152.183.230 <none> 14267/TCP,14268/TCP 15h
istio-system service/jaeger-query ClusterIP 10.152.183.166 <none> 16686/TCP 15h
istio-system service/prometheus ClusterIP 10.152.183.127 <none> 9090/TCP 15h
istio-system service/servicegraph ClusterIP 10.152.183.139 <none> 8088/TCP 15h
istio-system service/tracing ClusterIP 10.152.183.99 <none> 80/TCP 15h
istio-system service/zipkin ClusterIP 10.152.183.79 <none> 9411/TCP 15h
kube-system service/heapster ClusterIP 10.152.183.229 <none> 80/TCP 15h
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP 15h
kube-system service/kubernetes-dashboard ClusterIP 10.152.183.237 <none> 443/TCP 15h
kube-system service/monitoring-grafana ClusterIP 10.152.183.19 <none> 80/TCP 15h
kube-system service/monitoring-influxdb ClusterIP 10.152.183.182 <none> 8083/TCP,8086/TCP 15h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
default daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 15h
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
default deployment.apps/default-http-backend 1 1 1 0 15h
istio-system deployment.apps/grafana 1 1 1 0 15h
istio-system deployment.apps/istio-citadel 1 1 1 0 15h
istio-system deployment.apps/istio-egressgateway 1 1 1 1 15h
istio-system deployment.apps/istio-galley 1 1 1 0 15h
istio-system deployment.apps/istio-ingressgateway 1 1 1 0 15h
istio-system deployment.apps/istio-pilot 1 1 1 0 15h
istio-system deployment.apps/istio-policy 1 1 1 0 15h
istio-system deployment.apps/istio-sidecar-injector 1 1 1 0 15h
istio-system deployment.apps/istio-statsd-prom-bridge 1 1 1 1 15h
istio-system deployment.apps/istio-telemetry 1 1 1 0 15h
istio-system deployment.apps/istio-tracing 1 1 1 0 15h
istio-system deployment.apps/prometheus 1 1 1 1 15h
istio-system deployment.apps/servicegraph 1 1 1 0 15h
kube-system deployment.apps/heapster-v1.5.2 1 1 1 1 15h
kube-system deployment.apps/hostpath-provisioner 1 1 1 0 15h
kube-system deployment.apps/kube-dns 1 1 1 0 15h
kube-system deployment.apps/kubernetes-dashboard 1 1 1 0 15h
kube-system deployment.apps/monitoring-influxdb-grafana-v4 1 1 1 1 15h
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/default-http-backend-587b7d64b5 1 1 0 15h
istio-system replicaset.apps/grafana-689f8f94c4 1 1 0 15h
istio-system replicaset.apps/istio-citadel-74fc7fd7c 1 1 0 15h
istio-system replicaset.apps/istio-egressgateway-6984cd8bd7 1 1 1 15h
istio-system replicaset.apps/istio-galley-6c65f59796 1 1 0 15h
istio-system replicaset.apps/istio-ingressgateway-66c647cdb6 1 1 0 15h
istio-system replicaset.apps/istio-pilot-c7c9568f7 1 1 0 15h
istio-system replicaset.apps/istio-policy-6b6db7547 1 1 0 15h
istio-system replicaset.apps/istio-sidecar-injector-5b7d457c87 1 1 0 15h
istio-system replicaset.apps/istio-statsd-prom-bridge-67bbcc746c 1 1 1 15h
istio-system replicaset.apps/istio-telemetry-64859fb894 1 1 0 15h
istio-system replicaset.apps/istio-tracing-6445d6dbbf 1 1 0 15h
istio-system replicaset.apps/prometheus-6967c997cf 1 1 1 15h
istio-system replicaset.apps/servicegraph-c8747cf8d 1 1 0 15h
kube-system replicaset.apps/heapster-v1.5.2-74f7978789 1 1 1 15h
kube-system replicaset.apps/hostpath-provisioner-7d7c578f6b 1 1 0 15h
kube-system replicaset.apps/kube-dns-67b548dcff 1 1 0 15h
kube-system replicaset.apps/kubernetes-dashboard-67d4c89764 1 1 0 15h
kube-system replicaset.apps/monitoring-influxdb-grafana-v4-8467db6558 1 1 1 15h
NAMESPACE NAME DESIRED CURRENT AGE
default statefulset.apps/mysql 1 1 14h
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
default horizontalpodautoscaler.autoscaling/istio-pilot Deployment/istio-pilot <unknown>/55% 1 1 0 15h
istio-system horizontalpodautoscaler.autoscaling/istio-egressgateway Deployment/istio-egressgateway <unknown>/60% 1 5 1 15h
istio-system horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway <unknown>/60% 1 5 1 15h
istio-system horizontalpodautoscaler.autoscaling/istio-policy Deployment/istio-policy <unknown>/80% 1 5 1 15h
istio-system horizontalpodautoscaler.autoscaling/istio-telemetry Deployment/istio-telemetry <unknown>/80% 1 5 1 15h
NAMESPACE NAME COMPLETIONS DURATION AGE
istio-system job.batch/istio-cleanup-secrets 1/1 6m39s 15h
istio-system job.batch/istio-grafana-post-install 0/1 15h 15h
PFA mysql_service.yml:
apiVersion: v1
kind: Service
metadata:
name: mysql-service
labels:
app: app1
spec:
ports:
- port: 3306
name: mysql
clusterIP: None # Headless
selector:
app: app1
tier: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
spec:
serviceName: "mysql-service"
podManagementPolicy: "Parallel" # Default is OrderedReady
replicas: 1 # Default is 1
selector:
matchLabels:
app: app1 # Has to match .spec.template.metadata.labels
template:
metadata:
labels:
app: app1 # Has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mysql
image: mysql:latest
env:
# Use secret in real usage
- name: MYSQL_ROOT_USERNAME
valueFrom:
secretKeyRef:
name: mysql-secret
key: username
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumeClaimTemplates:
- metadata:
name: mysql-persistent-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 50Gi
Yes I agree that https://github.com/kubernetes/kubernetes is very hard to setup. The kubeadm is buggy. Don't use it!
khteh
on 16 Nov 2018
Can you show me the logs and description of the hostpath pod?
microk8s.kubectl descirbe -n kube-system pod/hostpath-provisioner-7d7c578f6b-9wz4x
microk8s.kubectl logs -n kube-system pod/hostpath-provisioner-7d7c578f6b-9wz4x
We need to find out why the hostpath-provisioner says it cannot start. As you can see in https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/storage.yaml#L28 the hostpath-provisioner makes use of a directory in $SNAP_COMMON/default-storage (this is normally in /var/snap/microk8s/common/default-storage/) I wonder if there is something wrong with that part of your filesystem. Could you also do a df -h just to make sure we are not running out of disk and a mount to verify no one has mounted that path?
I am sure the team behind kubeadm is doing its best. Sometimes kubeadmin may work where microk8s would fail. Kubernetes deployments do not always go as planned and you need to file a bug/ticket/issue so that developers become aware of the specific problem you are facing and address it in the next release. This is exactly what we are doing in this issue and this is why your feedback is so valuable to us.
ktsakalozos
on 16 Nov 2018
$ microk8s.kubectl describe -n kube-system pod/hostpath-provisioner-7d7c578f6b-9wz4x
Name: hostpath-provisioner-7d7c578f6b-9wz4x
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: khteh-t580/192.168.86.93
Start Time: Thu, 15 Nov 2018 17:25:22 +0800
Labels: k8s-app=hostpath-provisioner
pod-template-hash=7d7c578f6b
Annotations: <none>
Status: Running
IP: 10.1.1.148
Controlled By: ReplicaSet/hostpath-provisioner-7d7c578f6b
Containers:
hostpath-provisioner:
Container ID: docker://8ce8c2b5672a6657c39c5c07de14115718554776d45bdb68df513b2581609d01
Image: cdkbot/hostpath-provisioner:latest
Image ID: docker-pullable://cdkbot/hostpath-provisioner@sha256:339f78eabc68ffb1656d584e41f121cb4d2b667565428c8dde836caf5b8a0228
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 16 Nov 2018 16:02:45 +0800
Finished: Fri, 16 Nov 2018 16:03:15 +0800
Ready: False
Restart Count: 228
Environment:
NODE_NAME: (v1:spec.nodeName)
PV_DIR: /var/snap/microk8s/common/default-storage
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cxvc6 (ro)
/var/snap/microk8s/common/default-storage from pv-volume (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pv-volume:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/common/default-storage
HostPathType:
default-token-cxvc6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cxvc6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 39m (x75 over 7h34m) kubelet, khteh-t580 Created container
Warning BackOff 4m31s (x1796 over 7h34m) kubelet, khteh-t580 Back-off restarting failed container
khteh@khteh-T580:/opt/tomcat/latest/conf 1865 $ microk8s.kubectl logs -n kube-system pod/hostpath-provisioner-7d7c578f6b-9wz4x
F1116 08:03:15.777865 1 hostpath-provisioner.go:162] Error getting server version: Get https://10.152.183.1:443/version: dial tcp 10.152.183.1:443: i/o timeout
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 2.6M 3.2G 1% /run
/dev/nvme0n1p5 467G 21G 422G 5% /
tmpfs 16G 432M 16G 3% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/loop0 88M 88M 0 100% /snap/core/5662
/dev/loop3 35M 35M 0 100% /snap/gtk-common-themes/808
/dev/loop1 142M 142M 0 100% /snap/slack/9
/dev/loop2 174M 174M 0 100% /snap/spotify/24
/dev/loop4 13M 13M 0 100% /snap/gnome-characters/124
/dev/loop5 13M 13M 0 100% /snap/gnome-characters/139
/dev/loop6 2.3M 2.3M 0 100% /snap/gnome-calculator/238
/dev/loop7 3.8M 3.8M 0 100% /snap/gnome-system-monitor/57
/dev/loop8 141M 141M 0 100% /snap/gnome-3-26-1604/70
/dev/loop9 88M 88M 0 100% /snap/core/5742
/dev/loop10 2.3M 2.3M 0 100% /snap/gnome-calculator/260
/dev/loop11 15M 15M 0 100% /snap/gnome-logs/45
/dev/loop12 130M 130M 0 100% /snap/postman/73
/dev/loop13 141M 141M 0 100% /snap/gnome-3-26-1604/74
/dev/loop14 215M 215M 0 100% /snap/microk8s/266
/dev/loop15 43M 43M 0 100% /snap/gtk-common-themes/701
/dev/nvme0n1p1 256M 35M 222M 14% /boot/efi
tmpfs 3.2G 92K 3.2G 1% /run/user/1000
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/51e89c93-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kube-dns-token-fs8wd
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/551417b8-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-cxvc6
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/550bb5ca-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kubernetes-dashboard-token-gfpj7
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/550bb5ca-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kubernetes-dashboard-certs
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5519a27f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/heapster-token-6wfcl
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5531c446-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-cxvc6
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c7aa878-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c8fdab2-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5ca1490e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c8fdab2-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5ca1490e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/55451370-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-xqg9m
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5cb82637-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/prometheus-token-pclsr
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/60605281-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c8c87d8-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5ce5e504-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/605f9293-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/istio-sidecar-injector-service-account-token-56srd
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-egressgateway-service-account-token-gb6gd
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/egressgateway-certs
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/egressgateway-ca-certs
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c85b470-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-galley-service-account-token-nbwhx
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5cc71578-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-citadel-service-account-token-shz25
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5cb1f38d-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-pilot-service-account-token-744rs
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/ingressgateway-certs
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/6093d201-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/nginx-ingress-microk8s-serviceaccount-token-xjljb
tmpfs 16G 12K 16G 1% /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-ingressgateway-service-account-token-ms4k5
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs
tmpfs 16G 0 16G 0% /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/ingressgateway-ca-certs
$ mount | grep microk8s
/var/lib/snapd/snaps/microk8s_266.snap on /snap/microk8s/266 type squashfs (ro,nodev,relatime,x-gdu.hide)
/dev/nvme0n1p5 on /var/snap/microk8s/common/var/lib/docker/aufs type ext4 (rw,relatime,errors=remount-ro)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/51e89c93-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kube-dns-token-fs8wd type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/551417b8-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-cxvc6 type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/550bb5ca-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kubernetes-dashboard-token-gfpj7 type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/550bb5ca-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/kubernetes-dashboard-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5519a27f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/heapster-token-6wfcl type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5531c446-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-cxvc6 type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c7aa878-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c8fdab2-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5ca1490e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-mixer-service-account-token-c8ppm type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c8fdab2-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5ca1490e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/55451370-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-xqg9m type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5cb82637-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/prometheus-token-pclsr type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/60605281-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c8c87d8-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5ce5e504-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/default-token-t2jvl type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/605f9293-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/istio-sidecar-injector-service-account-token-56srd type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-egressgateway-service-account-token-gb6gd type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/egressgateway-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c872d1f-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/egressgateway-ca-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c85b470-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-galley-service-account-token-nbwhx type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5cc71578-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-citadel-service-account-token-shz25 type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5cb1f38d-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-pilot-service-account-token-744rs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/ingressgateway-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/6093d201-e8b8-11e8-b22f-482ae31e6a94/volumes/kubernetes.io~secret/nginx-ingress-microk8s-serviceaccount-token-xjljb type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-ingressgateway-service-account-token-ms4k5 type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/istio-certs type tmpfs (rw,relatime)
tmpfs on /var/snap/microk8s/common/var/lib/kubelet/pods/5c89f30e-e8b8-11e8-877b-482ae31e6a94/volumes/kubernetes.io~secret/ingressgateway-ca-certs type tmpfs (rw,relatime)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/58e114cf1ca8f418836acc2e8fea7cbe606548339d61d0954e00a76144d939ae type aufs (rw,relatime,si=23c982543d5e018c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/a45da5a556b4ea997c28e06eed01233eeb9b088be0a6cd49de6bbfbb9112f55d type aufs (rw,relatime,si=23c982543d5e298c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/cb1295c190ea550fe8e5e6cd1084296573302ea69ca7500a8bef6f975377b425/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
shm on /var/snap/microk8s/common/var/lib/docker/containers/aac7ac34ee38587d2d2ba99b6d8979423a1a8096ddc9a627ead6b44f324b8d32/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/26a7941db660dfb7dece92de7201ec78ac1ca9d31ebb47b7e3b77dcacd136d18 type aufs (rw,relatime,si=23c98255b9f4718c,dio,dirperm1)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/420a01ae8acf type nsfs (rw)
shm on /var/snap/microk8s/common/var/lib/docker/containers/7fa4350eec0f9e114c6fb0230e3d503793b2eb6e03c6b84a1e904284727d5247/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/82c60613de6c type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/2224fec8c6dd28fb2870e099223650c401d99c0a6234622209d670fa80355541 type aufs (rw,relatime,si=23c9822b1a89f18c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/abc043d0d326e90108a42e16bf00bb6983407b9514630c02187a4108d012bd60/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/e5269f683d5e type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/33a023243f3ffbb244b311a3dd1a7e4ab72e11f5359eb665890e5e9109ef3013 type aufs (rw,relatime,si=23c982566fc0a18c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/83e3ccdc2fb4f32b3b27a64f63cc0f6998a26fa1c7dac73732da6557932eb9f7 type aufs (rw,relatime,si=23c982543ffe618c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/b191831554ada8255cf3a4005cc5b01c62e88e34667c5d7c94c26148ccca8e39/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/212155ebe03fe1629b546d96dbd45b8de77bd97426158a44f2af6262eac1a25a type aufs (rw,relatime,si=23c9825468bf898c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/9a652a555091476b7122be9aece19b4e18f2ef3f56941f223195cedcfb2f9f29/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/b7b112840553 type nsfs (rw)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/d7ebf1b80db5 type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/915e36bf3ba456361a42a10643df2f69818960644e72b935394904dd1f29e2ac type aufs (rw,relatime,si=23c98256302b518c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/4965899b7319f01a9d9195721252c616a0901881fe1f2e79188104b44bce1542/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/f2e878edec2841666f543b9ec953068444155c64a2a934a2ffd7ccc116ec66fd type aufs (rw,relatime,si=23c982543472e98c,dio,dirperm1)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/5bd527bd229f type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/f56f7329cf5be77f57dfd4b3e437643daf12d8a1fb69eb0301cf19dab22df26c type aufs (rw,relatime,si=23c98254231de18c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/c84060b439bde436940253cc6b19549e5d6c29b92d7930e0d8b0f7bd24d1491e/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/48c55dd6667ff0fbdbcd431d195e7c94572a4142c188443d9e6ff61919d7db78 type aufs (rw,relatime,si=23c9822b38cfb18c,dio,dirperm1)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/1ebd7617a7ce type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/0a97c5f0cfc5c0f6aea0244bc618cdc6019a9415fdac056da071645f53653bf6 type aufs (rw,relatime,si=23c982566fd9a98c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/7b806fe3f16a66cfa92836becd91e60e27f9b7acd6462b580dc8ed422a710564 type aufs (rw,relatime,si=23c982546cea698c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/8482c4fb0e8bb4c0da35cbe795900683a45330ebcd502592610c95850cbfc245 type aufs (rw,relatime,si=23c982543ff3718c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/0bc5bea8e2634a4bddef5510abab60bca8591b9703d7cae11263f75ca48e8d3a type aufs (rw,relatime,si=23c982543d7e618c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/b9adbacbff47ff7b3523f8d1dbff7634d660b4a0a7071a1af9957a315cbc005e/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/88235572343f6e7fdb35d463dd04f795b28c37d16f2eae8b57ce823e2d855128 type aufs (rw,relatime,si=23c9822b38cfe18c,dio,dirperm1)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/b074ed0a0490 type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/3ce95031945f0ac48008852942bd4ea0c1b5c9462f00e1376472e0cdaeabf48f type aufs (rw,relatime,si=23c9822b0019318c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/08b1774ba891eb7c81705984cefe786cd62c9701c2a3db7c62e192b96c96e39e type aufs (rw,relatime,si=23c982543d42f98c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/3130f5df2050e5ca11fb140bd02e5b8d9fa719b8ba7d46e5c9f8915f90daaf4c type aufs (rw,relatime,si=23c98254080a898c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/c4456c422fd6a9191becb983927202466545f8e675161a585c20a4e65a060fb1 type aufs (rw,relatime,si=23c982566fc0e18c,dio,dirperm1)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/134d1262caf8 type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/5a266551caab45f17adc3c86e3e8597ea4cb127f319ea761e1ea70cbdbb8a1db type aufs (rw,relatime,si=23c982543aea718c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/eddaee315890c843018a96efcb0745ef6e1a88fdaeb31950eaf7203d7f59d158 type aufs (rw,relatime,si=23c98255f7c3818c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/b17053e45ac444426cf106bdc895bc41be362359fac91c1dbf485261d986b22f/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
shm on /var/snap/microk8s/common/var/lib/docker/containers/9ca62799ef9a012b5826bc0c6b00fc0111966248dd7e426f8a6709172b390558/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/14b8abb9b32c type nsfs (rw)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/26fc23cac116 type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/6256fdd2737fbf9e7a4148c9fb4e9cc61e8f404a37892dcf8ac9baf735ba5e6a type aufs (rw,relatime,si=23c982540806a98c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/1f9e3e2aa2156a066f03b7c33dcb649f3252230cdb7c9ddf49bde13def6f9495 type aufs (rw,relatime,si=23c982540806a18c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/7f6379821d13b4071de0bb7e8daca052dccff4a5adff2f6257a048e7d9808c61/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
shm on /var/snap/microk8s/common/var/lib/docker/containers/025ad420be48ec66cfe586791baad80f10ca32b8a5c0e53c71abf44064382436/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/b901310ec78ba2020fbc65baa1917e42fe7f6d083aae2a2ad4b51b1ba264e0ce type aufs (rw,relatime,si=23c98254230d118c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/3c5b3438c668c96fc5639f607787bb332d6a1f40ea429a6e7a74d8af6e56b952/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/942b59b8f437 type nsfs (rw)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/34d98e6da1a0 type nsfs (rw)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/ef8ad846e01c type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/f1be265152384c95291c61a161261a811fbc041216a9faa039a7a2e5637ea995 type aufs (rw,relatime,si=23c98256a93b898c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/b118a16666055a47f160e0fe89fc4d22ef6eff90e670285989802d6414334913/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/c9ef248f05fd type nsfs (rw)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/1b532ea11e899c9c3378ccbe575695627c31e6dda9d99b7fd8d01d34d264b519 type aufs (rw,relatime,si=23c982565c92318c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/02a1edb7bfd9c2a4e10d8be15116e448a1015996a6cfbfa5cedaac8760aa1d25 type aufs (rw,relatime,si=23c9822b0459618c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/9b6ab2518fdd7e39d257a7dd9159991363ff56c4ac655575e4cd320c385425e7 type aufs (rw,relatime,si=23c9822b0459718c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/15ed273cac781a1fd53267b56386a8e42e14cd3dec8b9b92f00fb6aa31251dee type aufs (rw,relatime,si=23c982565c9f118c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/8966c2c6edb1edd073456e13e3686dd8038479b67f88f3903b56e423802f4c31 type aufs (rw,relatime,si=23c982565c9f298c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/aebd42d4f3f01ec33a203ef537f130aa3465d9088e387d626ec17f38124ab393 type aufs (rw,relatime,si=23c9825498cf618c,dio,dirperm1)
shm on /var/snap/microk8s/common/var/lib/docker/containers/5b9b11aa0b2257500fbc99d2530bac31d411eb404387cf5f13fdfce357100565/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/60d5843491e1 type nsfs (rw)
nsfs on /var/snap/microk8s/common/var/run/docker/netns/default type nsfs (rw)
shm on /var/snap/microk8s/common/var/lib/docker/containers/ac30b71c28fb31030bdaf666d8196b3401f29c16bbb326bd1192dd796ed95846/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/364b7050c19bc3616cea8e9b56d099ab4d3ce9f3ca917d039152171481d19bd9 type aufs (rw,relatime,si=23c982546770098c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/701fd54013e728e6efe4ea012231c372ba3f2cba5fcc2a9c0b60d5783a452f38 type aufs (rw,relatime,si=23c9822b9050818c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/b39ea8fe47c333d7366af57ce22f129ffe8f6916a66752f9aea642a775238bdb type aufs (rw,relatime,si=23c9825508f4e18c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/d5ca60bd85c239576c784cee64607bf3198bef57a35a1c2aa012577216c0dc49 type aufs (rw,relatime,si=23c9825468ba198c,dio,dirperm1)
none on /var/snap/microk8s/common/var/lib/docker/aufs/mnt/814f9cbaa726ec9378d1405b791373924b219ca9341f0342dc4799c4dc177341 type aufs (rw,relatime,si=23c982543fc1f98c,dio,dirperm1)
@khteh I see kube-dns is crashlooping. Have you enabled traffic forwarding?
sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0
sudo iptables -P FORWARD ACCEPT
And then do a disable/enable to force a restart on the services:
sudo snap disable microk8s
sudo snap enable microk8s
There is a sort explanation of the above in the troubleshooting section in the README https://github.com/ubuntu/microk8s/blob/master/README.md
ktsakalozos
on 16 Nov 2018
Yes, it works now. What's cbr0 interface?
Does it make sense to automatically set this as part of snap install?
khteh
on 16 Nov 2018
However, mysql-0 pod remains in "ImagePullBackoff" status:
k get all
NAME READY STATUS RESTARTS AGE
pod/default-http-backend-587b7d64b5-87kfk 1/1 Running 2 23h
pod/mysql-0 0/1 ImagePullBackOff 0 8h
pod/nginx-ingress-microk8s-controller-slrtw 1/1 Running 2 23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/default-http-backend ClusterIP 10.152.183.78 <none> 80/TCP 23h
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 30h
service/mysql-service ClusterIP None <none> 3306/TCP 8h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 23h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/default-http-backend 1 1 1 1 23h
NAME DESIRED CURRENT READY AGE
replicaset.apps/default-http-backend-587b7d64b5 1 1 1 23h
NAME DESIRED CURRENT AGE
statefulset.apps/mysql 1 1 8h
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/istio-pilot Deployment/istio-pilot <unknown>/55% 1 1 0 23h
cbr0 is the interface through which pods talk to each other.
Indeed, it is in our intentions to automate as many of these configuration steps as possible.
Can you microk8s.kubectl describe pod/mysql-0? Why does it say it cannot pull the Image?
ktsakalozos
on 16 Nov 2018
$ k describe pod mysql-0
Name: mysql-0
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: khteh-t580/192.168.86.93
Start Time: Fri, 16 Nov 2018 16:57:28 +0800
Labels: app=app1
controller-revision-hash=mysql-656f495585
statefulset.kubernetes.io/pod-name=mysql-0
Annotations: <none>
Status: Pending
IP: 10.1.1.182
Controlled By: StatefulSet/mysql
Containers:
mysql:
Container ID:
Image: mysql:latest
Image ID:
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment:
MYSQL_ROOT_USERNAME: <set to the key 'username' in secret 'mysql-secret'> Optional: false
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-secret'> Optional: false
Mounts:
/var/lib/mysql from mysql-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xqg9m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
mysql-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mysql-persistent-storage-mysql-0
ReadOnly: false
default-token-xqg9m:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xqg9m
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 10m (x2849 over 8h) default-scheduler pod has unbound immediate PersistentVolumeClaims
Normal Scheduled 7m34s default-scheduler Successfully assigned default/mysql-0 to khteh-t580
Warning Failed 2m49s kubelet, khteh-t580 Failed to pull image "mysql:latest": rpc error: code = Unknown desc = read tcp 192.168.86.93:35170->104.18.124.25:443: read: connection reset by peer
Warning Failed 2m49s kubelet, khteh-t580 Error: ErrImagePull
Normal BackOff 2m48s kubelet, khteh-t580 Back-off pulling image "mysql:latest"
Warning Failed 2m48s kubelet, khteh-t580 Error: ImagePullBackOff
Normal Pulling 2m34s (x2 over 7m32s) kubelet, khteh-t580 pulling image "mysql:latest"
I think I need this: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
khteh
on 16 Nov 2018
Is it still failing? It is strange to be failing to pull the image since other images from docker hub were fetches with no problems.
ktsakalozos
on 16 Nov 2018
If you are using a private registry have a look at this reply https://github.com/ubuntu/microk8s/issues/175#issuecomment-438988564 on the files you need to update.
ktsakalozos
on 16 Nov 2018
@ktsakalozos, do I need to do anything to pull from public registry?
khteh
on 16 Nov 2018
Hello!
After default install on Bionic, all my pods are in pending state. Did basic tshoot but nothing seems to be wrong.
dev@kube1:~$ sudo microk8s.kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 0/4 Pending 0 30m
kube-system kube-dns-6ccd496668-79d6g 0/3 Pending 0 30m
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0/1 Pending 0 30m
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 0/2 Pending 0 30m
dev@kube1:~$
shukalo83
on 4 Mar 2019
Hi @shukalo83
Could you share the output of microk8s.kubectl describe no and the tarball produced by microk8s.inspect? I suspect kubelet is failing to start. Often pods fail to be scheduled due to not enough free disk space.
Thanks
ktsakalozos
on 5 Mar 2019
Hello @ktsakalozos
I've created new issue here:
https://github.com/ubuntu/microk8s/issues/346
I've just checked the filesystem and you might be right.
I'll post the output of df in the original thread.
Thank you.
shukalo83
on 5 Mar 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
avarf
路
5Comments
arno01
路
4Comments
mthaddon
路
5Comments
carmine
路
3Comments
alphawolf1988
路
3Comments
Most helpful comment
Hi @khteh ,
microk8s.enable storagewill set a default storage class as you can see here (get pvis expected to be empty):Can you make sure that
microk8s.kubectl get all -n kube-systemreports back adeployment.apps/hostpath-provisionerand a corresponding pod? For some reason themicrok8s.inspectreport you attached does not include thekubectl get all outputas if your api server is not running.Here is a PVC we use during our testing https://github.com/ubuntu/microk8s/blob/master/tests/templates/pvc.yaml
Would would you like to share a minimal PVC that is failing for you?
Thank you for using microk8s and for opening this issue.