Microk8s: Pods stuck in pending after install
Hello,
After installing microk8s from snap in Ubuntu 18.04.1 LTS (VirtualBox VM under Win10 Home) I've got pods in pending state.
dev@kube1:~$ sudo microk8s.kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 0/4 Pending 0 108m
kube-system kube-dns-6ccd496668-79d6g 0/3 Pending 0 108m
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0/1 Pending 0 108m
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 0/2 Pending 0 108m
dev@kube1:~$
Don't know what else to do or how to tshoot specific pods. I can't seem to find the logs...
inspection-report-20190304_152125.tar.gz
Inspect attached.
One of the cases mentions DNS problem on the host but I've checked it and it's ok:.
Here are the iptables rules in case it matters.
iptables.rules.txt
shukalo83
All 11 comments
@ktsakalozos wanted to check fs. Here's the output:
dev@kube1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.2G 0 1.2G 0% /dev
tmpfs 234M 1.1M 233M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 3.9G 3.3G 448M 89% /
tmpfs 1.2G 0 1.2G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.2G 0 1.2G 0% /sys/fs/cgroup
/dev/loop0 56M 56M 0 100% /snap/google-cloud-sdk/71
/dev/loop1 56M 56M 0 100% /snap/google-cloud-sdk/72
/dev/loop3 3.7M 3.7M 0 100% /snap/doctl/18
/dev/loop2 16M 16M 0 100% /snap/aws-cli/135
/dev/loop4 98M 98M 0 100% /snap/docker/321
/dev/loop5 87M 87M 0 100% /snap/core/4917
/dev/loop6 56M 56M 0 100% /snap/google-cloud-sdk/70
/dev/loop7 21M 21M 0 100% /snap/aws-cli/141
/dev/loop8 91M 91M 0 100% /snap/core/6405
/dev/loop9 220M 220M 0 100% /snap/microk8s/444
/dev/sda2 976M 142M 767M 16% /boot
tmpfs 234M 0 234M 0% /run/user/1000
dev@kube1:~$
I'll try and expand the LV group and let you know.
shukalo83
on 5 Mar 2019
Can you share the output of microk8s.kubectl get no? Is the node under disk pressure? I see a few log messages like: eviction manager: eviction thresholds have been met, but no pods are active to evict on kubelet.
ktsakalozos
on 5 Mar 2019
dev@kube1:~$ sudo microk8s.kubectl get no
NAME STATUS ROLES AGE VERSION
kube1 Ready <none> 19h v1.13.4
dev@kube1:~$
Might be under disk pressure
/dev/mapper/ubuntu--vg-ubuntu--lv 3.9G 3.5G 201M 95% /
I'll expand fs in a few minutes.
Cheers.
shukalo83
on 5 Mar 2019
dev@kube1:~$ sudo microk8s.kubectl get pod --all-namespaces
[sudo] password for dev:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 0/4 Pending 0 22h
kube-system kube-dns-6ccd496668-79d6g 0/3 Pending 0 22h
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0/1 Pending 0 22h
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 0/2 Pending 0 22h
dev@kube1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.2G 0 1.2G 0% /dev
tmpfs 234M 1.1M 233M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 84G 3.5G 77G 5% /
Hello. I've managed lv partition and the disk space is now ok. Still, the pods continue to be in pending state. I've rebooted after lv expansion.
shukalo83
on 5 Mar 2019
Great. Can you run again a microk8s.inspect and share again the output tarball? Could you also share the output of microk8s.kubectl describe no?
ktsakalozos
on 5 Mar 2019
dev@kube1:~$ sudo microk8s.kubectl describe no
Name: kube1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=kube1
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 04 Mar 2019 13:30:14 +0000
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 05 Mar 2019 12:42:34 +0000 Tue, 05 Mar 2019 12:32:33 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 05 Mar 2019 12:42:34 +0000 Tue, 05 Mar 2019 12:32:33 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 05 Mar 2019 12:42:34 +0000 Tue, 05 Mar 2019 12:32:33 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 05 Mar 2019 12:42:34 +0000 Tue, 05 Mar 2019 12:32:33 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.0.2.15
Hostname: kube1
Capacity:
cpu: 1
ephemeral-storage: 87656480Ki
hugepages-2Mi: 0
memory: 2395608Ki
pods: 110
Allocatable:
cpu: 1
ephemeral-storage: 86607904Ki
hugepages-2Mi: 0
memory: 2293208Ki
pods: 110
System Info:
Machine ID: c554505f84d7497bbfcc6d920878b522
System UUID: B2E48F70-FBE6-4B1B-BBF9-8B570C8C518F
Boot ID: 93a530b5-b921-4cb9-bd77-88f417d94ccd
Kernel Version: 4.15.0-45-generic
OS Image: Ubuntu 18.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.13.4
Kube-Proxy Version: v1.13.4
Non-terminated Pods: (4 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 100m (10%) 100m (10%) 184720Ki (8%) 184720Ki (8%) 23h
kube-system kube-dns-6ccd496668-79d6g 260m (26%) 0 (0%) 110Mi (4%) 170Mi (7%) 23h
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 23h
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 200m (20%) 200m (20%) 600Mi (26%) 600Mi (26%) 23h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 560m (56%) 300m (30%)
memory 911760Ki (39%) 973200Ki (42%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ImageGCFailed 22h kubelet, kube1 failed to garbage collect required amount of images. Wanted to free 358484377 bytes, but freed 0 bytes
Warning EvictionThresholdMet 21h (x508 over 22h) kubelet, kube1 Attempting to reclaim ephemeral-storage
Warning FreeDiskSpaceFailed 21h (x10 over 21h) kubelet, kube1 (combined from similar events): failed to garbage collect required amount of images. Wanted to free 363014553 bytes, but freed 0 bytes
Warning ImageGCFailed 3h19m (x11 over 21h) kubelet, kube1 (combined from similar events): failed to garbage collect required amount of images. Wanted to free 363039129 bytes, but freed 0 bytes
Normal Starting 3h17m kube-proxy, kube1 Starting kube-proxy.
Normal Starting 3h17m kubelet, kube1 Starting kubelet.
Normal NodeHasSufficientMemory 3h17m (x2 over 3h17m) kubelet, kube1 Node kube1 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 3h17m (x2 over 3h17m) kubelet, kube1 Node kube1 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 3h17m (x2 over 3h17m) kubelet, kube1 Node kube1 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 3h17m kubelet, kube1 Updated Node Allocatable limit across pods
Warning EvictionThresholdMet 3h14m (x17 over 3h17m) kubelet, kube1 Attempting to reclaim ephemeral-storage
Warning ImageGCFailed 3h12m kubelet, kube1 failed to garbage collect required amount of images. Wanted to free 621681049 bytes, but freed 0 bytes
Normal Starting 19m kube-proxy, kube1 Starting kube-proxy.
Normal Starting 10m kube-proxy, kube1 Starting kube-proxy.
Normal Starting 10m kubelet, kube1 Starting kubelet.
Normal NodeAllocatableEnforced 10m kubelet, kube1 Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 10m (x3 over 10m) kubelet, kube1 Node kube1 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 10m (x3 over 10m) kubelet, kube1 Node kube1 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 10m (x3 over 10m) kubelet, kube1 Node kube1 status is now: NodeHasSufficientPID
Warning Rebooted 10m kubelet, kube1 Node kube1 has been rebooted, boot id: 93a530b5-b921-4cb9-bd77-88f417d94ccd
Normal NodeReady 10m kubelet, kube1 Node kube1 status is now: NodeReady
dev@kube1:~$
Inspect and kubectl.describe.nodes attached.
inspection-report-20190305_124144.tar.gz
kubectl.nodes.describe.txt
shukalo83
on 5 Mar 2019
Huh, it seems something is happening now.
`NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 0/4 ContainerCreating 0 23h
kube-system kube-dns-6ccd496668-79d6g 0/3 ContainerCreating 0 23h
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0/1 ContainerCreating 0 23h
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 0/2 ContainerCreating 0 23h
dev@kube1:~$`
Stuck in ContainerCreating for too long. I consider this to be apparmor related.
What's the best way to proceed?
dev@kube1:~$ sudo microk8s.kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system heapster-v1.5.2-6bc7c4965d-sdkcz 0/4 ContainerCreating 0 23h
kube-system kube-dns-6ccd496668-79d6g 0/3 ContainerCreating 0 23h
kube-system kubernetes-dashboard-654cfb4879-dh6gb 0/1 ContainerCreating 0 23h
kube-system monitoring-influxdb-grafana-v4-6679c46745-md7tx 0/2 ContainerCreating 0 23h
dev@kube1:~$
I'll mark this as closed and open another issue.
shukalo83
on 5 Mar 2019
I am trying to follow the thread that was started with https://github.com/ubuntu/microk8s/issues/184
Can you please post the link to the new Issue that was created for this? @shukalo83
tdensmore
on 29 May 2019
👍1
In case anyone is following the same breadcrumb trail, the new issue created by @shukalo83 was #347
chradcliffe
on 12 Feb 2020
😄1
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
linxuyalun
路
3Comments
tvansteenburgh
路
6Comments
devbharat
路
3Comments
toxsick
路
4Comments
cnadeau
路
4Comments
Most helpful comment
In case anyone is following the same breadcrumb trail, the new issue created by @shukalo83 was #347