Microk8s: Cannot schedule PODs: NodeHasDiskPressure

Created on 26 May 2018  路  5Comments  路  Source: ubuntu/microk8s

I am trying microk8s on Ubuntu 18.04 and it cannot run any POD, these are the statuses after each command:

after fresh install (no dns, no dashboard)

Name:               monotop
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=monotop
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Sat, 26 May 2018 11:09:11 -0600
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Sat, 26 May 2018 11:09:31 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Sat, 26 May 2018 11:09:31 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Sat, 26 May 2018 11:09:31 -0600   Sat, 26 May 2018 11:09:21 -0600   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Sat, 26 May 2018 11:09:31 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 26 May 2018 11:09:31 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.36.100
  Hostname:    monotop
Capacity:
 cpu:                4
 ephemeral-storage:  575354004Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             8074920Ki
 pods:               110
Allocatable:
 cpu:                4
 ephemeral-storage:  530246249209
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             7972520Ki
 pods:               110
System Info:
 Machine ID:                 813e56ef1c4f171bda95b46b5448007c
 System UUID:                CA17CB86-CBF2-E054-A153-18E4E8C4154B
 Boot ID:                    d69b0b06-7267-40bd-80e8-b53992bf96c5
 Kernel Version:             4.15.0-12-generic
 OS Image:                   Ubuntu 18.04 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://1.13.1
 Kubelet Version:            v1.10.3
 Kube-Proxy Version:         v1.10.3
ExternalID:                  monotop
Non-terminated Pods:         (0 in total)
  Namespace                  Name    CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----    ------------  ----------  ---------------  -------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  0 (0%)        0 (0%)      0 (0%)           0 (0%)
Events:
  Type    Reason                   Age                From                 Message
  ----    ------                   ----               ----                 -------
  Normal  Starting                 33s                kube-proxy, monotop  Starting kube-proxy.
  Normal  Starting                 30s                kubelet, monotop     Starting kubelet.
  Normal  NodeHasSufficientPID     28s (x5 over 30s)  kubelet, monotop     Node monotop status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  28s                kubelet, monotop     Updated Node Allocatable limit across pods
  Normal  NodeHasSufficientDisk    27s (x6 over 30s)  kubelet, monotop     Node monotop status is now: NodeHasSufficientDisk
  Normal  NodeHasSufficientMemory  27s (x6 over 30s)  kubelet, monotop     Node monotop status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    27s (x6 over 30s)  kubelet, monotop     Node monotop status is now: NodeHasNoDiskPressure

after microk8s.enable dns, node status

Name:               monotop
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=monotop
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Sat, 26 May 2018 11:09:11 -0600
Taints:             <none>
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Sat, 26 May 2018 11:12:30 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Sat, 26 May 2018 11:12:30 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Sat, 26 May 2018 11:12:30 -0600   Sat, 26 May 2018 11:11:10 -0600   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Sat, 26 May 2018 11:12:30 -0600   Sat, 26 May 2018 11:09:11 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 26 May 2018 11:12:30 -0600   Sat, 26 May 2018 11:11:10 -0600   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.36.100
  Hostname:    monotop
Capacity:
 cpu:                4
 ephemeral-storage:  575354004Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             8074920Ki
 pods:               110
Allocatable:
 cpu:                4
 ephemeral-storage:  530246249209
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             7972520Ki
 pods:               110
System Info:
 Machine ID:                 813e56ef1c4f171bda95b46b5448007c
 System UUID:                CA17CB86-CBF2-E054-A153-18E4E8C4154B
 Boot ID:                    d69b0b06-7267-40bd-80e8-b53992bf96c5
 Kernel Version:             4.15.0-12-generic
 OS Image:                   Ubuntu 18.04 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://1.13.1
 Kubelet Version:            v1.10.3
 Kube-Proxy Version:         v1.10.3
ExternalID:                  monotop
Non-terminated Pods:         (0 in total)
  Namespace                  Name    CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----    ------------  ----------  ---------------  -------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  0 (0%)        0 (0%)      0 (0%)           0 (0%)
Events:
  Type     Reason                   Age              From                 Message
  ----     ------                   ----             ----                 -------
  Normal   Starting                 3m               kube-proxy, monotop  Starting kube-proxy.
  Normal   Starting                 3m               kubelet, monotop     Starting kubelet.
  Normal   NodeAllocatableEnforced  3m               kubelet, monotop     Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientPID     3m (x5 over 3m)  kubelet, monotop     Node monotop status is now: NodeHasSufficientPID
  Normal   NodeHasSufficientDisk    3m (x6 over 3m)  kubelet, monotop     Node monotop status is now: NodeHasSufficientDisk
  Normal   NodeHasNoDiskPressure    3m (x6 over 3m)  kubelet, monotop     Node monotop status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientMemory  3m (x6 over 3m)  kubelet, monotop     Node monotop status is now: NodeHasSufficientMemory
  Normal   NodeNotReady             1m               kubelet, monotop     Node monotop status is now: NodeNotReady
  Normal   Starting                 1m               kubelet, monotop     Starting kubelet.
  Normal   NodeHasSufficientDisk    1m               kubelet, monotop     Node monotop status is now: NodeHasSufficientDisk
  Normal   NodeHasNoDiskPressure    1m (x2 over 1m)  kubelet, monotop     Node monotop status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     1m               kubelet, monotop     Node monotop status is now: NodeHasSufficientPID
  Normal   NodeHasSufficientMemory  1m               kubelet, monotop     Node monotop status is now: NodeHasSufficientMemory
  Normal   NodeAllocatableEnforced  1m               kubelet, monotop     Updated Node Allocatable limit across pods
  Normal   NodeHasDiskPressure      1m               kubelet, monotop     Node monotop status is now: NodeHasDiskPressure
  Normal   NodeReady                1m               kubelet, monotop     Node monotop status is now: NodeReady
  Warning  EvictionThresholdMet     39s              kubelet, monotop     Attempting to reclaim imagefs
  Warning  EvictionThresholdMet     9s (x8 over 1m)  kubelet, monotop     Attempting to reclaim nodefs

as you can see it reports DiskPressure, but there is around 30G free on my system, the Pod statuses are:

$ microk8s.kubectl get pods --all-namespaces
NAMESPACE     NAME                        READY     STATUS    RESTARTS   AGE
kube-system   kube-dns-598d7bf7d4-q26rl   0/3       Pending   0          2m

any advice is appreciated.

picture edsiper

Most helpful comment

Hi @edsiper ,

In microk8s your machine is also the node kubernetes is using. Disk pressure means you do not have enough resources for kubernetes to schedule pods. You can either free disk space by deleting files, or set the pod eviction limits as described in [1].

To set the pod eviction limits you have to set the --eviction-hard and --eviction-minimum-reclaim parameters in kubelet. The arguments used to start the daemons microk8s ships are under /var/snap/microk8s/current/args/. I suggest you append the following line in /var/snap/microk8s/current/args/kubelet

 --eviction-hard="memory.available<500Mi,nodefs.available<1Gi,imagefs.available<1Gi"

And then you will need to restart kubelet: sudo systemctl restart snap.microk8s.daemon-kubelet

You bring forward a valid point, the default eviction limits make sense for a server setup but may be to conservative for a standalone dev environment. We should set sensible defaults here: https://github.com/juju-solutions/microk8s/blob/master/microk8s-resources/default-args/kubelet

Thanks

[1] https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/

picture ktsakalozos on 27 May 2018
👍3 🎉1

All 5 comments

Hi @edsiper ,

In microk8s your machine is also the node kubernetes is using. Disk pressure means you do not have enough resources for kubernetes to schedule pods. You can either free disk space by deleting files, or set the pod eviction limits as described in [1].

To set the pod eviction limits you have to set the --eviction-hard and --eviction-minimum-reclaim parameters in kubelet. The arguments used to start the daemons microk8s ships are under /var/snap/microk8s/current/args/. I suggest you append the following line in /var/snap/microk8s/current/args/kubelet

 --eviction-hard="memory.available<500Mi,nodefs.available<1Gi,imagefs.available<1Gi"

And then you will need to restart kubelet: sudo systemctl restart snap.microk8s.daemon-kubelet

You bring forward a valid point, the default eviction limits make sense for a server setup but may be to conservative for a standalone dev environment. We should set sensible defaults here: https://github.com/juju-solutions/microk8s/blob/master/microk8s-resources/default-args/kubelet

Thanks

[1] https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/

ktsakalozos picture ktsakalozos on 27 May 2018
👍3 🎉1

that fixed the issue!, thanks for your help.

I'd suggest those values comes by default...

edsiper picture edsiper on 27 May 2018

I agree, I'll open a pull request to start that process

marcoceppi picture marcoceppi on 27 May 2018

Works here @ktsakalozos. I was going through the same issue.

viniciusbds picture viniciusbds on 3 Jun 2019

Can't those arguments be passed by kubectl or API ?

rzr picture rzr on 2 Aug 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

carmine picture carmine  路  4Comments
qbx2 picture qbx2  路  4Comments
termie picture termie  路  5Comments
khteh picture khteh  路  5Comments
avarf picture avarf  路  5Comments