K3d: [COLLECTION] Pods evicted due to `NodeHasDiskPressure`

[iwilltry42]

About

This is some sort of "umbrella" issue to combine replies for related issues.

Issue

Pods go to evicted state...

• after issuing the k3d import-images command several times
• after creating some (more) pods

Related Issues (closed in favor of this one)

Closes #119 Pods go to evicted state after import an image multiple times
Closes #130 [BUG] Running out of space?

Debugging & Possible Solutions

• we found, that in most cases a docker system prune would fix the issue (temporarily)
  
  
  
  • it doesn't seem to be related directly to k3s/k3d
  
  
• possible cause: not enough disk space left on the host machine
  
  
  
  • on docker-for-desktop on MacOS, the host machine is the docker VM
  
  
  • it seems like kubelet's threshold for NodeHasDiskPressure is somewhere around 5% (of space left on the filesystem)
  
  
• @viklund found, that a docker update (MacOS) fixed the issue for him

[iwilltry42]

Updates from OPs @adibrastegarnia and @alexellis are very welcome :)

[zer0def]

Pretty sure this article covers most cases. I'd wager Docker's storage backend is set to something that doesn't properly clean up after itself (like devicemapper), something Docker's documentation discourages in favor of overlay2, unless you run a kernel that doesn't have it.

In my particular case, I've just added --agent-arg '--eviction-hard=imagefs.available<1%,nodefs.available<1%' --agent-arg '--eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%' to my k3d create invocation.

[iwilltry42]

Thanks for this @zer0def! I'll add it to the docs/wiki :+1:
@adibrastegarnia and @alexellis is this of help for you?

[iwilltry42]

@zer0def added to FAQ in dc19eb7

[kizzx2]

As of today, this is the command format that actually worked for me:

k3d c \
    --server-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%' \
    --server-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%'

[iwilltry42]

@kizzx2 so the same thing, but with server-arg instead of agent-arg? :thinking:

[kizzx2]

@iwilltry42

Yes, I am on k3d v1.6.0 and the --agent-arg version gives me this warning (and subsequently it doesn't work):

$ k3d c --agent-arg '--eviction-hard=imagefs.available<1%,nodefs.available<1%' --agent-arg '--eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%'
INFO[0000] Created cluster network with ID 57edfaa5f42198061311f6642c00be6e34014dbc92da5d0bbe58e05dd9440b65 
WARN[0000] --agent-arg supplied, but --workers is 0, so no agents will be created 

It's basically the same thing, but with the correct syntax (at least for my case) so that it actually gets passed into kubelet. (note that I also needed to add kubelet-arg in front)

[iwilltry42]

Ah yes, that makes sense @kizzx2 , it depends on your cluster setup. If you only have a single server with no workers, then --server-arg is the way to go (actually having both in multi-node clusters should be correct) :+1:

As this issue seems to be resolved by the kubelet flags, I'm going to close this issue now.
Feel free to reopen if issues persist.

[pnboot]

As of today, this is the command format that actually worked for me:

k3d c \
    --server-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%' \
    --server-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%'

k3d create cluster c2021 -w 3     --k3s-server-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%' \
    --k3s-server-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%' 

[iwilltry42]

for v3: https://k3d.io/faq/faq/#pods-evicted-due-to-lack-of-disk-space