Amazon-eks-ami: Docker in EKS AMI will automatically cleanup when disk is almost full?

[t0ny-peng]

What happened:

We have a node using the latest EKS AMI. The node will run a lot of different images so the amount of disk used by Docker keeps increasing. The node has 256GB EBS mounted, and whenever the disk usage is around 85%, all the docker images will be deleted except for those who has container running. The behavior is very much like docker system prune.

But it is really annoying because we want to have some image to be persistent on the node as cache.

What you expected to happen:

We want to disable the auto cleanup feature, but don't know if it's a feature of EKS AMI, or docker.

How to reproduce it (as minimally and precisely as possible):

Just keep pulling different images on a relatively tiny machine. When the DISK usage is above some threshold, the auto cleanup will occur

Anything else we need to know?:

Environment:

• AWS Region: US west 2
• Instance Type(s): t3.medium
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.2
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.14
• AMI Version: ami-0c3a09195c638b01b
• Kernel (e.g. uname -a): Linux ip-172-31-35-52.us-west-2.compute.internal 4.14.146-119.123.amzn2.x86_64 #1 SMP Mon Sep 23 16:58:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Release information (run cat /etc/eks/release on a node):
  

BASE_AMI_ID="ami-0c3a09195c638b01b"
BUILD_TIME="Fri Sep 27 07:38:16 UTC 2019"
BUILD_KERNEL="4.14.138-114.102.amzn2.x86_64"
ARCH="x86_64"

[rajeshneo]

Does your kubelet have --eviction-hard or --eviction-soft parameters defined? It could be one reason for auto purging of images.

[max-rocket-internet]

You can change this in kubelet settings: https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/#image-collection

You just need to set the thresholds for your env.

[t0ny-peng]

@rajeshneo @max-rocket-internet You guys are awesome.