Amazon-eks-ami: [discussion] Default root size - 20GB

Created on 24 Jul 2018 · 5Comments · Source: awslabs/amazon-eks-ami

I run EKS cluster for one week, already get "no space" issue on EKS workers.

Found the code here that the default root size is 20GB.

https://github.com/awslabs/amazon-eks-ami/blob/master/eks-worker-al2.json#L22

      "volume_size": 20,

Should we increase it to 50GB or more?

I can update my codes to start with more space, but 20GB is too small, which I recommend to update in this AMI directly.

Source

ozbillwang

Most helpful comment

This can be overridden when you launch a node group by editing the CloudFormation template,

  NodeLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ...
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          DeleteOnTermination: true
          VolumeType: gp2
          VolumeSize: 50

micahhausler on 26 Jul 2018

👍5

All 5 comments

This can be overridden when you launch a node group by editing the CloudFormation template,

  NodeLaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ...
      BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          DeleteOnTermination: true
          VolumeType: gp2
          VolumeSize: 50

micahhausler on 26 Jul 2018

👍5

@micahhausler

Thanks.

I know how to customize this option, but my question is, should we increase the default value directly from source here?

ozbillwang on 27 Jul 2018

I think we’ll keep the default to 20. A more advanced configuration could include mounting a second volume for docker, so the kubelet can still operate even if the primary disk is full

micahhausler on 27 Jul 2018

👍3

For kops, they use a much higher default (120GB). In my opinion, this is way to low to run in with any sort of production traffic.

njfix6 on 9 Oct 2019

How does network traffic increase disk usage @njfix6?

I think it matters more what the nature of the workload is. For example we have node pools where the workload uses no (local) disk at all. If the root disk is big enough for a dozen images and 10MB log/container, then it is plenty big enough. In other cases all writeable disk are separate mounts as @micahhausler mentioned. For those workloads 120 GB or even 50 GB would be a waste of many TB of unused storage.