Eksctl: create nodegroup creates 2 volumes, the root doesn't honor volumeSize

[derekwlmscfa]

What happened?
create nodegroup is creating the EC2 nodes with two (not one) attached EBS volumes. The root volume is only 20GB, too small for use. The second volume is 100GB, honoring the volumeSize I specified.

What you expected to happen?
There should only be the one volume I specified in the config file, honoring the volumeSize. Or if two volumes are created, provide a way to control the root volume size.

How to reproduce it?
eksctl create nodegroup --include='ng-2020-10-26b*' --config-file=cfg.yaml

Anything else we need to know?
I did not see this behavior with other nodegroups I created in the past. Differences here are 1) different eksctl version, and 2) using Bottlerocket amiFamily.

Relevant config file portion (sensitive data redacted):

  - name: ng-2020-10-26br
    amiFamily: Bottlerocket
    ami: auto-ssm
    labels:
      "bottlerocket.aws/updater-interface-version": "1.0.0"
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
      withAddonPolicies:
        appMesh: true
        autoScaler: true
        ebs: true
    instanceType: c5n.4xlarge
    maxSize: 75
    desiredCapacity: 8
    minSize: 8
    privateNetworking: true
    volumeSize: 100
    volumeType: gp2

Versions
Please paste in the output of these commands:

$ eksctl version
0.30.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-065dce", GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean", BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Logs
Include the output of the command line when running eksctl. If possible, eksctl should be run with debug logs. For example:
eksctl get clusters -v 4
Make sure you redact any sensitive information before posting.
If the output is long, please consider a Gist.

[aclevername]

I've managed to reproduce this using the config given, thanks for reporting the issue @derekwlmscfa .

Looking into it I can see that the NodeGroupLaunchTemplate created as part of the CloudFormation stack for the nodegroup only specifies the 100gb volume in its storage section and is also respecting the gp2 volume type. However going to the EC2 instance I can see an additional 20gb volume mounted.

[derekwlmscfa]

I've managed to reproduce this using the config given, thanks for reporting the issue @derekwlmscfa .

Looking into it I can see that the NodeGroupLaunchTemplate created as part of the CloudFormation stack for the nodegroup only specifies the 100gb volume in its storage section and is also respecting the gp2 volume type. However going to the EC2 instance I can see an additional 20gb volume mounted.

Awesome! Thanks for jumping on this quickly; good to hear you can re-create it.

[aclevername]

I can reproduce this with any BottleRocket nodegroup configuration, any other AMI results in only the 100gb disk being present. I cannot find any mention about how or why this disk is being attached to the BottleRocket documentation either. Its definitely not coming from our side as our CloudFormation templates for the nodegroup only define the single 100gb disk.

[derekwlmscfa]

Thanks! I've raised a support ticket with AWS for this; I'll keep you posted on that.

[aclevername]

Hey @derekwlmscfa 👋 did you get a response from AWS?

[derekwlmscfa]

Hi, Jake, thanks for checking back. AWS did re-create it but no new news. I pinged them for an update.

[derekwlmscfa]

Hi, Jake, AWS did confirm that the Bottlerocket AMIs will have two attached volumes (this is deliberate, not an aberration). This can be confirmed with aws ec2 describe-images --image-ids ami-0c77a7529bf5dfaa1 (or any of the applicable AMIs listed here - see the BlockDeviceMappings).

As you mentioned, eksctl's CloudFormation template does properly override the VolumeSize for /dev/xvda volume, but not for the /dev/xvdb volume, which is mounted as root. Perhaps you could add the BlockDeviceMapping for this second volume to respect the given volumeSize.

[aclevername]

Thanks for following up @derekwlmscfa, good to know about this bottlerocket behaviour.

Perhaps you could add the BlockDeviceMapping for this second volume to respect the given volumeSize.

👍 makes sense

[aclevername]

For further clarity we reached out to the bottlerocket folks to gain a better understanding of the usage of the two volumes https://github.com/bottlerocket-os/bottlerocket/issues/1203#

/dev/xvda is used to hold the operating system itself and contains the active & passive partitions, the bootloader, the dm-verity data, and the data store for the Bottlerocket API. /dev/xvdb is used for all of the things you run on top of Bottlerocket: container images, storage for running containers, and persistent storage/volumes.

[michaelbeaumont]

Ok, so it sounds like we definitely want to be increasing the size of /dev/xvdb with volumeSize, which eksctl doesn't right now, have I understood correctly?

[derekwlmscfa]

Yes, I definitely think so, since that /dev/xvdb volume is what's available to us. In some cases, the 20GB default isn't even enough to pull the image.

[aclevername]

@derekwlmscfa should be fixed in the next release 😄

[derekwlmscfa]

@derekwlmscfa should be fixed in the next release 😄

Awesome, thanks!