Containers-roadmap: [EKS/Fargate] [Volumes]: Add support for EFS volumes to EKS Fargate Containers

[hlascelles]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Allow mounting of at EFS volumes onto Fargate tasks running in an EKS Cluster (specifically). This is a follow-up issue to the original that tracked adding this capability to AWS Kubernetes clusters in general, and has been delivered for ECS: https://github.com/aws/containers-roadmap/issues/53

Which service(s) is this request for?
Fargate, EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Like the linked issue, we understand that stateless containers are the way forward, but this change would give us the breathing space and motivation to move to Fargate.

Are you currently working around this issue?
We are staying on standard worker nodes.

Additional context
See the linked issue (that delivered for ECS) for use cases and ideas.

[jgoeres]

To add our motivation to 👍 this issue: we were researching to use Fargate for running Spark Applications with the K8s Spark operator. Alas, Spark effectively requires any input files to reside on a shared file system and this restriction effectively kills Fargate for our workload.

[ndydng]

To add our motivation to 👍 this issue: we were researching to use Fargate for running Spark Applications with the K8s Spark operator. Alas, Spark effectively requires any input files to reside on a shared file system and this restriction effectively kills Fargate for our workload.

@jgoeres we've some success with Apache Spark + S3 + EKS + Fargate using Hadoop AWS S3. You'll need to figure out a way to bring in the latest AWS SDK though since the one Spark is built with (2.7.4) uses a very old SDK that doesn't support container credentials. See:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html

[maynardflies]

To add our use case to this, we want to move our EKS workloads to Fargate to improve our scaling response but our image size is over 2GB, and takes a very long time to start a pod. Using EFS would allow us to move the vast majority of big stuff off into another filesystem and not have to pull it down i.e. from S3 which would take just as long. EFS with some provisioned throughput would allow us to access it as a block device and would be the last part of the puzzle to move to Fargate and unlock all that juicy serverless scaling goodness that goes along with it!

[bgsilvait]

Is HERE!!

https://aws.amazon.com/blogs/aws/new-aws-fargate-for-amazon-eks-now-supports-amazon-efs/

[mikestef9]

Support for EFS volumes running on EKS/Fargate pods is now available! See the launch blog and documentation for more details.

The UX is very similar to running EFS based pods on EC2 instances, except you no longer need to manually install or maintain the EFS CSI driver, as Fargate seamlessly handles that for any pods that request an EFS persistent volume claim.

Note that if your cluster also has EC2 instances running pods requiring EFS file systems, you still need to install the driver.

Today, this feature is available on newly created EKS clusters running Kubernetes version 1.17. Over the coming weeks, support will be rolled out to all (new and existing) EKS clusters running Kubernetes version 1.15 and above.

[jrtcppv]

I followed the blog post and am getting the following from kubectl describe pod <pod that uses EFS>:

Events:
  Type     Reason       Age        From                                                            Message
  ----     ------       ----       ----                                                            -------
  Normal   Scheduled    <unknown>  fargate-scheduler                                               Successfully assigned default/tusd-deployment-8d7fdc67-pd7dj to fargate-ip-192-168-178-193.us-east-2.compute.internal
  Warning  FailedMount  5s         kubelet, fargate-ip-192-168-178-193.us-east-2.compute.internal  MountVolume.SetUp failed for volume "efs-pv" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = Could not mount "fs-XXXXXXXX:/" at "/var/lib/kubelet/pods/02df7849-4429-4157-94a8-23e3177981a1/volumes/kubernetes.io~csi/efs-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o tls fs-XXXXXXXX:/ /var/lib/kubelet/pods/02df7849-4429-4157-94a8-23e3177981a1/volumes/kubernetes.io~csi/efs-pv/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "supervisord"
mount.nfs4: Connection reset by peer

I saw some similar errors here and here but not sure how to resolve this on my end. The pods just stay in ContainerCreating indefinitely. Any ideas?

[wochanda]

I followed the blog post and am getting the following from kubectl describe pod <pod that uses EFS>:

Events:
  Type     Reason       Age        From                                                            Message
  ----     ------       ----       ----                                                            -------
  Normal   Scheduled    <unknown>  fargate-scheduler                                               Successfully assigned default/tusd-deployment-8d7fdc67-pd7dj to fargate-ip-192-168-178-193.us-east-2.compute.internal
  Warning  FailedMount  5s         kubelet, fargate-ip-192-168-178-193.us-east-2.compute.internal  MountVolume.SetUp failed for volume "efs-pv" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = Could not mount "fs-XXXXXXXX:/" at "/var/lib/kubelet/pods/02df7849-4429-4157-94a8-23e3177981a1/volumes/kubernetes.io~csi/efs-pv/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o tls fs-XXXXXXXX:/ /var/lib/kubelet/pods/02df7849-4429-4157-94a8-23e3177981a1/volumes/kubernetes.io~csi/efs-pv/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "supervisord"
mount.nfs4: Connection reset by peer

I saw some similar errors here and here but not sure how to resolve this on my end. The pods just stay in ContainerCreating indefinitely. Any ideas?

The "Connection reset by peer" usually means something is blocking the NFS connection at the network layer, most commonly the security group either on the EFS mount targets or the K8S worker. Doublecheck that both are in the same VPC, and that security groups allow port 2049 to get from the worker to the mount target.

[jrtcppv]

Yes this actually turned out to be a security group issue. I was able to mount successfully after fixing this. Thanks!

[tarunreddy06]

Yes this actually turned out to be a security group issue. I was able to mount successfully after fixing this. Thanks!

Hi,
Could you please guide me about which security group to make changes ... I have ingress 2049 port allowed for EFS mount security group from same VPC, do I need to add anything at WorkerNode Security groups.

[jrtcppv]

@tarunreddy06 I created my kubernetes cluster with eksctl, which also creates a VPC with subnets and security groups. I created my EFS filesystem in the same VPC and added all three subnets for my EFS filesystem (reached in the AWS console via EFS>your filesystem>Network tab>Manage button) to the following security groups:

• eksctl-<clustername>-cluster-ClusterSharedNodeSecurityGroup-########
• eks-cluster-sg-<clustername>-#########

Not sure if both were required, but I was able to access my filesystem from pods running on Fargate afterwards.

[tarunreddy06]

@tarunreddy06 I created my kubernetes cluster with eksctl, which also creates a VPC with subnets and security groups. I created my EFS filesystem in the same VPC and added all three subnets for my EFS filesystem (reached in the AWS console via EFS>your filesystem>Network tab>Manage button) to the following security groups:

• eksctl-<clustername>-cluster-ClusterSharedNodeSecurityGroup-########
• eks-cluster-sg-<clustername>-#########

Not sure if both were required, but I was able to access my filesystem from pods running on Fargate afterwards.

Thank you @jrtcppv , I was able to follow the same steps and add subnets( cidrBlocks) of VPC to EFS which fixed the problem.