Copilot-cli: EFS Support for Copilot

[bvtujo]

This issue is meant to track the ongoing development of EFS support for Copilot.

Milestones

• [x] New CF template written for Mount Points and Volumes, allowing external EFS file systems
• [x] New Cloudformation template written for EFS at the environment level
• [x] New Cloudformation template written for managed EFS (access points at service level, IAM policy changes)
• [x] Update the Env Controller lambda to handle as-needed EFS creation
• [x] Data structures and methods written to validate external EFS
• [x] Data structures and methods written to validate managed EFS
• [x] Define new manifest structures and expose to all workloads
• [ ] Implement task run --volume flag

We'd like to support the following two use cases:

1. Bring one or more existing file systems and have them work with Copilot services or environments. This would allow customers to specify the EFSVolumeConfiguration struct from the ECS task definition as fields in the manifest, and enable full access point, encryption, and IAM auth capability.
2. Create empty file systems automatically to enable FS persistence for CMS applications. This would allow customers to run Wordpress or Drupal without extracurricular configuration with Copilot.

1. Expose Task Definition EFS Levers

We’ll allow the following optional key in the manifest for frontend services, backend services, and scheduled jobs. This condenses the MountPoint and Volume concepts from the task definition into a single struct.

volumes:
  - name: my-efs-volume        \
    path: /container/path      | These keys map to MountPoint.
    read_only: true            /
     efs:
       id: fs-12345             \
       auth:                    |
         access_point_id: ""    | These keys map to EfsVolumeConfiguration.
         iam: true              | `name` is shared.
       root_dir: "/"            |
       transit_encryption: true /

The minimal configuration for a volume will be

volumes:
  - name: my-efs-volume
    path: /container/path
    efs: fs-12345

Specifying simply the filesystem ID will assume the defaults above.

The volumes key can be overridden by environment, like all other aspects of the manifest. This helps us get around the inherent limitations of EFS. Those limits include the fact that a file system can only be effectively used in one environment due to the 1-mount-target-per-AZ and 1-VPC-per-file-system limits.

In order to use an external file system with Copilot, customers will need to manually create mount targets in the subnets of the environment they wish to use an external file system with. For applications with a single environment, this is simple:

1. Using the new Copilot NFS security group, create mount targets in the public subnets of your Copilot environment. This security group allows TCP/2049 ingress from the Environment Security Group.
2. Configure the environment overrides section of any services you wish to use with this file system in that environment:

environments:
  production:
    volumes:
      - name: my-efs-volume
        path: /container/path
        efs: fs-12345

You’ll need to either allow your service to function without the network file system, or specify one file system per environment.

2. Allow the creation of empty filesystems

Some applications, like Wordpress or other CMS frameworks, only require that they be backed by shared storage so that plugins, themes, and uploads are persisted between container restarts. For applications of this nature, which need only a shared space of persistent storage, we’ll add a “magic” key to the above definition.

Specifying copilot in the filesystem_id key, or omitting this key entirely, will trigger the creation of an EFS file system at the environment level. Access will be granted to individual services to automatically created subdirectories using IAM policies and an Access Point. Therefore, it should “just work” to specify the following configuration and deploy it to your environments.

name: wp
type: Load Balanced Web Service

... // All current manifest configuration

volumes:
  - name: wp-content                 // Omitting the efs parameter CREATES a FS
    read_only: false
    path: /var/www/html/wp-content

environments:
  production:
    volumes:
      - name: wp-content
        read_only: false
        efs: 
          filesystem_id: fs-12345
          authorization_config:
            iam: true
          root_directory: /wp-data-on-efs

This will do the following:

1. Create a Copilot-managed filesystem in any environments other than production
2. Create an access point for my wp service in its own subdirectory of the new filesystem(s).
3. In production, attempt to mount fs-12345 as a volume with root directory on the FS in the /wp-data-on-efs and container mount point at /var/www/html/wp-content.

More advanced use cases, such as the ability to add multiple volumes from different directories, can be accomplished by first deploying a service with a volume specified, then “hydrating” that volume through the use of one-off tasks or a codebuild project, and specifying additional volumes with different root directory configuration.

Even more advanced use cases can be handled by managing the file system external to Copilot.

3. Allow volumes to be mounted in one-off tasks.

We’ll add a flag to copilot task run to accept all of the above parameters as either a key1=value1,key2=value2 list or a yaml file. This will enable one-off tasks to mount existing file systems, including those created by Copilot, and populate them with data.

aws iam create-role --role-name my-task-role 
POLICY_ARN=$(aws iam create-policy \
  --policy-name EFSAccessForCopilotTasks \
  --policy-document file://efs-policy.json \
  --query 'Policy.Arn')
aws iam attach-role-policy --role-name my-task-role --policy-arn $POLICY_ARN

copilot task run --dockerfile Dockerfile_hydrate \
  --app myapp \
  --env prod \
  --task-role my-task-role \
  --volume file://volumedefinition.yml
  --command "/bin/sh setup-efs.sh"

//volumedefinition.yml
name: my-efs-volume
path: /container/path
read_only: false            
efs:
  filesystem_id: fs-12345  
  authorization_config:    
    iam: true              
  root_directory: "/"      
  transit_encryption: true

[bvtujo]

Related #1166

[revmischa]

Regarding 2. -
Do you think you're going to eventually recreate the ecs-cli ecs-params.yml file? And docker-compose config?

[bvtujo]

@revmischa we always try for a balance of opinionated and customizable, so I suspect when we start adding more fields that can affect the task definition they'll be driven more by the high level functionality our customers need than a one to one correspondence like ecs-params.yml. I think the Docker Compose CLI is doing exciting things along that axis though!

[kohidave]

Howdy ya'll - that's a great question. I think the way we're thinking about approaching this is by allow folks to specify task-definition overrides directly in the manifest. We're tracking this issue here: https://github.com/aws/copilot-cli/issues/948

I think that'll open up a lot of advanced features that folks want.

[bvtujo]

Hi everyone, I've updated this issue to be a much more comprehensive summary of the direction we're going to take for EFS support.

It would REALLY help us if you thumbs-upped this issue if you could tell us what kind of use cases you had in mind for EFS and Copilot.

Do you need the ability to use a sub directory of an existing FS for ML purposes?
Do you need shared persistence for wordpress or drupal?
Do you need a file system for Jenkins? For a database container that you don't want to offload to Aurora or DDB?

We're really excited to learn exactly what our customers want and how we can best tailor our approach in the future to make it work for you!

[revmischa]

My use case is I am deploying an application that I did not write, that is not cloud-native. If I wrote it, it would use Lambda and S3 and I would have no need for copilot in the first place. The application in question is Lemmy which makes use of pict-rs. Pict-rs uses the filesystem as a database and storage for uploaded images.

[aaronwwsuedu]

Current case is to deploy naive implementation of mailman3 (I know, not a great use case)

• 1 RDS instance
• 3-4 containers (core, web, proxy, and possibly MTA)
• depending on implementation, mta, core, and web require persistent storage, proxy could be built without external requirement

[justin8]

Is there any ETA for this? I see it was marked as In progress and in to the sprint board 21 days ago, but I'm not sure how long your sprints are?

[bvtujo]

@justin8 We're waiting on security review but we aim to ship milestone 1 (externally created EFS file systems) next week!

[efekarakus]

Hi @justin8, the first milestone is now released in v1.3.0! https://github.com/aws/copilot-cli/releases/tag/v1.3.0

[bvtujo]

The current thinking on this is for the following UX for managed EFS:

storage:
  volumes:
    managedVolume:
      efs: true
      path: /etc/mount1

Custom creation info for the managed filesystem can be specified by using the uid and gid keys under the efs struct. This is necessary because when Copilot creates an access point for the service, Cloudformation requires that we pass in creation info for the directory which the access point will refer to, including a POSIX UID and GID.

Under the hood, Copilot will use the name of the service or job as the directory name, guaranteeing uniqueness. We also generate a semi-random UID and GID to ensure that unauthorized users can't modify files between services. This is overridable by specifying uid and gid in the efs map.

storage:
  volumes:
    managedVolume:
      efs:
        uid: 1001
        gid: 10000
      path: /etc/mount1

Note that the uid and gid keys are mutually exclusive with any other custom EFS config, like auth, id, and root_directory.

Custom EFS volumes can still be specified with the syntax above:

storage:
  volumes:
    customVolume:
      efs:
        id: fs-1234567
        auth:
          iam: true
          access_point_id: fsap-12345678
      path: /etc/mount1

[bvtujo]

As of Copilot v1.6.0, managed EFS support is now available! You can now rely on Copilot to create and manage EFS and service-specific Access Points for you, instead of fiddling with security groups and mount targets in your VPC.