Copilot-cli: Share same S3 storage between services?

[renatogbp]

Is it possible to share the same S3 storage add-on between two services?

My use case: I have a back-end service that uploads/deletes images onto S3 bucket, and a front-end service that reads those images from same S3 bucket.

Any ideas how I can accomplish this using copilot-cli?

[bvtujo]

Hi @renatogbp! You can do something like this using copilot storage init, our built-in way of creating an S3 Bucket for a service, and some clever usage of the addons directory for your frontend service.

For starters, we'll create a bucket with storage init for your backend service. We'll give the FE service access to it later.

copilot storage init -t S3 -n ${BUCKET_NAME} -w ${BACKEND_SERVICE_NAME}
copilot svc deploy -n ${BACKEND_SERVICE_NAME}

This will create an S3 bucket associated with your Backend Service and grant it permissions to read, write, and delete objects in it.

Then, check out the files generated at copilot/${BACKEND_SERVICE_NAME}/addons. There should be a file called ${BUCKET_NAME}.yml. If you open this and go looking for the IAM policy, you can repurpose it for your frontend.

You'll need to grab your bucket's name, which you can do with aws s3 ls using the profile you've deployed to.
Create a new file called S3Access.yml in copilot/${FRONTEND_SERVICE_NAME}/addons, and put the following text in it, where you replace ${BUCKET_NAME} with the name of your new bucket.

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  S3AccessPolicy:
    Metadata:
      'aws:copilot:description': 'An IAM ManagedPolicy for your service to access the bucket'
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: !Sub
        - Grants Read access to the S3 bucket.
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: S3ObjectActions
            Effect: Allow
            Action:
              - s3:GetObject
            Resource: "arn:aws:s3:::${BUCKET_NAME}/*"
          - Sid: S3ListAction
            Effect: Allow
            Action: s3:ListBucket
            Resource: "arn:aws:s3:::${BUCKET_NAME}"
Outputs:
  bucketAccessPolicy:
    Description: "The IAM::ManagedPolicy to attach to the task role"
    Value: !Ref S3AccessPolicy

Then you can run

copilot svc deploy -n ${FRONTEND_SERVICE_NAME}

and you should be good to go, as the above policy will be attached to your frontend containers.

[thstart]

When executing "copilot init"
--type 'Load Balanced Web Service'
I specify:

This will create an S3 bucket associated with your Backend Service and grant it permissions to read, write, and delete objects in it.
It did not created an S3 bucket so I've to create it.
Then after copilot svc deploy ...
I get the message
" Bucket name should be between 3 and 63 characters long"
The bucket name is only 13 characters long.

What is wrong?

[bvtujo]

Hi @thstart, could you tell us specifically what commands you ran? At a glance I can't tell what your inputs to copilot storage init were.

Also, if you used the above template for an add-on, you'll have to replace ${BUCKET_NAME} with your actual bucket name.

Thanks so much!

[thstart]

1

copilot init --app <myservice> --name <myservice> --type 'Load Balanced Web Service' --dockerfile './Dockerfile' --port 80 --deploy
That command finished successfully and the service is working.

Now I want to attach a S3 bucket.

2

copilot storage init -t S3 -n <myservicebucket> -w <myservice>
✔ Wrote CloudFormation template for S3 Bucket <myservicebucket> at 
copilot/ <myservice> /addons/ <myservicebucket> .yml

Recommended follow-up actions:

• Update 's code to leverage the injected environment variable myservicebucket_NAME
• Run copilot deploy --name <myservice> to deploy your storage resources.
  ran copilot deploy --name <myservice>.

The following resource(s) failed to create: [<myservicebucket>]. Rollback

...

• An Amazon S3 bucket to store and retrieve objects for
  Bucket name should be between 3 and 63 characters long

My bucket name is 13 character long so this should not be the problem.
Do I have to give some special permissions to copilot? It cannot create the S3 bucket.

[efekarakus]

Hi @thstart ! This is very strange, I don't think it's related to permissions. Do you see the bucket name written in the addons/<bucket>.yml file?

[thstart]

Hi @thstart ! This is very strange, I don't think it's related to permissions. Do you see the bucket name written in the addons/<bucket>.yml file?
yes, I changed the name to MyBucket, but here is the yaml file generated from copilot :

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  MyBucket:
    Metadata:
      'aws:copilot:description': 'An Amazon S3 bucket to store and retrieve objects for MyBucket'
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      AccessControl: Private
      BucketEncryption:
        ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: AES256
      BucketName: !Sub '${App}-${Env}-${Name}-MyBucket'
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true

  MyBucketBucketPolicy:
    Metadata:
      'aws:copilot:description': 'A bucket policy to deny unencrypted access to the bucket and its contents'
    Type: AWS::S3::BucketPolicy
    DeletionPolicy: Retain
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: ForceHTTPS
            Effect: Deny
            Principal: '*'
            Action: 's3:*'
            Resource: 
              - !Sub ${ MyBucket.Arn}/*
              - !Sub ${ MyBucket.Arn}
            Condition: 
              Bool:
                "aws:SecureTransport": false
      Bucket: !Ref MyBucket

  MyBucketAccessPolicy:
    Metadata:
      'aws:copilot:description': 'An IAM ManagedPolicy for your service to access the MyBucket bucket'
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: !Sub
        - Grants CRUD access to the S3 bucket ${Bucket}
        - { Bucket: !Ref MyBucket }
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: S3ObjectActions
            Effect: Allow
            Action:
              - s3:GetObject
              - s3:PutObject
              - s3:PutObjectACL
              - s3:PutObjectTagging
              - s3:DeleteObject
              - s3:RestoreObject
            Resource: !Sub ${ MyBucket.Arn}/*
          - Sid: S3ListAction
            Effect: Allow
            Action: s3:ListBucket
            Resource: !Sub ${ MyBucket.Arn}

Outputs:
  MyBucketName:
    Description: "The name of a user-defined bucket."
    Value: !Ref MyBucket
  MyBucketAccessPolicy:
    Description: "The IAM::ManagedPolicy to attach to the task role"
    Value: !Ref MyBucketAccessPolicy

[bvtujo]

Hi @thstart, does a bucket with that name already exist in your account? That could be a reason for the failure. Can you check the cloudformation outputs for the addons stack?

Currently, the deletion policy is set by default to Retain for S3 buckets but it looks like in our next release we'll remove that, so buckets will be freely torn down on service deletion.

[thstart]

Yes, that was the cause - I investigated after many experiments ;)

Now I have the next task - how to transfer the files from S3 to EFS?
Are there an automatic script to handle all permissions, etc?