Copilot-cli: Application-level resources?

[aviflax]

Hi, very cool tool!

• I need to create application-level resources (add-ons) that will be used by multiple services.
  
  
  
  • For example, S3 buckets.
  
  
• I don’t see any indication of any support for this in the docs.
• I suppose, conceptually, I could create a “virtual” service named “shared” and associate the add-ons with that service… but I can think of a significant problem with this approach: it’d create an ECS service that I don’t actually need.

So:

• In the short term, with the current version of the tool (0.3.0) is there some workaround I might be able to try?
• In the medium term, any chance support for application-level add-ons could be added?
  
  
  
  • I guess this would require changing the semantics of the deploy command… but that might be worth doing… I guess the semantics might be that you specify either a service to deploy, or to deploy add-ons
  
  

I hope that makes sense — thank you!

[efekarakus]

Heya @aviflax !

We don't have a great story at the moment with v0.3.0 for adding new shared AWS resources between services. The somewhat good news is that we're in the design phase to enable us to support this use case.
Our recommendation at the moment is exactly like you described to have each resource be fronted by a private Backend API service but I understand how that can increase the cost 😢

In the mean time, to mitigate the issue would the proposal below work for you?

1. Adding an IAM ManagedPolicy as an addon to the services that needs to access the shared resource:

Resources:
  AccessSharedS3BucketPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowForSharedResources
            Effect: Allow
            Action: '*' # can be scoped further
            Resource: 'arn:aws:s3:::*/*' 
            Condition: 
              StringEquals:
                'aws:ResourceTag/copilot-application': !Sub '${App}'
                'aws:ResourceTag/copilot-environment': !Sub '${Env}'

# This will make sure that the tasks in the service will have the permission above.
Outputs:
  AccessSharedS3BucketPolicyArn:
    Value: !Ref AccessSharedS3BucketPolicy

By default Copilot tags all resources created with copilot-application and copilot-environment keys, so this will grant permission for your service to all S3 resources created with Copilot.

1. Creating the S3 bucket in one of the services:

$ copilot storage init # Create the S3 bucket in one of the services.

[aviflax]

Thanks @efekarakus that’s very clear and comprehensive. I’ll report back on which way I go. Thank you!

[aviflax]

Hi @efekarakus sorry to bother you but I’m a little stuck on something. I’m trying to create a “dummy” service to host application-level shared resources (as addons) and I think I’ve got the service creation and addon creation more or less working. What’s unclear to me is whether or how I might be able to use an output from one of these addons in one of my other services.

For example, if I create an RDS instance in service A, and want to use the Endpoint.Address of that RDS instance in service B — is that possible?

Thanks!

[efekarakus]

It's no bother at all!

I think with the following proposal we can get it to work (I tested it locally), let me know how it turns out:

1. We'll leverage the AWS::SSM::Parameter resource to store anything that serviceA wants to expose to other services in its addons template.

# In copilot/serviceA/addons/template.yaml

EndpointAddressParam:
  Type: AWS::SSM::Parameter
  Properties:
    Name: !Sub ${App}-${Env}-${Name}-rdsEndpoint   # Give a path to the parameter
    Type: String
    Value: !GetAtt MyRDSResource.Endpoint.Address # Retrieve the RDS endpoint address

1. Now we need to refer to this SSM parameter from our second service's manifest:

# in copilot/serviceB/manifest.yml

# Although rdsEndpoint is not a secret this property allows us to read SSM parameters.
secrets:
  RDS_ENDPOINT: myApp-myEnv-serviceA-rdsEndpoint 

And that should be it! At this point serviceB will have the new environment variable RDS_ENDPOINT which was a property from serviceA.

Hope this helps :)

[aviflax]

Thanks @efekarakus I’ll try that and report back!

[aviflax]

Hi @efekarakus I’m trying this now and I have a few questions:

1. You suggested above adding this to serviceB/manifest.yml:

   secrets:
    RDS_ENDPOINT: myApp-myEnv-serviceA-rdsEndpoint 
   

   this makes sense, except I don’t think I can hard-code the environment name. I can hard-code the app name and service name, but I need to be able to use this manifest for various environments. Is there some way to make this dynamic?

2. Just curious: could we possibly use CloudFormation exports rather than SSM parameters to pass these values around? I’m just wondering if that might be a little more intuitive/natural, rather than (slightly) misusing the secrets feature.

Thanks again!!!!

[efekarakus]

Hi @aviflax !

1. One way I can think of is by leveraging the environments field to override secrets for each env:

secrets:
  RDS_ENDPOINT: myApp-test-serviceA-rdsEndpoint

environments:
  prod:
    secrets:
      RDS_ENDPOINT: myApp-prod-serviceA-rdsEndpoint  

1. There is a nice integration with CloudFormation Outputs only if serviceA needs to use them (the service that defines the addon). The outputs will automatically get injected as env vars to serviceA (same for secrets). Unfortunately, I can't think of a way of serviceB to leverage those exports.
   Our recommendation if it's affordable is for each DB to gets its own service. Other services can access the DB through an API. So for example, serviceB would make an HTTP request to serviceA to retrieve the data.

While it's not ideal, there is a small advantage of using SSM parameters over exports (https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-systems-manager-parameter/). When it's time to delete serviceA the deletion won't fail because serviceB depends on its exports.

[aviflax]

Great stuff, thanks again Efe!

FWIW, in my case in porting an existing legacy system (with many subsystems) over to Fargate, and the system design has included shared data resources for 10+ years. So I’m not gonna be able to change that anytime soon!

I'll report back on how this all works out.

[efekarakus]

No problem!

Also if you have your environments defined in different accounts & regions, i.e. test and prod are not overlapping, then you can remove the ${Env} from the SSM parameter. In that case life is easier and we can just write RDS_ENDPOINT: myApp-serviceA-rdsEndpoint.

We added ${Env} to disambiguate the params in case you have multiple environments in the same account and region.

[aviflax]

Also if you have your environments defined in different accounts & regions, i.e. test and prod are not overlapping, then you can remove the ${Env} from the SSM parameter. In that case life is easier and we can just write RDS_ENDPOINT: myApp-serviceA-rdsEndpoint.

Thanks for the tip!

We added ${Env} to disambiguate the params in case you have multiple environments in the same account and region.

Yeah, I’m going to leave it in, because that might happen in my case. E.g. multiple demo environments, dev, test, etc. It’d definitely be better to use a dedicated account for each environment, but hey, a belt-and-suspenders approach is generally a good idea, as long as neither the belt nor the suspenders are too expensive.

Thanks!

[aviflax]

Hi @efekarakus I’m still working on this… the pattern overall is mostly working, and working well, but I keep running into snags.

Current snag:

I’m deploying my service again (maybe the 20th time this week) into a fresh environment… but for some reason the service execution role doesn’t seem to have access to the param containing a DB password; I’m getting this in the “stopped reason” field for my service tasks:

Fetching secret data from SSM Parameter Store in us-west-2: AccessDeniedException: User: arn:aws:sts::453029712710:assumed-role/plotwatt-test-pienado-ExecutionRole-1P56QZ6LCLOP5/094510bf0b0d43f5ba82ea9c6ed7ae02 is not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:us-west-2:453029712710:parameter/aws/reference/secretsmanager/plotwatt/test/databases/main/pienado status code: 400, request id: f414df50-e113-414d-afb9-dad57c740465

My param definition, in the add-on to my _other_ service, the one hosting the shared resources, looks like this:

  PienadoPassword:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub '${App}/${Env}/databases/main/pienado'
      Description: The password for the user 'pienado' of the main DB cluster.
      GenerateSecretString:
        PasswordLength: 16
        # RDS and/or MySQL disallow some or all of these chars
        ExcludeCharacters: '"=@/\'

back in my failing service, I have this addon:

access-shared-resources.yaml

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.

Resources:
  AccessSharedResourcesPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowForSharedResources
            Effect: Allow
            Action: '*' # SHOULD be scoped further
            Resource:   # Could be scoped further, but the Condition below makes this maybe OK
              - 'arn:aws:s3:::*/*'
              - 'arn:aws:secretsmanager:::*/*'
              - 'arn:aws:ssm:::*/*'
              - 'arn:aws:rds:::*/*'
            Condition:
              StringEquals:
                'aws:ResourceTag/copilot-application': !Sub '${App}'
                'aws:ResourceTag/copilot-environment': !Sub '${Env}'

# This causes the tasks in *this* service to be associated with the above policy.
Outputs:
  AccessSharedResourcesPolicyArn:
    Value: !Ref AccessSharedResourcesPolicy

in my failing service’s manifest, I’ve got an entry in secrets that I was hoping would bring it in:

environments:
  test:
    secrets:
      DB_PASSWORD: /aws/reference/secretsmanager/plotwatt/test/databases/main/pienado

I tried referencing the secret without the prefix /aws/reference/secretsmanager/ but that didn’t work — which makes sense because the secret doesn’t exist in SSM, it’s in Secret Manager. So the prefix should be required, and hopefully work.

I thought maybe the secret didn’t have the required tags, but it does, which makes sense, since the the secret is defined as an add-on resource that Copilot passes along to CloudFormation, and as expected, Copilot automatically adds the proper tags.

I’m sure I’m missing something but I’m baffled and my eyes are glazing over so I need to take a break from this.

If you have any suggestions for what to look at, or something to try, I’d appreciate anything! Thanks!

[efekarakus]

Hi @aviflax !

I’m deploying my service again (maybe the 20th time this week) into a fresh environment

Oh no I'm sorry to hear that :( Do you have high-level feedback for us to make this process easier?

I tried referencing the secret without the prefix /aws/reference/secretsmanager/ but that didn’t work

I wonder if the reason why it doesn't work is because the SecretManager secret needs to be defined as an ARN (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html)?

Would you mind trying this out to see if this works:

test:
  secrets:
    DB_PASSWORD: 'arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf'

[aviflax]

Hi @efekarakus — I hope you had a good weekend!

Oh no I'm sorry to hear that :( Do you have high-level feedback for us to make this process easier?

Well, I’ve tried to file specific, focused issues for each problem I encounter. (10 so far.)

High-level, though, I’d say that it’s all about feedback from the tool as its doing its thing: more feedback, deeper feedback, and faster feedback.

For example:

• When deploying a service, which generally entails starting the service and waiting for it to be up and running, show the logs of the service tasks as they start up — just stream them right to the terminal.
• When deploying a service, show all the various CF resources being created/updated and their status/progress.
• I had a service that was stuck in "deploying" at the CLI, and creating/updating (don’t recall) in CF. When I took a closer look at the tasks of the service — after ~10–15 mins — I found that there had already been a dozen tasks that had been created, and failed; I just hadn’t been aware that that was happening. I had to go look. Once I did, there was a useful error message under the “reason” field of the task in the Console, and the task logs were helpful too. But I wasted time just sitting around waiting for something to happen, and getting frustrated, before I went to look. So it would have been better for the tool to have surfaced what was going on, proactively, even though that was all happening at a deeper level.

Would you mind trying this out to see if this works:

That worked! Thank you!

Not only did it work, but I was able to use the secret name as the last segment, rather than the secret… id? (The name + some random characters… I guess that’s the ID?) That’s good; it makes the config more portable.

However, the ARN still has the account number and the region hard-coded in the YAML file — I’d like to avoid this so that the config is more flexible and can be used with an arbitrary account and an arbitrary region.

Any ideas on how to accomplish that? Or should I file a new issue?

Thanks again!

[efekarakus]

(10 so far.)
😅

The 3 bullet points make a lot of sense to me! I copied them over to https://github.com/aws/copilot-cli/issues/1421 so that we can track all the UX related changes needed there.

To give you visibility into our current sprint: https://github.com/aws/copilot-cli/projects/1, we're currently working on a job command to create scheduled jobs as well as building the foundational work to support new shared environment level resources.

We expect after the next release to get back to ironing out these kinks around UX.

Any ideas on how to accomplish that? Or should I file a new issue?

This would be a new feature request for us. One thing that comes to my mind is to possibly provide some template functions to support this functionality:

secrets:
  RDS_PASSWD: {{.SecretsManagerARN "secretName"}}

[aviflax]

@efekarakus —

This would be a new feature request for us.

Created #1449.

At this point I’m pretty much done, and I was able to accomplish this pattern (shared resources) and make it work for my use case. Thanks so much for all the help!

[efekarakus]

No problem, thanks for all the feedback! It really helps us improve the tool and its direction :)