Containers-roadmap: ECS Secrets Management

[abby-fuller]

Enable customers to vend sensitive information from AWS Systems Manager Parameter Store and AWS Secrets Manager in the form of environment variables when a service on ECS or Fargate is started. The various components of this issue are:

• ECS support for AWS Systems Manager Parameter Store [[Live](https://aws.amazon.com/about-aws/whats-new/2018/11/aws-launches-secrets-support-for-amazon-elastic-container-servic/)!]
• ECS support for AWS Secrets Manager [Live!]
• Fargate support for AWS Systems Manager Parameter Store [[Live in Platform Version 1.3](https://aws.amazon.com/about-aws/whats-new/2018/12/aws-fargate-platform-version-1-3-adds-secrets-support/)]
• Fargate support for AWS Secrets Manager [[Live in Platform Version 1.3](https://aws.amazon.com/about-aws/whats-new/2018/12/aws-fargate-platform-version-1-3-adds-secrets-support/)]

[whithajess]

Will this only be AWS Secrets Manager and not AWS Systems Manager Parameter Store?

[Bensign]

@whithajess ECS supports AWS Systems Manager Parameter Store as of 11/15/18: https://aws.amazon.com/about-aws/whats-new/2018/11/aws-launches-secrets-support-for-amazon-elastic-container-servic/

[whithajess]

@Bensign Thanks for the info I guess I missed that - Unfortunately still lacking support for Fargate at the moment so will have to stick to: https://github.com/segmentio/chamber

[Bensign]

@whithajess We are actively working on Fargate support for both options: AWS Systems Manager Parameter Store & AWS Secrets Manager. I'll update back here on this thread as things go live.

[whithajess]

@Bensign Awesome - Thanks I really like this new public roadmap and hope it continues, has already given me a lot of useful info.

I follow a RSS feed of whats new in AWS but its pretty easy to miss relevant stuff given the amount of new things daily.

[maartenvanderhoef]

First of all. It's great this roadmap is shared, kudos for that!

The new Task definition parameter secrets is a good start for variables stored in SSM/KMS. But it's not a replacement for Chamber unfortunately. It would be great to have a parameter with which one can expose all variables of a certain path inside SSM as Environment Variables like chamber does.

Next to that SSM has been problematic when it comes to API limits. It would help if AWS would come clear regarding the API limits so that it's possible to use SSM more extensively without the risk of hitting API limits.

[garrardkitchen]

@maartenvanderhoef This is such a great idea...I would request for Secrets and Fargate too.

We've also been hit by SSM API throttling (via Lambda) :(

[Trane9991]

What about being able to mount secret into the container as the file, as Kubernetes does?

[tabern]

@Bensign Awesome - Thanks I really like this new public roadmap and hope it continues, has already given me a lot of useful info.

I follow a RSS feed of whats new in AWS but its pretty easy to miss relevant stuff given the amount of new things daily.

You can also check out https://aws.amazon.com/containers/new – we keep this regularly updated with the latest containers related release postings and blog links.

[Bensign]

Hi all,

A few updates here:

• Fargate Platform Version 1.3 now supports AWS Systems Manager Parameter Store: https://aws.amazon.com/about-aws/whats-new/2018/12/aws-fargate-platform-version-1-3-adds-secrets-support/
• ECS now supports direct AWS Secrets Manager integration. With the latest version of the agent you may now specify AWS Secrets Manager ARNs. This will be coming to Fargate in the not so distant future as well so hang tight for that.

Volume mounts for secrets rather than environment variables has been something we've discussed in the past. Thanks for opening the issue on that. I think I'll also separately create an issue for vending an entire path as that has popped up a few times as well.

Thanks everyone for the continued feedback on this!

[sj26]

vending an entire path

Yes please! This would be very useful for mounting in config files for community built images of non-cloud-native software without having to re-bake images or juggle sidecar containers.

[rlafferty]

Would this potentially change the # of api calls that are made to the SSM service at container start? Currently, if you have a custom startup script to get N number of secrets from SSM? We've found it fairly cost prohibitive if you start/stop containers very frequently and have 10+ secrets that get fetched/decrypted at container start time in terms of the SSM costs.

[Nr18]

Will this also contain using the secrets from the AWS::ECS::TaskDefinition resource in CloudFormation?

[cervantek]

Will this also contain using the secrets from the AWS::ECS::TaskDefinition resource in CloudFormation?

Just want to second this. You (Amazon) would know better than us, but I think many of your customers manage their TaskDefinitions in CFN. Also, looks like it is a separate issue to track this #97 .

[ronkorving]

Is it me, or does https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html not yet reflect the new Secrets property that is mentioned here?

[hluchej]

What about similar support in EKS? We have run into it here: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/754#issuecomment-442459674

[Zanndorin]

Hi,

Is there an ETA for

Fargate support for AWS Secrets Manager [In Progress]

Has been in Coming Soon for quite a while now :)

[Zanndorin]

If anyone missed it:
Beginning on April 2, 2019, any new Fargate task launched supports injecting sensitive data into your containers by storing your sensitive data in either AWS Secrets Manager secrets or AWS Systems Manager Parameter Store.

[sarojgharat]

But this still is not supported in cloudformation

On Thu, Apr 4, 2019, 13:39 Max Scholz [e-mail] wrote:

If anyone missed it:
Beginning on April 2, 2019, any new Fargate task launched supports
injecting sensitive data into your containers by storing your sensitive
data in either AWS Secrets Manager secrets or AWS Systems Manager Parameter
Store.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/8#issuecomment-479862354,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ab5nJA7NHJMdgASdbKnnuoH404n9iaXxks5vdeSFgaJpZM4Y4Ej4
.

[Zanndorin]

@sarojgharat We have started doing a workaround creating the container_definitions outside of CFN and then letting CFN referencing it when creating the task. Which works just fine for our needs..

[sarojgharat]

Thanks. Can you give me some hints on how to achieve this ?

On Thu, Apr 4, 2019, 13:43 Max Scholz [e-mail] wrote:

@sarojgharat https://github.com/sarojgharat We have started doing a
workaround creating the container_definitions outside of CFN and then
letting CFN referencing it when creating the task.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/containers-roadmap/issues/8#issuecomment-479863319,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ab5nJDPW_NTkx_rrrfDZVa2biX85CcA6ks5vdeVRgaJpZM4Y4Ej4
.

[Zanndorin]

Well

{
   "containerDefinitions": [ 
      { 
         "name": "Service",
         "image": "XYZ:1.0.37",
         "environment": [
          {
            "name": "ENV_NAME",
            "value": "prod"
          },
        "secrets": [
          {
            "name": "XYZ_PASSWORD",
            "valueFrom": "arn:aws:ssm:region:accountid:parameter/secretname"
          }],
        "logConfiguration": { 
          "logDriver": "awslogs",
            "options": { 
              "awslogs-group" : "",
              "awslogs-region": "",
              "awslogs-stream-prefix": "XYZService"
            }
          }
      },

Then we have a script that checks for changes on this file and pushes it. We then change the Revision number we want to deploy by hand actually (organizational process) but you could for example let your CFN always run the latest (by not specifying a number) but then I think CFN has problems finishing the creation of the stack...

[cornerman]

Is it somehow possible to specify the Version of a systems manager parameter?

I tried parameter NAME:VERSION but then I get an error:

> The Systems Manager parameter name specified for secret NAME is invalid. The parameter name can be up to 2048 characters and include the following letters and symbols: a-zA-Z0-9_.-,

[kiros-sc]

Hello I believe I might have an issues with this integration?
I'm setting up FireLens using Fargate and Http container as a test, sending the httpd log to datadog. When I tried to define the "apiKey" using a secret in secretsmanager the containers starts but the connection to datadog returns 400. When I pass the "api-key" literally, I see it on the conf of the container, and the connection works.

I check if I was able to get the value of the secret using the aws CLI, and yes, doe it brings more than just the value of the secret. I'm wondering if maybe there might be and issue with this when using the secret on ECS?

[fenxiong]

@kiros-sc can you send your task definition (the one that failed) to ecs-agent-external AT amazon.com so we can take a look? thanks.

[kiros-sc]

Ok, mail sent with the TaskDef on it.

[kiros-sc]

I hope this doesn't bother anyone but I found the solution to my issue. It was more related to how I was creating the "secret" in the secret manager.
At first I thought the issue was related to having a diferent "secret name" and "Name" in the "secretOptions" (task definition), yesterday that seem to be solving the issue, but, looking into two differents documents from amazon I change the way I created the "secret", instead of using the "key/value" pair, I created it as "plaintext", replacing any existing text with the "value" of my datadog apiKey, and when require to set the "secret Name" set the work apikey (just in case). After that the issue was gone.

This morning I used another secret which was created the same way, using "plain text" ()not the key/value, but as "secret Name" it was used the word "datadog", updated my task configuration with the new ARN and carefull here, setting the "name" in the "secretOptions" to "ApiKey" (just in case try different way to set that up >> apikey, Apikey, apiKey and ApiKey) and the issue was gone aswell.

So resuming all of this... I believe my issue was resolved when creating the secre using the "plaintext" setting.
Hope this helps someone.