Containers-roadmap: [ECS] [secrets]: support parameter name from AWS Secrets Manager

[ramilexe]

Tell us about your request
I want to be able to inject Secret Key item from Secrets Manager to ECS container_definitions.secrets

Which service(s) is this request for?
ECS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
In task definition I have the following configuration:

"secrets": [
  {
    "valueFrom": "arn:aws:secretsmanager:us-east-1:account:secret:SECRET_NAME",
    "name": "RABBITMQ_DEFAULT_USER"
  }
]

but it gets injected as whole json object with other parameters:

Is it possible to get only one value from the secret?
Thank you

[nsbialek]

This came up in a recent conversation as if the feature already exists. If someone has documentation for how to implement this please drop a note here. It would be much appreciated.

[kadrach]

Ran into this issue earlier this week with RDS credentials generated by SecretsManager. The ECS ContainerDefinition.Secrets only support an Arn for ValueFrom.

CloudFormation offers support for "dynamic references" for SecretsManager of the form {{resolve:secretsmanager:secret-id:secret-string:json-key:version-stage:version-id}} (note json-key).

The downside of this approach is that it bakes the secret into the stack on create/update operations, meaning a rotation of the secret would not be picked up by new tasks until the stack is updated.

[hspak]

I would love this functionality so we can have 1 secret in secrets manager per ECS task instead of 1 per every secret a task requires.

[ramilexe]

Any updates on that @coultn ? Is it on roadmap?

[galindro]

This is something really nice because it would avoid unnecessary costs.

For example:

Imagine that, to access an API, we should inform username and password. We could simply store them together in one secret:

Secret name: prod/api/trackify
Secret Value:

{
  "username": "myusername",
  "password": "mypassword"
}

With the current limitation, we can store each json property on its own secret, incurring in more costs:

Secret name: prod/api/trackify/username
Secret Value: _myusername_

Secret name: prod/api/trackify/password
Secret Value: _mypassword_

[coultn]

Any updates on that @coultn ? Is it on roadmap?

Apologies for the delay in responding - we are aware of the request and are evaluating the priority. Additional thumbs-up on the issue are always welcome and we do pay attention to that when prioritizing.

[stephen-164]

Also note that this requirement (one plaintext value per secret) is incompatible with the key rotation mechanisms and secret target attachments (eg RDS attachment, which requires a JSON structure to add the host/engine/port values). This makes the implementation particularly confusing.

[jritsema]

I agree that this would be a great option to have in ECS/Fargate. Maybe a new setting like injectSecretProperties that would inject each json property key/value as an envvar.

In the mean time, we created a docker shim that can do this.
https://github.com/turnerlabs/sm_entrypoint

We also use parameter store to achieve this but would prefer to standardize on Secrets Manager.

[cervantek]

Heads up that CodeBuild team just released exactly this with the following format. The ECS team should be talking to the CodeBuild team about these type of things, you guys are solving a lot of the same problems.

secret-id:json-key:version-stage:version-id

https://aws.amazon.com/about-aws/whats-new/2019/11/aws-codebuild-adds-support-for-aws-secrets-manager/

[rich4j]

Assuming this release addressed ask here? Open while AWS add support to CloudFormation?

https://aws.amazon.com/about-aws/whats-new/2020/02/amazon-ecs-now-supports-aws-secrets-manager-version-and-json-keys/

[coultn]

@rich4j This is supported in CloudFormation now. We will keep this issue open while we are adding support for Fargate tasks in ECS.

[rich4j]

@rich4j This is supported in CloudFormation now. We will keep this issue open while we are adding support for Fargate tasks in ECS.

Hi there. Thanks for response. Does this mean that the feature is only available for ECS tasks using EC2 launch type? Don't think that comes across clearly in release and docs.

[coultn]

Correct, it is EC2 only for now, as stated here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html

[acaire]

@coultn any prospect of this JSON key functionality trickling through to IAM to allow fine grain write permissions on individual fields, which would allow updating a secret without having access to retrieve it?

[pandayanlu]

I am blocked as ContainerDefinition:Secret doesn't support json key for ECS Fargate, it would be great appreciate if the support for Fargate can be added ASAP.

[eberkund]

I am blocked as ContainerDefinition:Secret doesn't support json key for ECS Fargate, it would be great appreciate if the support for Fargate can be added ASAP.

I am using jq

along with

export DB_HOST=$(echo "$DB_SECRET" | jq .host)

to extract the secrets inside my container. Not ideal, but it works.

[ngamradt-turner]

@coultn Do we know when this is going to be available in Fargate?

[coultn]

@coultn Do we know when this is going to be available in Fargate?

We don't generally provide specific dates on this roadmap, however you can see that this issue is in the "We're Working On It" phase. From the readme:

Q: What do the roadmap categories mean?

Just shipped - obvious, right?
Coming soon - coming up. Think a couple of months out, give or take.
We're working on it - in progress, but further out. We might still be working through the implementation details, or scoping stuff out.
Researching - We're thinking about it. This might mean we're still designing, or thinking through how this might work. This is a great phase to send how you want to see something implemented! We'd love to see your usecase or design ideas here.

[ChristophEckerle]

@coultn

Hi Nick!
Till this feature is online, could you please provide a workaround for Farate Launch type?
It blocks the work for me by using Secret from Secret Manager by using GenerateSecretString :

DBMasterPassword:
Type: AWS::SecretsManager::Secret
Properties:
Description: Mysql root password generation
GenerateSecretString:
ExcludeCharacters: "! # $ % & ' ( ) * + , - . \ / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ "
SecretStringTemplate: '{"username": "root"}'
GenerateStringKey: 'password'
PasswordLength: 16
IncludeSpace: false
RequireEachIncludedType: True
ExcludePunctuation: false

And Injecting it somehow into the ContainerDefinition:

        - Name: DATABASE_PASSWORD
          ValueFrom: !Join ['', ['{{resolve:secretsmanager:' , "Fn::ImportValue" : {!Sub '${StackNameDatabase}:${DBMasterPassword}'}, ':SecretString:password}}' ]]

Is there a work around based on RDS, ECS/Fargate and CFN with or without key generation?

Thanks!

[srrengar]

hi @ChristophEckerle
Is it possible to run the container on EC2 so you can take advantage of the feature while we work on it for Fargate?

[ChristophEckerle]

Hi @srrengar, it is against our system architecture idea and would effect our setup. Is there no workaround available to pass DB secrets into a container definition on Fargate and in the same time to RDS?

In RDS/Aurora Serverless it is not possible to put any ARN for the secrets like in Fargate:
'arn:aws:secretsmanager:eu-central-1:485695650525:secret:dev/mysqlrootpassword-0x0xx'

In Container Definition it is not possible to use resolve, like for Aurora Serverless/RDS:
'MasterUserPassword: !Join ['', ['{{resolve:secretsmanager:', dev/mysqlrootpassword, :SecretString:password}}'
OR
'!Join ['', ['{{resolve:secretsmanager:' , "Fn::ImportValue" : {!Sub '${StackNameDatabase}:${DBMasterPassword}'}, ':SecretString:password}}' ]]'

Every workaround is welcome!

[rrijkse]

@ChristophEckerle This is my workaround

Pass in an ENV variable without the specific secret (from a taskdefintion.json file)

"secrets": [
        {
          "name": "DB_JSON",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:<account>:secret:<secret name>"
        }

and then parse it in an entrypoint script like start.sh

export DB_HOST=$(echo ${DB_JSON} | jq -r '.endpoint')
/bin/yourapp

The ".endpoint" would be whatever the secret's label is.

[dzmitry-kankalovich]

My case is that we did application services with ECS/EC2 and now we want to run some finite tasks on schedule with ECS/Fargate - and we've bumped into this problem. This is a small but crucial feature, I hope AWS prioritizes implementing it ASAP.

[ophintor]

I'm on the same boat. We just implemented secret rotation for RDS but now we need to change the way we get those secrets in the app, which is pretty painful.

[memark]

Any update on this? It would be really useful.

[StevePavlin]

+1, one monolithic secret in SSM injected as specific env vars seems to be the use case that makes sense for us coming off heroku to fargate.

[ophintor]

AWS Secret Manager with ECS (Fargate) ---> https://www.youtube.com/watch?v=DPXFknB1up0&t=35s&ab_channel=AWSLinuxWindows

The video is 1h long. Could you please point us to the relevant part of the video? Thanks.

[jwmonroe-outschool]

I believe would be possible to support JSON keys by reverting this PR. We were using an older version of cdk (1.36.1) and were able to set environment variables for secret keys JSON keys through the ECS agent like such:

secrets: {
    DB_CONNECTION_URI: ecs.Secret.fromSecretsManager(secretStore, "DB_CONNECTION_URI")
}

I believe that a team upstream of CDK (CF? ECS?) must have added support to fix this issue.

We tried upgrading to 1.62.0 and found that CDK was blocking deployments with "Cannot specify secret JSON field for a task using the FARGATE launch" despite the same Fargate task working right before the upgrade.

[memark]

Hi there, Simply Use a policy for Secret Manager in ---> Task Role and Task Execution Role (requires permissions for secret manager)
...

Step 2

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"arn of the secret created"
]
}
]
}

@quickbooks2018 Just to be clear, are you saying that adding this policy makes parameter names from secrets work again?

That kinda makes me wonder _why_ it was disabled in the first place (in the PR mentioned above). I assumed it was not safe, or not working correctly?

[ophintor]

Hi there, Simply Use a policy for Secret Manager in ---> Task Role and Task Execution Role (requires permissions for secret manager)
...

Step 2

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"arn of the secret created"
]
}
]
}

@quickbooks2018 Just to be clear, are you saying that adding this policy makes parameter names from secrets work again?

That kinda makes me wonder _why_ it was disabled in the first place (in the PR mentioned above). I assumed it was not safe, or not working correctly?

I don't think @quickbooks2018's answer relates to the original question. The whole point of this thread is to find a way to pass a value from a secret when the secret is in JSON format, by specifying a key. The example he describes works but it's simply for string secrets rather than secrets stored as part of a JSON structure.

[dzmitry-kankalovich]

@quickbooks2018 I am still not sure what you are suggesting here. If you are talking about programmatic access to secrets manager from within the application - there are neither doubts nor problems with that, but that's not the problem we discussing here.

Here we are talking about the absence of a declarative way to pass secret values as JSON property via ECS Task definition, e.g.

          Secrets:
            - Name: redshift_db_password
              ValueFrom: !Sub "arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:some-secret-with-json-value:some-json-property::"

UPDATE: I took some time and watched your video. You reference raw value from the secret manager, not the JSON property. Your suggestion is invalid, therefore. Curiously you yourself tell about this problem in your video and even reference this ticket, so not sure why you are confusing people here.

[ngamradt-turner]

Is there an update as to when this feature will be coming to Fargate?

[jonathan-atom]

Totally agree that this is a big gap and look forward to a solution... If you can live with Parameter Store, securely injecting single-values as secrets/env in the task definition works just fine.

[ngamradt-turner]

@jonathan-atom Yes, we are already leveraging workarounds, but we want to get rid of them. Since this works for classic ECS, it needs to get rolled out to Fargate for parity.

[dennisbrouwer91]

This would indeed really help!

[srrengar]

Hi everyone, thank you for your patience. This feature is now available in Fargate as of today.

https://aws.amazon.com/about-aws/whats-new/2020/11/aws-fargate-for-amazon-ecs-launches-features-focused-on-configuration-and-metrics/