Aws-cdk: [rds] Database Secret broken after upgrade to 1.67

[dirknilius]

After upgrading a stack from CDK 1.60 to 1.67 not only the password was regenerated without changing the database password (see https://github.com/aws/aws-cdk/issues/10716), also the values for host, port, dbName are entirely missing. Our application cannot access the DB anymore as all connection data was pulled from Secret Manager.

It seems like the changed ExcludeCharacters value caused a regenerate of the secret.

Environment

• CLI Version :
• Framework Version: 1.67.0
• Node.js Version: 14.11
• OS : MacOS 10.15.7
• Language (Version): TypeScript 4.0.3

---

This is :bug: Bug Report

[jogold]

@skinny85 this comes from https://github.com/aws/aws-cdk/pull/10583 where a property of the secret (ExcludeCharacters) was changed. This leads to a secret update but the AWS::SecretsManager::SecretTargetAttachment doesn't run again (I think we discussed this already in another issue).

For stacks that have already been updated I think that the only option is to manually update the secret to add the missing dbname, engine, port and host fields?

For stacks that have not been updated, this can be avoided with something like:

const dbSecret = instance.node.tryFindChild('Secret') as rds.DatabaseSecret;
const cfnSecret = dbSecret.node.defaultChild as secretsmanager.CfnSecret;
cfnSecret.addPropertyOverride('GenerateSecretString.ExcludeCharacters', '"@/\\'); // use previous default

[skinny85]

So I think there are 2 things happening here:

1. The secret. Yes, like @jogold said, this is a consequence of adding more characters to the default exclusion set in #10583. I don't think you need that low-level of a hack to change it, you can simply go:

   new rds.DatabaseInstance(this, 'Db', {
     credentials: rds.Credentials.fromUsername('old-username', {
       excludeCharacters: '"@/\\', // old exclude set
     },
     // ...
   

2. The host, port, dbName disappearing - not sure that's related to the secret change. @dirknilius @Urokhtor can you show the code that created the database, and resulted in these values disappearing?

Apologies for the breakage - it was caused by the fact that the previous exclude character set was too permissive, and had a tendency to break with other services and in shell scripts. Now that RDS is stable, no such breakage will happen in the future.

[jogold]

For 1. indeed cleaner with credentials, overlooked this option
For 2. this is because those fields are populated by the AWS::SecretsManager::SecretTargetAttachment. The secret string template of the secret only specifies username and ask for password generation. If it's regenerated the fields populated by the attachment are gone.

[skinny85]

Sorry, I think I'm missing something still about point 2.

Where are those fields (host, port, dbName) actually stored?

[jogold]

Where are those fields (host, port, dbName) actually stored?

In the secret (JSON), this is what the AWS::SecretsManager::SecretTargetAttachment resource does. It exists only to create those fields because otherwise you would have a CF circular dependency between the secret and the DB.

[jogold]

This is the discussion we had in https://github.com/aws/aws-cdk/issues/7518 and the solution I think is still to have the logical id of the secret derived from the props.

Actually anyone changing the excludeCharacters of an already deployed stack now will have the secret loose the information added by the attachment.

[skinny85]

I wonder whether creating a new Secret will allow to re-connect to the database...?

const secret = new rds.DatabaseSecret(this, 'Secret', {
  username: 'old-username',
});
new rds.DatabaseInstance(this, 'Db', {
  credentials: rds.Credentials.fromSecret(secret),
  // ...

[fennb]

I wonder whether creating a new Secret will allow to re-connect to the database...?

@skinny85

I attempted this but it appears (surprisingly) to want to replace the entire DB cluster/instance in this case. Output from cdk diff:

[-] AWS::EC2::SecurityGroupIngress useast1sandboxeksclusterControlPlaneSecurityGroupfromuseast1sandboxuseast1sandboxeksclusterKubectlProviderSecurityGroup6D5584B0443B450864F destroy
[-] AWS::EC2::SecurityGroup useast1sandboxeksclusterKubectlProviderSecurityGroupF0CA4A5B destroy
[-] AWS::SecretsManager::Secret useast1sandboxauroraSecretE9A1B731 destroy
[-] AWS::SecretsManager::SecretTargetAttachment useast1sandboxauroraSecretAttachmentA47EBB78 destroy
[+] AWS::SecretsManager::Secret us-east-1-sandbox-aurora-secret useast1sandboxaurorasecretE96248F6
[+] AWS::SecretsManager::SecretTargetAttachment us-east-1-sandbox-aurora-secret/Attachment useast1sandboxaurorasecretAttachment358D1CBF
--snip--
[~] AWS::RDS::DBSubnetGroup us-east-1-sandbox-aurora/Subnets useast1sandboxauroraSubnetsD87B4BC8
 └─ [~] Metadata
     └─ [~] .aws:cdk:path:
         ├─ [-] us-east-1-sandbox/us-east-1-sandbox-aurora/Subnets
         └─ [+] us-east-1-sandbox/us-east-1-sandbox-aurora/Subnets/Default
[~] AWS::RDS::DBCluster us-east-1-sandbox-aurora useast1sandboxaurora09AD84FE replace
 ├─ [~] MasterUserPassword
 │   └─ [~] .Fn::Join:
 │       └─ @@ -3,7 +3,7 @@
 │          [ ] [
 │          [ ]   "{{resolve:secretsmanager:",
 │          [ ]   {
 │          [-]     "Ref": "useast1sandboxauroraSecretE9A1B731"
 │          [+]     "Ref": "useast1sandboxaurorasecretE96248F6"
 │          [ ]   },
 │          [ ]   ":SecretString:password::}}"
 │          [ ] ]
 └─ [~] MasterUsername (requires replacement)
     └─ [~] .Fn::Join:
         └─ @@ -3,7 +3,7 @@
            [ ] [
            [ ]   "{{resolve:secretsmanager:",
            [ ]   {
            [-]     "Ref": "useast1sandboxauroraSecretE9A1B731"
            [+]     "Ref": "useast1sandboxaurorasecretE96248F6"
            [ ]   },
            [ ]   ":SecretString:username::}}"
            [ ] ]
[~] AWS::RDS::DBInstance us-east-1-sandbox-aurora/Instance1 useast1sandboxauroraInstance111170B3A replace
 └─ [~] DBClusterIdentifier (requires replacement)
     └─ [~] .Ref:
         ├─ [-] useast1sandboxaurora09AD84FE
         └─ [+] useast1sandboxaurora09AD84FE (replaced)
[~] AWS::RDS::DBInstance us-east-1-sandbox-aurora/Instance2 useast1sandboxauroraInstance25DA34471 replace
 └─ [~] DBClusterIdentifier (requires replacement)
     └─ [~] .Ref:
         ├─ [-] useast1sandboxaurora09AD84FE
         └─ [+] useast1sandboxaurora09AD84FE (replaced)

Admittedly, this may due to the apparent change in subnet groups - it's unclear to me what changes are coming from where as this is also a CDK 1.60 -> 1.67 upgrade, which is what caused the problem in the first place.

[skinny85]

I think it's actually because of the username change - unfortunately, according to the docs, changing the username requires replacement:

MasterUsername
The name of the master user for the DB cluster.

Note
If you specify the SourceDBInstanceIdentifier or SnapshotIdentifier property, don't specify this property. The value is inherited from the source DB instance or snapshot.

Required: No

Type: String

Update requires: Replacement

It's because the reference to the Secret containing the username changes, even though the username itself does not...

 └─ [~] MasterUsername (requires replacement)
     └─ [~] .Fn::Join:
         └─ @@ -3,7 +3,7 @@
            [ ] [
            [ ]   "{{resolve:secretsmanager:",
            [ ]   {
            [-]     "Ref": "useast1sandboxauroraSecretE9A1B731"
            [+]     "Ref": "useast1sandboxaurorasecretE96248F6"
            [ ]   },
            [ ]   ":SecretString:username::}}"
            [ ] ]

[mmeylan]

Experiencing the same issue on our side. Is there a way to make sure the DB's master password is updated with the new generated secret ?

[skinny85]

I wonder whether re-creating the SecretAttachment might help here...? Something like this:

const secretAttachment = cluster.secret?.node.tryFindChild('Attachment') as cdk.CfnResource;
secretAttachment.overrideLogicalId('DifferentAttachmentId');

That _should_ re-populate the Database with the changed credentials...

Can somebody try the above, and let me know if it works?

[jogold]

Can somebody try the above, and let me know if it works?

Will try but I don't expect this to work as the secret attachment only populates connection fields in the secret's JSON.

[jogold]

I can confirm that overriding the logical ID of the secret attachment only repopulates the connection fields (dbname, engine, port, host) and doesn't update the credentials in the DB.

I think that solution should be to reference the username as a string in the masterUsername prop (not a dynamic reference to the username in the secret) and then have the logical id of the secret be a hash of its props/options (excluded characters, etc.). This way the secret will be replaced but the username remains the same so the instance/cluster doesn't get replaced when the secret is. UPDATE: tested this solution and it works.

This would be a breaking change as it would replace existing instances/clusters because we would be switching from masterUsername that is a dynamic reference to a string... one way to avoid this would be to introduce a new Credentials.fromFixedUsername() and deprecated the Credentials.fromUsername().

[skinny85]

@jogold but how does that help? You said changing the SecretAttachment does not re-populate the credentials of the database... so the Secret will change, yes, but the Database will still use the old Secret...?

[jogold]

No the secret will be replaced (by CF, new logical id), this updates the database (tested) and then the attachment also gets updated because it references a new secret. You end up with an updated database and all the fields in your secret.

The secret attachment never updates the database, it only acts on the content of the secret (populates fields so that it can be used in a rotation).

[skinny85]

Ok, thanks for the explanation.

I understand the appeal of changing it so that this doesn't happen in the future, but how can people unblock themselves with the current situation? Would deleting the database, and then restoring it from the snapshot be the only viable way?

[fennb]

Ok, thanks for the explanation.

I understand the appeal of changing it so that this doesn't happen in the future, but how can people unblock themselves with the current situation? Would deleting the database, and then restoring it from the snapshot be the only viable way?

This is the critical point for us right now - We're currently now stuck on an old version of CDK because we cannot upgrade without breaking multiple clusters simultaneously. We have a large number of CDK provisioned databases and fixing them all manually in some way is obviously less than ideal :)

I'm happy to perform a fairly involved workaround as a once-off if we can devise one/to get us to a better place but haven't had time to devise/test much in this direction yet. It'd be ideal if the workaround could be implemented "in-CDK" so to speak rather than having to fiddle around externally (ie: direct AWS API manipulation/etc).

[jdvornek]

I'd just like to add agreement with @fennb's point above. We're also stuck on an old version not wanting to update because we're not confident we wouldn't have access issues moving forward. I'd also like to add a request that the developers update documentation with their view of the way things are intended to work so that we can all decide if that intention meets our various needs or whether we might have to find a different way to accomplish things.

[skinny85]

@fennb @jdvornek Yep, that's exactly the plan. @jogold has graciously agreed to look into how to allow this upgrade, as he's very experienced with RDS.

[fennb]

@skinny85 @jogold very much appreciated - thanks for the responsiveness!

[jdvornek]

@skinny85 @jogold thank you!

[jogold]

This is how you can unblock yourself and upgrade (database is a rds.DatabaseInstance or rds.DatabaseCluster in the code below).

// Create a new DB secret, use the same username as the one currently used
const dbSecret = new rds.DatabaseSecret(this, 'Secret', { username: '<your master username here>' });

// Override MasterUserPassword property in AWS::RDS::DBInstance or AWS::RDS::DBCluster
const cfnDatabase = database.node.defaultChild as cdk.CfnResource;
cfnDatabase.addPropertyOverride('MasterUserPassword', dbSecret.secretValueFromJson('password').toString());

// Override SecretId property in the AWS::SecretsManager::SecretTargetAttachment
const attachment = database.secret?.node.defaultChild as cdk.CfnResource;
attachment.addPropertyOverride('SecretId', dbSecret.secretArn);

After this, if you need to reference your secret elsewhere in your code you can still use database.secret as it references the attachment and not the secret itself.

(your old secret remains in place and linked to your instance/cluster but for the username only: its password is now useless, not in sync with your database anymore)

[alukach]

@jogold Great tip, thanks so much!

The following worked for me:

1. Add the code mentioned by @jogold to my construct. Deploy.
2. Copy the newly generated secret's value and save it as the value of the old DB secret.
3. Remove the above-mentioned code from my construct. Deploy.

[dquist]

Adding another datapoint here - my team experienced the same behavior in our RDS cluster which is configured with automatic secret rotation. The secret value was updated when we deployed the stack, however the rotation Lambda was not triggered to update the RDS cluster.

From CloudTrail, I can see that the updated stack resulted in a PutSecretValue event, whereas previously we've seen a RotateSecret event when changing the rotation schedule for instance.

We thought a workaround might be to change the rotation schedule to force a rotation, but that ended up doing both! The password was rotated and then it was subsequently updated with a new value without a rotation.

Is there a workaround to always trigger a rotation when updating the value?