Aws-cdk: (rds): Is it really that easy to destroy a database?

[claudioc]

The Question

This is the situation: we are developing an application which has CDK stack using the RDS module for Postgresql. The application is normally deployed using Github Actions and the database has (for now) only ONE user defined.

One day one developer deployed from their local environment (we should not do this, but it happened) and all of the sudden, the database in RDS disappeared.

During the deployment, CDK detected a change: that only change is in the credentials of the database; the value in the live environment differs (of course) from the value from the developer's laptop. This single change, for some reason, made CDK (or Cloudformation) take the decision to destroy the database and recreating it from scratch. This is part of the Cloudformation logs:

UPDATE_IN_PROGRESS: Requested update requires the creation of a new physical resource; hence creating one.

I know that there are options to avoid this potential disaster, but they should be enabled by default.

Our running theory is that CDK is not able to understand if/how to change credentials in a running db, hence it goes with the simplest solution: destroy/create. We tried more than once, and this is apparently the effect.

This is the relevant part of our ifrastructure:

    const instance = new rds.DatabaseInstance(this, 'SomeDatabaseInstance', {
      vpc,
      engine: rds.DatabaseInstanceEngine.postgres({
        version: rds.PostgresEngineVersion.VER_12_3,
      }),
      vpcPlacement: {
        subnetType: SubnetType.PUBLIC,
      },
      databaseName: config.database.name,
      credentials: {
        username: config.database.username,
        password: secret,
      },
      instanceType: config.database.instanceType,
    });

Environment

• CDK CLI Version: 1.73.0
• Module Version: "@aws-cdk/[email protected]"
• Node.js Version: 14.13.0
• OS: macOS Catalina
• Language (Version): TypeScript (4.0.3)

[skinny85]

Hey @claudioc ,

thanks for opening the issue. A couple of points:

1. The default retention policy of the Database Instance is Snapshot, which means the data is not lost, and it should be possible to create a new Database with it.
2. If you're worried about this happening in the future, there are a couple of things you can do:
   
   
   
   • Change the removalPolicy from the default SNAPSHOT to RETAIN.
   
   
   • Turn on deletion protection: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-rds.DatabaseInstance.html#deletionprotection
   
   
3. The weird behavior around the credentials might be solved if you used our recommended Credentials.fromGeneratedSecret static factory method.

Hope this clarifies things!

Thanks,
Adam

[rychkog]

@skinny85 Could you, please, give a bit of advice on how to pass credentials generated by Credentials.fromGeneratedSecret to ECS Fargate Construct, so that I can access the database instance from there? I mean the application running inside a container under ECS should have access to DB password and username.

What is the best way for doing this, or maybe it is described somewhere in docs already?

[jogold]

What is the best way for doing this, or maybe it is described somewhere in docs already?

Use the secret property of the DatabaseInstance as parameter in ecs.Secret.fromSecretsManager():

https://github.com/aws/aws-cdk/blob/v1.79.0/packages/@aws-cdk/aws-ecs/README.md#environment-variables

[rychkog]

@jogold Thanks for the answer. I've tried this and now getting this error:
Cannot specify secret JSON field for a task using the FARGATE launch type

I believe it is related to https://github.com/aws/aws-cdk/issues/7272

Though isn't that implemented already?

cdk version: 1.73.0

[jogold]

Though isn't that implemented already?

You need v1.75.0 or use this https://github.com/aws/aws-cdk/issues/11341#issuecomment-723553936

[rychkog]

@jogold Thanks for the tips!

@skinny85 The Credentials.fromGeneratedSecret method doesn't affect the behavior described in the issue.

The Credentials.fromGeneratedSecret accepts a username as a first parameter and whenever the value of username gets changed - the CloudFormation makes a decision to create a new database, while destroying the old one.

So termination protection definitely should be enabled by CDK users explicitly as you've described in the comment

[skinny85]

Of course. Changing the username of the master user requires the replacement of the database. But I don't see why you would ever need to change that value (including in the developer stacks).

[rychkog]

Of course. Changing the username of the master user requires the replacement of the database. But I don't see why you would ever need to change that value (including in the developer stacks).

@skinny85 Got your point, thanks for the help and support!

[skinny85]

I'm going to resolve this one. @claudioc please comment if you have any more questions!

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.