Aws-cdk: [aws-secretsmanager] Unnecessary updates of SecretRotationApplication if no resources have been modified

[asterikx]

The SecretRotationApplication construct is updated even if no resources of the enclosing stack have been modified.

• Running cdk diff DbStack produces There were no differences.
• Running cdk deploy DbStack produces:
  0/3 | 10:48:21 | UPDATE_IN_PROGRESS | AWS::CloudFormation::Stack | Database/RotationSingleUser (DatabaseRotationSingleUser65F55654) 1/3 | 10:48:32 | UPDATE_COMPLETE | AWS::CloudFormation::Stack | Database/RotationSingleUser (DatabaseRotationSingleUser65F55654) 1/3 | 10:48:35 | UPDATE_COMPLETE_CLEA | AWS::CloudFormation::Stack | DbStack
• Running cdk deploy DbStack --verbose, I see the following lines:
  DbStack: parameters have changed DbStack: deploying... Attempting to create ChangeSet CDK-6225bb6f-6be9-4297-a89c-c6c49a19d2f8 to update stack DbStack DbStack: creating CloudFormation changeset...
  But I cannot see which parameters actually changed. (How can I see those?)

Reproduction Steps

Database stack:

const parameterGroup = new rds.ClusterParameterGroup(this, 'ClusterParameterGroup', {
  family: 'aurora5.6',
  parameters: {
    general_log: '1',
    slow_query_log: '1',
  },
});

const db = new AuroraServerless(this, 'Database', {
  engine: rds.DatabaseClusterEngine.AURORA,
  engineVersion: '5.6.10a',
  vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.ISOLATED },
  parameterGroup,
  defaultDbName: 'demo',
  scalingConfig: {
    // ...
  },
});

// Rotate secret every 30 days
db.addRotationSingleUser(cdk.Duration.days(30));

Custom AuroraServerless construct:

export class AuroraServerless extends cdk.Resource implements ec2.IConnectable, secretsmanager.ISecretAttachmentTarget {
  public readonly secret: rds.DatabaseSecret;
  private readonly secretRotationApplication: secretsmanager.SecretRotationApplication;

  // endpoints, connections, subnets, security group

  constructor(scope: cdk.Construct, id: string, props: AuroraServerlessProps) {
    super(scope, id);

    const { engine, engineVersion } = props;

    // Setup subnets and security group

    // DB secret
    const secret = new rds.DatabaseSecret(this, 'DbSecret', {
      username: 'root',
    });
    this.secret = secret;
    this.secretRotationApplication = engine.singleUserRotationApplication;

    // DB cluster
    const cluster = new rds.CfnDBCluster(this, 'DbCluster', {
      engine: engine.name,
      engineVersion,
      engineMode: 'serverless',
      masterUsername: secret.secretValueFromJson('username').toString(),
      masterUserPassword: secret.secretValueFromJson('password').toString(),
      // parameters, subnets, security group, roles, scaling config, ...
    });
    this.clusterIdentifier = cluster.ref;

    secret.attach(this);

    // Setup cluster endpoint, "connections", and clusterArn
  }


  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L542
  public addRotationSingleUser(automaticallyAfter?: cdk.Duration): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add single user rotation for a cluster without secret.');
    }

    const id = 'RotationSingleUser';
    const existing = this.node.tryFindChild(id);
    if (existing) {
      throw new Error('A single user rotation was already added to this cluster.');
    }

    return new secretsmanager.SecretRotation(this, id, {
      secret: this.secret,
      automaticallyAfter,
      application: this.secretRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }

  public asSecretAttachmentTarget(): secretsmanager.SecretAttachmentTargetProps {
    return {
      targetType: secretsmanager.AttachmentTargetType.RDS_DB_CLUSTER,
      targetId: this.clusterIdentifier,
    };
  }
}

Environment

• CLI Version : 1.47.0 (build c2b499a)
• Framework Version: 1.47.0
• Node.js Version: v12.16.1
• OS : macOS 10.15.5 (19F101)
• Language (Version): TypeScript (3.9.5)

---

This is :bug: Bug Report

[skinny85]

Hello @asterikx ,

I wonder whether the problem was a CDK version upgrade. In some cases, the hash of the assets that implement our Custom Resources change, and the result is a different parameter (for an example, see this PR: https://github.com/aws/aws-cdk/pull/8632 ).

Can you try if you can reproduce this problem when upgrading between CDK versions 1.45.0 and 1.46.0 (that's when the above PR was released, so if I'm right, the problem should manifest itself when changing between those 2 versions)?

Thanks,
Adam

[asterikx]

Hi Adam,

the problem occurs all the time (not only occur after upgrading between CDK versions).

When I deploy the stack multiple times in a row, the Database/RotationSingleUser resource is being updated each time, e.g. [(0) cdk synth ->] (1) cdk deploy DbStack -> (2) cdk deploy DbStack -> ... updates the resource in step 1, step 2, ...

[asterikx]

@skinny85

Below is the changeset created during cdk deploy in case that's of any help. But I cannot find what parameter changes cause cdk deploy to create a changeset in the first place.

Changeset

[
  {
    "resourceChange": {
      "logicalResourceId": "DatabaseDbSecretRotationScheduleD981E273",
      "action": "Modify",
      "physicalResourceId": "arn:aws:secretsmanager:eu-central-1:REDACTED:secret:DatabaseDbSecret1098DC6E-9NuNsZTRT7rt-S0gB3F",
      "resourceType": "AWS::SecretsManager::RotationSchedule",
      "replacement": "Conditional",
      "details": [
        {
          "target": {
            "name": "RotationLambdaARN",
            "requiresRecreation": "Conditionally",
            "attribute": "Properties"
          },
          "causingEntity": "DatabaseRotationSingleUser65F55654.Outputs.RotationLambdaARN",
          "evaluation": "Dynamic",
          "changeSource": "ResourceAttribute"
        }
      ],
      "scope": [
        "Properties"
      ]
    },
    "type": "Resource"
  },
  {
    "resourceChange": {
      "logicalResourceId": "DatabaseRotationSingleUser65F55654",
      "action": "Modify",
      "physicalResourceId": "arn:aws:cloudformation:eu-central-1:REDACTED:stack/DbStackPreProd-DatabaseRotationSingleUser65F55654-BQDCZORTLESZ/074473f0-a2ad-11ea-890f-02508c92de46",
      "resourceType": "AWS::CloudFormation::Stack",
      "replacement": "False",
      "details": [
        {
          "target": {
            "name": "TemplateURL",
            "requiresRecreation": "Never",
            "attribute": "Properties"
          },
          "causingEntity": null,
          "evaluation": "Static",
          "changeSource": "DirectModification"
        }
      ],
      "scope": [
        "Properties"
      ]
    },
    "type": "Resource"
  }
]

[skinny85]

@asterikx Can I ask you to try something for me? Can you let me know if you get the same behavior for the "standard" DatabaseCluster from RDS, when you use the addRotationSingleUser method?

Thanks,
Adam

[asterikx]

@skinny85 Just tried it. rds.DatabaseCluster with rds.DatabaseClusterEngine.AURORA engine works as expected (no unnecessary updates to SecretRotationApplication).

Have you any suspicion what might be causing the problem in my code? Is there a way to see which parameters have changed (more details on DbStack: Parameters have changed)?

[skinny85]

Nothing jumps out to me at a quick glance...

Can you maybe text-compare two templates that your code generates, and see if that turns up anything? (cdk synth once, copy the template you get in cdk.out somewhere else, cdk synth again, compare the new file in cdk.out to the one that was copied before)

[asterikx]

I tried that already, the produced templates are identical (used a diff-tool to compare them).

The updating behavior is a bit annoying, but not a deal-breaker. So I guess the best I can do right now is to wait until Aurora Serverless makes it to an L2 construct ;)

[jogold]

@asterikx Are your referencing SSM parameters?

[asterikx]

@jogold wow, you seem to have pinpointed the source of the problem.

I'm using SSM parameters (indirectly). I have a BastionLinuxHost in my stack, that connects to the database. The BastionLinuxHost construct is receiving its instance image config from SSM parameter store (see here and here).

const db = new AuroraServerless(/* ... */);

const bastion = new ec2.BastionHostLinux(this, 'BastionHost', {
  vpc,
});
bastion.role.addManagedPolicy(iam.ManagedPolicy.fromAwsManagedPolicyName(ManagedPolicies.AMAZON_SSM_MANAGED_INSTANCE_CORE));

db.connections.allowDefaultPortFrom(bastion);

I temporarily removed the bastion host, and then the unnecessary updates were gone.

(I did not show it in my first post because I didn't expect it to be causing the problem. Also, the bastion host is not being updated, only the secret rotation application is being updated.)

Do you have an idea how I can have both the bastion host and no unnecessary updates?

[jogold]

The secret rotation application creates a nested stack and CloudFormation will always attempt to update nested stacks when you start a deploy (even if they did not change).

To avoid unnecessary updates the CDK compares the templates, tags and parameters and if it can it skips deploy:
https://github.com/aws/aws-cdk/blob/254556d875f9a378ac98d5c3193306250068d3c9/packages/aws-cdk/lib/api/deploy-stack.ts#L202-L203

But if any of the parameters are SSM parameters, deploy is not skipped (which makes sense):
https://github.com/aws/aws-cdk/blob/254556d875f9a378ac98d5c3193306250068d3c9/packages/aws-cdk/lib/api/util/cloudformation.ts#L342-L345

Not sure what the right solution might be...

[asterikx]

@jogold thanks for the detailed explanations. Well, I guess, then I have to live with those updates ...

One solution I can think of is to rewrite the rotation functionality within the CDK, rather than using the rotation application(s) from the AWS Serverless Application Repository. But probably that shouldn't be done.

[njlynch]

Given the above, I think this is an "as designed" piece of functionality. Closing this out.