Aws-cdk: Cannot create Aurora Serverless cluster using RDS Construct

[patoncrispy]

There doesn't seem to be a way to create an Aurora Serverless database cluster using the RDS Construct. EngineMode is available in the underlying cfn but not in the Construct library.

Target Framework: netcoreapp2.1
Amazon.CDK: 0.12.0
Amazon.CDK.AWS.RDS: 0.12.0

[eladb]

Thanks for reporting. As a workaround, you should be able to use property overrides

[patoncrispy]

Ah sweet, thanks!

[ApocDev]

I set this up for our usage, it's ugly, but it may be of use to someone who needs a "quick and dirty" version of Serverless Aurora:

Please ignore the lack of comments, and possibly broken code, I removed some proprietary information. 🤕

import {Connections, ISecurityGroup, IVpc, Port, SecurityGroup, SubnetSelection} from "@aws-cdk/aws-ec2";
import {
    CfnDBCluster,
    CfnDBSubnetGroup,
    DatabaseSecret,
    Endpoint,
    SecretRotation,
    SecretRotationApplication,
    SecretRotationOptions
} from "@aws-cdk/aws-rds";
import {AttachmentTargetType, ISecretAttachmentTarget, SecretAttachmentTargetProps, SecretTargetAttachment} from "@aws-cdk/aws-secretsmanager";
import {Construct, RemovalPolicy, Token} from "@aws-cdk/core";

export interface ServerlessAuroraProps {
    readonly vpc: IVpc;
    readonly subnets: SubnetSelection;

    readonly clusterName: string;

    readonly masterUsername?: string;
    readonly securityGroup?: ISecurityGroup;
    readonly secretRotationApplication?: SecretRotationApplication;

    readonly maxCapacity: number;
}

export class ServerlessAurora extends Construct implements ISecretAttachmentTarget {
    public securityGroupId: string;
    public clusterIdentifier: string;
    public clusterEndpoint: Endpoint;
    public secret: SecretTargetAttachment;
    public connections: Connections;
    public vpc: IVpc;
    public vpcSubnets: SubnetSelection;
    public secretRotationApplication: SecretRotationApplication;
    public securityGroup: ISecurityGroup;

    constructor(scope: Construct, id: string, private props: ServerlessAuroraProps) {
        super(scope, id);

        this.vpc = props.vpc;
        this.vpcSubnets = props.subnets;
        this.secretRotationApplication = props.secretRotationApplication || SecretRotationApplication.MYSQL_ROTATION_SINGLE_USER;

        const secret = new DatabaseSecret(this, "MasterUserSecret", {
            username: props.masterUsername || "root",
        });

        const securityGroup = props.securityGroup || new SecurityGroup(this, "DatabaseSecurityGroup", {
            allowAllOutbound: true,
            description: `DB Cluster (${props.clusterName}) security group`,
            vpc: props.vpc
        });
        this.securityGroup = securityGroup;
        this.securityGroupId = securityGroup.securityGroupId;

        const cluster = new CfnDBCluster(this, "DatabaseCluster", {
            engine: "aurora",
            engineMode: "serverless",
            engineVersion: "5.6",

            dbClusterIdentifier: props.clusterName,

            masterUsername: secret.secretValueFromJson("username").toString(),
            masterUserPassword: secret.secretValueFromJson("password").toString(),

            dbSubnetGroupName: new CfnDBSubnetGroup(this, "db-subnet-group", {
                dbSubnetGroupDescription: `${props.clusterName} database cluster subnet group`,
                subnetIds: props.vpc.selectSubnets(props.subnets).subnetIds
            }).ref,

            vpcSecurityGroupIds: [securityGroup.securityGroupId],

            storageEncrypted: true,

            // Maximum here is 35 days
            backupRetentionPeriod: 35,

            scalingConfiguration: {
                autoPause: true,
                secondsUntilAutoPause: 300,
                minCapacity: 1,
                maxCapacity: props.maxCapacity
            }
        });
        cluster.applyRemovalPolicy(RemovalPolicy.DESTROY, {applyToUpdateReplacePolicy: true});

        this.clusterIdentifier = cluster.ref;
        // create a number token that represents the port of the cluster
        const portAttribute = Token.asNumber(cluster.attrEndpointPort);
        this.clusterEndpoint = new Endpoint(cluster.attrEndpointAddress, portAttribute);

        if (secret) {
            this.secret = secret.addTargetAttachment('AttachedSecret', {target: this});
        }
        const defaultPort = Port.tcp(this.clusterEndpoint.port);
        this.connections = new Connections({securityGroups: [securityGroup], defaultPort});

        // This is currently causing errors when deploying, since it uses a SAM template under the hood
        // Error: "Received malformed response from transform AWS::Serverless-2016-10-31"
        // It also adds in a warning from the CDK:
        // "This stack is using the deprecated `templateOptions.transform` property. Consider switching to `templateOptions.transforms`."
        // Which has notified *all* of our DevOps when we upgraded.
        // this.addRotationSingleUser("Rotation");
    }

    /**
     * Adds the single user rotation of the master password to this cluster.
     */
    public addRotationSingleUser(id: string, options?: SecretRotationOptions): SecretRotation {
        if (!this.secret) {
            throw new Error('Cannot add single user rotation for a cluster without secret.');
        }
        return new SecretRotation(this, id, {
            secret: this.secret,
            application: this.secretRotationApplication,
            vpc: this.vpc,
            vpcSubnets: this.vpcSubnets,
            target: this,
            automaticallyAfter: options ? options.automaticallyAfter : undefined,
        });
    }

    public asSecretAttachmentTarget(): SecretAttachmentTargetProps {
        return {
            targetId: this.clusterIdentifier,
            targetType: AttachmentTargetType.CLUSTER
        };
    }
}

[ryanvade]

Does #688 not implement this (it adds Aurora RDS)?
(Side note, @eladb your link is no longer valid, have a new one?)

[ApocDev]

Does #688 not implement this (it adds Aurora RDS)?
(Side note, @eladb your link is no longer valid, have a new one?)

No. The DatabaseCluster doesn't expose the engineMode and scalingConfiguration properties. It also has other property requirements that are only valid for non-serverless instances. (Instance type, size, etc) Those options are not available in the serverless mode.

[ryanvade]

I see, would it be preferable to add a "Serverless" Database Prop or instead change DatabaseClusterProps to support "Serverless" configurations?

[ApocDev]

I think creating a separate resource is probably the best option considering most of the options on the current cluster type don't quite apply. E.g only MySQL and Postgres are available, and you can't call lambdas from stored procedures or queries, etc.

Serverless also has the "Web API" feature that may be useful to turn on from some stacks. (Though, I'm not sure that it's currently supported in Cfn)

[seawatts]

Thanks for reporting. As a workaround, you should be able to use property overrides

This link doesn't seem to work. Can you give an example of how to do this?

[shivaramani]

Thanks for reporting. As a workaround, you should be able to use property overrides

This link doesn't seem to work. Can you give an example of how to do this?

I also dont see that link working. Can you please an example of property overrides?

[yargevad]

This doc shows how to do property overrides.

[apurvis]

any hopes for a release that allows this?

[skinny85]

@jogold is this a matter of adding an EngineMode enum with the values listed in https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html#cfn-rds-dbcluster-enginemode and an engineMode property to DatabaseClusterProps, or is this more complicated?

[jogold]

I think that it's more complicated than that and that it requires a separate class.

[aripalo]

@skinny85: It should also support EnableHttpEndpoint boolean property to allow enabling Data API.

Although this is not yet implemented in CfnDBCluster yet. See https://github.com/aws/aws-cdk/issues/5216 for more.

[jogold]

@skinny85 the DatabaseCluster was originally authored by @rix0rrr and @eladb I think. I only did the DatabaseInstance and added secret rotation to both. Maybe also check with them.

[bionicles]

I would use this

[asterikx]

Is there any progress on this?

It would be really helpful to have an L2 construct for Aurora Serverless DB cluster including Data API, secret rotation, and methods to easily grant read/write access Lambdas or AppSync.

[StevenAskwith]

You can use Escape Hatches to modify the L2 Aurora Server construct into a Serverless version:

from aws_cdk import (
    aws_ec2 as ec2,
    aws_rds as rds,
    core
)

class CdkRdsPipelineStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(
            self, "MyVpc",
            max_azs=2
        )

        aurora_sg = ec2.SecurityGroup(
            self, 'AU-SG',
            vpc=vpc,
            description="Allows MySQL connections",
        )

        # Aurora MySQL Cluster with Secrets Manager managed admin user
        aurora_cluster = rds.DatabaseCluster(
            self, "AU-CLUSTER-1",
            engine=rds.DatabaseClusterEngine.AURORA,
            engine_version="5.6.10a", # aws rds describe-db-engine-versions --engine aurora --query "DBEngineVersions[].EngineVersion" --region eu-west-1
            master_user=rds.Login(
                username='admin'
            ),
            default_database_name='MyDatabase',
            instance_props=rds.InstanceProps(
                instance_type=ec2.InstanceType('t3.small'),
                vpc=vpc,
                security_group=aurora_sg,
            ), 
            instances=1 # delete this 'Server' in an escape hatch below
        )

        # escape hatch https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html#cfn_layer_raw
        cfn_aurora_cluster = aurora_cluster.node.default_child
        cfn_aurora_cluster.add_override("Properties.EngineMode", "serverless")
        cfn_aurora_cluster.add_override("Properties.EnableHttpEndpoint",True) # Enable Data API
        cfn_aurora_cluster.add_override("Properties.ScalingConfiguration", { 
            'AutoPause': True, 
            'MaxCapacity': 4, 
            'MinCapacity': 1, 
            'SecondsUntilAutoPause': 600
        }) 
        aurora_cluster.node.try_remove_child('Instance1') # Remove 'Server' instance that isn't required for serverless Aurora

        # Add Secrets Manager Password rotation
        aurora_cluster.add_rotation_single_user()

[ericzbeard]

See https://github.com/aws/aws-cdk/pull/8686. That should be merged before making changes on this issue.

[mikestopcontinues]

I built on some of the work by @ApocDev and @StevenAskwith, in case it helps anyone else here:

  const vpc = new Vpc(this, 'Vpc', {maxAzs: 2});

  const securityGroup = new SecurityGroup(this, 'AuroraSecurityGroup', {
    vpc,
    allowAllOutbound: true,
  });

  const secret = new DatabaseSecret(this, 'AuroraSecret', {
    username: 'root',
  });

  const cluster = new CfnDBCluster(this, 'AuroraCluster', {
    engine: 'aurora',
    engineVersion: '5.6.10a',
    engineMode: 'serverless',
    masterUsername: secret.secretValueFromJson('username').toString(),
    masterUserPassword: secret.secretValueFromJson('password').toString(),
    vpcSecurityGroupIds: [securityGroup.securityGroupId],
    deletionProtection: false,
    enableHttpEndpoint: true,
    storageEncrypted: true,
    backupRetentionPeriod: 7,

    dbSubnetGroupName: new CfnDBSubnetGroup(this, 'AuroraSubnetGroup', {
      dbSubnetGroupDescription: 'AuroraSubnetGroup',
      subnetIds: vpc.selectSubnets({subnetType: SubnetType.PRIVATE}).subnetIds,
    }).ref,

    scalingConfiguration: {
      autoPause: true,
      minCapacity: 1,
      maxCapacity: 16,
      secondsUntilAutoPause: 300,
    },
  });

[asterikx]

Working construct with CDK 1.55.0

import * as cdk from '@aws-cdk/core';
import * as rds from '@aws-cdk/aws-rds';
import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as kms from '@aws-cdk/aws-kms';

export interface AuroraServerlessProps {
  readonly engine: rds.IClusterEngine;
  readonly clusterIdentifier?: string;
  readonly parameterGroup?: rds.ParameterGroup;
  readonly vpc: ec2.IVpc;
  readonly vpcSubnets: ec2.SubnetSelection;
  readonly securityGroups?: ec2.ISecurityGroup[];
  readonly masterUsername?: string;
  readonly defaultDatabaseName?: string;
  readonly backup?: rds.BackupProps;
  readonly preferredMaintenanceWindow?: string;
  readonly storageEncryptionKey?: kms.IKey;
  readonly deletionProtection?: boolean;
  readonly removalPolicy?: cdk.RemovalPolicy;
  readonly scalingConfig?: rds.CfnDBCluster.ScalingConfigurationProperty;
  readonly enableDataApi?: boolean;
}

// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html
// https://madabout.cloud/2019/09/01/aws-data-api-for-amazon-aurora-serverless/
// https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-rds/lib/cluster.ts
// https://github.com/ysku/aurora-serverless-example/blob/master/lib/database.ts
export class AuroraServerless extends cdk.Resource implements ec2.IConnectable, secretsmanager.ISecretAttachmentTarget {
  /**
   * Identifier of the cluster
   */
  public readonly clusterIdentifier: string;

  /**
   * ARN of the cluster
   */
  public readonly clusterArn: string;

  /**
   * The endpoint to use for read/write operations
   */
  public readonly clusterEndpoint: rds.Endpoint;

  /**
   * Access to the network connections
   */
  public readonly connections: ec2.Connections;

  /**
   * Security group identifier of this database
   */
  public readonly securityGroups: ec2.ISecurityGroup[];

  /**
   * The secret attached to this cluster
   */
  public readonly secret?: secretsmanager.ISecret;

  private readonly secretRotationApplication: secretsmanager.SecretRotationApplication;

  /**
   * The VPC where the DB subnet group is created.
   */
  private readonly vpc: ec2.IVpc;

  /**
   * The subnets used by the DB subnet group.
   */
  private readonly vpcSubnets: ec2.SubnetSelection;

  constructor(scope: cdk.Construct, id: string, props: AuroraServerlessProps) {
    super(scope, id);

    const {
      clusterIdentifier,
      engine,
      parameterGroup,
      storageEncryptionKey,
      backup,
      preferredMaintenanceWindow,
      deletionProtection,
      removalPolicy,
      defaultDatabaseName,
      vpc,
      vpcSubnets,
      scalingConfig,
      enableDataApi,
      masterUsername,
    } = props;

    this.vpc = vpc;
    this.vpcSubnets = vpcSubnets;

    // DB subnet group
    const { subnetIds } = vpc.selectSubnets(vpcSubnets);

    const dbSubnetGroup = new rds.CfnDBSubnetGroup(this, 'DbSubnetGroup', {
      dbSubnetGroupDescription: `CloudFormation managed DB subnet group.`,
      subnetIds,
    });

    // DB security group
    const securityGroups = props.securityGroups ?? [
      new ec2.SecurityGroup(this, 'DbSecurityGroup', {
        vpc: vpc,
        allowAllOutbound: false,
      }),
    ];
    this.securityGroups = securityGroups;

    // DB secret
    const secret = new rds.DatabaseSecret(this, 'DbSecret', {
      username: masterUsername || 'root',
    });
    this.secretRotationApplication = engine.singleUserRotationApplication;

    // bind the engine to the Cluster
    const clusterEngineBindConfig = engine.bindToCluster(this, {
      parameterGroup: parameterGroup,
    });
    const clusterParameterGroup = parameterGroup ?? clusterEngineBindConfig.parameterGroup;
    const clusterParameterGroupConfig = clusterParameterGroup?.bindToCluster({});

    // DB cluster
    const cluster = new rds.CfnDBCluster(this, 'DbCluster', {
      // basic
      engine: engine.engineType,
      engineVersion: engine.engineVersion?.fullVersion,
      engineMode: 'serverless',
      dbClusterIdentifier: clusterIdentifier,
      dbSubnetGroupName: dbSubnetGroup.ref,
      vpcSecurityGroupIds: securityGroups.map((sg) => sg.securityGroupId),
      // port: clusterEngineBindConfig.port, // Aurora Serverless always runs on default port
      dbClusterParameterGroupName: clusterParameterGroupConfig?.parameterGroupName,
      // associatedRoles: undefined,
      // admin
      masterUsername: secret.secretValueFromJson('username').toString(),
      masterUserPassword: secret.secretValueFromJson('password').toString(),
      backupRetentionPeriod: backup?.retention.toDays(),
      preferredBackupWindow: backup?.preferredWindow,
      preferredMaintenanceWindow: preferredMaintenanceWindow,
      databaseName: defaultDatabaseName,
      deletionProtection: deletionProtection,
      // encryption
      kmsKeyId: storageEncryptionKey?.keyArn,
      // storageEncrypted: storageEncryptionKey ? true : encrypted, // Aurora Serverless is always encrypted
      // serverless config
      enableHttpEndpoint: enableDataApi,
      scalingConfiguration: scalingConfig,
    });

    // Default deletion policy for AWS:RDS:DBCluster is SNAPSHOT
    // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
    if (removalPolicy) {
      cluster.applyRemovalPolicy(removalPolicy, {
        applyToUpdateReplacePolicy: true,
      });
    } else {
      // The CFN default doesn't cover UpdateReplacePolicy. Fix that here.
      cluster.cfnOptions.updateReplacePolicy = cdk.CfnDeletionPolicy.SNAPSHOT;
    }

    this.clusterIdentifier = cluster.ref;

    this.secret = secret.attach(this);

    this.clusterEndpoint = new rds.Endpoint(cluster.attrEndpointAddress, cdk.Token.asNumber(cluster.attrEndpointPort));

    const defaultPort = ec2.Port.tcp(this.clusterEndpoint.port);
    this.connections = new ec2.Connections({ securityGroups, defaultPort });

    const { region, account } = this.stack;
    this.clusterArn = `arn:aws:rds:${region}:${account}:cluster:${this.clusterIdentifier}`;
  }

  /**
   * Adds the single user rotation of the master password to this cluster.
   */
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L542
  public addRotationSingleUser(automaticallyAfter?: cdk.Duration): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add single user rotation for a cluster without secret.');
    }

    const id = 'RotationSingleUser';
    const existing = this.node.tryFindChild(id);
    if (existing) {
      throw new Error('A single user rotation was already added to this cluster.');
    }

    return new secretsmanager.SecretRotation(this, id, {
      secret: this.secret,
      automaticallyAfter,
      application: this.secretRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }

  public asSecretAttachmentTarget(): secretsmanager.SecretAttachmentTargetProps {
    return {
      targetType: secretsmanager.AttachmentTargetType.RDS_DB_CLUSTER,
      targetId: this.clusterIdentifier,
    };
  }

  /*
  // https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L566
  public addRotationMultiUser(id: string, options: rds.RotationMultiUserOptions): secretsmanager.SecretRotation {
    if (!this.secret) {
      throw new Error('Cannot add multi user rotation for a cluster without secret.');
    }
    return new secretsmanager.SecretRotation(this, id, {
      secret: options.secret,
      masterSecret: this.secret,
      automaticallyAfter: options.automaticallyAfter,
      application: this.multiUserRotationApplication,
      vpc: this.vpc,
      vpcSubnets: this.vpcSubnets,
      target: this,
    });
  }
   */
}

[manishkpr]

this article saved my day 🙇 🙇 🙇

[Wingjam]

Thanks @asterikx for sharing your construct, saved a lot of time!