Aws-cdk: [DatabaseProxy] Model validation failed (#: required key [TargetGroupName] not found)

[tbrand]

I tried DatabaseProxy with Database Cluster (Aurora, Postgres) but it's failed as titled.

Reproduction Steps

    const databaseUser = 'test';
    const secret = new secretsmanager.Secret(this, 'secret', {
      generateSecretString: {
        generateStringKey: 'password',
        secretStringTemplate: `{"username": "${databaseUser}"}`,
        excludePunctuation: true
      }
    });

    const cluster = new rds.DatabaseCluster(this, 'Database', {
      engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
      engineVersion: '11.7',
      defaultDatabaseName: 'mydatabase',
      masterUser: {
        username: databaseUser,
        password: secret.secretValueFromJson('password')
      },
      instanceProps: {
        instanceType: ec2.InstanceType.of(
          ec2.InstanceClass.R5,
          ec2.InstanceSize.LARGE
        ),
        vpcSubnets: {
          subnetType: ec2.SubnetType.PRIVATE
        },
        vpc
      },
      parameterGroup: new rds.ClusterParameterGroup(this, 'ParameterGroup', {
        family: 'aurora-postgresql11',
        parameters: {
          application_name: 'test',
        },
      }),
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    const proxy = new rds.DatabaseProxy(this, 'DatabaseProxy', {
      dbProxyName: 'test-proxy',
      debugLogging: true,
      iamAuth: false,
      requireTLS: true,
      secret,
      vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE,
      },
      proxyTarget: rds.ProxyTarget.fromCluster(cluster),
    });

Error Log

Model validation failed (#: required key [TargetGroupName] not found)

Environment

• CLI Version : 1.49.1
• Framework Version: 1.49.1
• Node.js Version: v13.7.0
• OS : 10.14.6
• Language (Version): TypeScript (3.7.5)

Other

---

This is :bug: Bug Report

[tbrand]

I also tried below but still failed.

    cluster.addProxy('DatabaseProxy', {
      secret,
      vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE,
      },
    });

[nideveloper]

I have hit the same issue, tried DatabaseCluster and DatabaseInstance (MySQL, Postgres and Aurora MySQL) - no difference

[nideveloper]

Incase it helps whoever picks this up, when digging in earlier. The only place I can see TargetGroupName is set is inside CfnDBProxyTargetGroup as cdk.Token.asString(this.getAtt('TargetGroupName'));

[civilizeddev]

I'll take a look.

[tbrand]

Thanks!

[guydelta]

This was my original workaround:
Prior to rds.DatabaseProxy's introduction in 1.49 I had to create the DB Proxy with CfnDBProxy.
This route required me to set the proxy's target group as follow:

const rdsProxy = new CfnDBProxy(this, id+'proxy',options);
const targetGroup = new CfnDBProxyTargetGroup(this, id + 'dbproxytarget', {
    dbProxyName: rdsProxy.ref,
    connectionPoolConfigurationInfo: {
        connectionBorrowTimeout: 120,
        maxConnectionsPercent: 99,
    },
    dbInstanceIdentifiers: [props.rds.instanceIdentifier]
});

This caused the same required key [TargetGroupName] error. After many hours of struggling and diving into the cdk source I realized there is no place where TargetGroupName* seems to be assigned to the target group, expept has has been pointed out, in cdk.Token.asString(this.getAtt('TargetGroupName')) which we can't use, and no way to pass it into the constructor as a property. My workaround was to add the following line immediately after creating the target group:

targetGroup.addOverride('Properties.TargetGroupName', 'default');

TargetGroupName is not documented and assigning anything but default caused an enum error.

In cdk 1.49.0 the DatabaseProxy class was finally added, alongside the very helpful rdsInstance.addProxy(...) for which I am very thankful. However, I notice in the new proxy.ts file that was introduced calls the following

    new CfnDBProxyTargetGroup(this, 'ProxyTargetGroup', {
      dbProxyName: this.dbProxyName,
      dbInstanceIdentifiers,
      dbClusterIdentifiers: bindResult.dbClusters?.map((c) => c.clusterIdentifier),
      connectionPoolConfigurationInfo: toConnectionPoolConfigurationInfo(props),
    });

yet without again assigning the TargetGroupName. Hence the error still remaining.

You can thus solve the error by adding the following at the end of your code after const proxy

const proxy = ...

const targetGroup = proxy.node.findChild('ProxyTargetGroup') as CfnDBProxyTargetGroup;
targetGroup.addOverride('Properties.TargetGroupName', 'default');

I just tested these two lines on the new implementation of the cdk rds proxy class and it is working with my new db stack. I hope this helps.

[tbrand]

That's awesome! I will try the workaround.

[civilizeddev]

Sorry for inconvenience caused.

It seems something has suddenly changed in CloudFormation. (They implicitly set TargetGroupName before)

Fortunately, I can find the point early with your help.

I hope this would work.

[tbrand]

Another problem has happened. After I applying this patch, the deployment was failed with the message "Timed out waiting for target group to become available.".
But I confirmed that the database proxy had been created successfully on the AWS Management Console.

Does anyone hit the same problem?
Should I open another issue?

[civilizeddev]

To initialize DB Proxy Target Group,

DB Proxy to DB connection should be established.

Check if the security groups specified in the DB Proxy are allowed to access network that DB resides.

[civilizeddev]

In AWS Management Console:

I found that DB Proxy target was staying unavailable when I put a security group which is not allowed to access RDS instance.

In CDK or AWS CloudFormation, It will time out.

@tbrand Can you open new issue for this?

[tbrand]

@civilizeddev Thanks for the digging!

when I put a security group which is not allowed to access RDS instance.

Is that correct? I thought that the closed security group occurs the timeout. So opened security group would solve the issue. (I haven't try.)

[civilizeddev]

This diagram is more descriptive.

[nideveloper]

This is still a work in progress, it will be launched on cdkpatterns in the next couple of days but if you or anyone else following this wants some working code for a MySQL DB Instance, an RDS Proxy and a Lambda function which runs several queries integrated with an API Gateway HTTP API - https://github.com/nideveloper/serverless/blob/master/the-rds-proxy/typescript/lib/the-rds-proxy-stack.ts

[d2kx]

I stumbled into this, and I also had to add another property override, because doing DBCluster#addProxy added both DBClusterIdentifiers and DBInstanceIdentifiers to the ProxyTargetGroup, which is not allowed. So my overrides look like this:

const dbProxyTargetGroup = dbProxy.node.findChild(
  'ProxyTargetGroup'
) as CfnDBProxyTargetGroup;
dbProxyTargetGroup.addPropertyDeletionOverride('DBInstanceIdentifiers');
dbProxyTargetGroup.addPropertyOverride('TargetGroupName', 'default');

which solves the problems.

[civilizeddev]

@d2kx Thank you for reporting.

fixed by 84d0a4a

[tbrand]

Thanks guys. Finally I successfully created the proxy! :tada:
Here is the code.

     this.proxy = new rds.DatabaseProxy(this, 'DatabaseProxy', {
      dbProxyName: 'database-proxy',
      debugLogging: true,
      iamAuth: false,
      requireTLS: true,
      secret: this.secret,
      vpc: props.vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE,
      },
      proxyTarget: rds.ProxyTarget.fromCluster(this.cluster),
    });

    const targetGroup = this.proxy.node.findChild('ProxyTargetGroup') as rds.CfnDBProxyTargetGroup;
    targetGroup.addOverride('Properties.TargetGroupName', 'default');
    targetGroup.addOverride('Properties.DBClusterIdentifiers', [this.cluster.clusterIdentifier]);
    targetGroup.addOverride('Properties.DBInstanceIdentifiers', []);

[bodokaiser]

Any updates on this issue?

I am using version 1.44 of aws-cdk and deploying a cloudformation stack with DatabaseProxy timeouts because the target group remains unavailable. I placed the proxy and the lambda into the same SecurityGroup and even created an extra role to confirm that the proxy has access to the database secret.

It would be very helpful if there would be an official tutorial or example to this as this seems to be a common problem.

import { App, Stack, StackProps, Duration } from '@aws-cdk/core'
import { Role, ServicePrincipal, PolicyStatement } from '@aws-cdk/aws-iam'
import {
  Vpc,
  SecurityGroup,
  SubnetType,
  InstanceType,
  InstanceClass,
  InstanceSize,
  Port,
  ISecurityGroup,
} from '@aws-cdk/aws-ec2'
import {
  CfnDBProxyTargetGroup,
  DatabaseInstance,
  DatabaseInstanceEngine,
  DatabaseProxy,
  PostgresEngineVersion,
} from '@aws-cdk/aws-rds'
import { Secret } from '@aws-cdk/aws-secretsmanager'

export class PostgresStack extends Stack {
  public readonly secret: Secret
  public readonly proxy: DatabaseProxy
  public readonly instance: DatabaseInstance
  public readonly proxySecurityGroup: ISecurityGroup

  constructor(scope: App, id: string, props?: StackProps) {
    super(scope, id, props)

    const vpc = new Vpc(this, 'VPC', {
      natGateways: 0,
      subnetConfiguration: [{ name: 'postgres', subnetType: SubnetType.PUBLIC }],
    })

    const proxySecurityGroup = new SecurityGroup(this, 'RDS Proxy Clients', {
      vpc,
    })

    const databaseSecurityGroup = new SecurityGroup(this, 'RDS Database Clients', {
      vpc,
    })
    databaseSecurityGroup.addIngressRule(databaseSecurityGroup, Port.tcp(5432), 'allow db conection')
    databaseSecurityGroup.addIngressRule(proxySecurityGroup, Port.tcp(5432), 'allow proxy connection')

    this.secret = new Secret(this, 'MasterSecret', {
      generateSecretString: {
        excludePunctuation: true,
      },
    })

    this.instance = new DatabaseInstance(this, 'PostgresInstance', {
      engine: DatabaseInstanceEngine.postgres({ version: PostgresEngineVersion.VER_11 }),
      masterUsername: 'master',
      masterUserPassword: this.secret.secretValue,
      vpc,
      vpcPlacement: { subnetType: SubnetType.PUBLIC },
      instanceType: InstanceType.of(InstanceClass.T2, InstanceSize.MICRO),
      instanceIdentifier: 'CoopSaas-Postgres',
      securityGroups: [databaseSecurityGroup],
    })
    this.instance.connections.allowDefaultPortFromAnyIpv4()

    const proxyRole = new Role(this, 'RdsProxyRole', {
      assumedBy: new ServicePrincipal('rds.amazonaws.com'),
    })
    proxyRole.addToPolicy(
      new PolicyStatement({
        actions: [
          'secretsmanager:GetResourcePolicy',
          'secretsmanager:GetSecretValue',
          'secretsmanager:DescribeSecret',
          'secretsmanager:ListSecretVersionIds',
        ],
        resources: [this.secret.secretArn],
      }),
    )

    this.proxy = this.instance.addProxy('PostgresProxy', {
      secrets: [this.secret],
      vpc,
      debugLogging: true,
      iamAuth: true,
      role: proxyRole,
      securityGroups: [databaseSecurityGroup],
      borrowTimeout: Duration.seconds(30),
    })
    let targetGroup = this.proxy.node.children.find((child: any) => {
      return child instanceof CfnDBProxyTargetGroup
    }) as CfnDBProxyTargetGroup
    targetGroup.addPropertyOverride('TargetGroupName', 'default')
  }
}

[tbrand]

I'm not a maintainer but I cloud say

1. Update cdk to the latest version
2. If (1) doesn't solve, try separating the stack of database and it's proxy as a workaround.

[bodokaiser]

@tbrand thank you for your response! I found in the CloudWatch LogGroups that RDSProxy requires the RDS credentials to be formated {"username":"...","password":"..."}, however, my previous cdk stack only stores the password as secret.