Aws-cdk: Graduate @aws-cdk/aws-config to stable

[iliapolo]

We plan on graduating this module to stable.

What does it mean?

It means we will start providing semantic versioning guarantees on the existing API's.

Use this issue to provide feedback about the current API and any changes you think are warranted.

Following are the tasks needed to complete before flipping the switch:

• [x] Validate no p0 bugs in the last 4 weeks.
• [ ] Validate L2 coverage.
• [x] Clear awslint exclusions (https://github.com/aws/aws-cdk/pull/6671)
• [ ] Conduct API review.

[richardhboyd]

https://github.com/aws/aws-cdk/issues/5874#issuecomment-581165099

[iliapolo]

@richardhboyd see my response: https://github.com/aws/aws-cdk/issues/5874#issuecomment-581398003

[jogold]

Also to be considered https://github.com/aws/aws-cdk/issues/5181

[iliapolo]

@jogold @MrArnoldPalmer @nija-at @ccfife

I'd like to kick-off a discussion on whether or not we should go ahead with graduating this module.

The problem is that in order to use the AWS config, one needs to first setup a ConfigurationRecorder and a DeliveryChannel. Before creating those resources, config rules cannot be created. The existing CDK constructs do not support creating them, and therefore, users have to either:

1. Create those resources using the L1 objects, as attempted here. This is a rather complex setup, and as the issue states, might be buggy and not fully supported.

2. Create those resources manually in the aws console prior to using CDK. This process is also quite complex and prone to user errors.

Neither of these methods provide a slick CDK experience that is in par with our other modules. When we graduate modules, we should consider the developer experience as a whole, not just with existing constructs.

I'd like your thoughts on this to decide if we push back on graduation until we provide support for ConfigurationRecorder and DeliveryChannel. My personal take is that we should put the breaks and invest our effort to adding those necessary L2 constructs.

[jogold]

My personal take is that we should put the breaks and invest our effort to adding those necessary L2 constructs.

Agree

[nija-at]

@iliapolo - can you continue this story, so I can understand what the experience would be like, once this gap is covered (and partly because I know nothing about the AWS Config service)?

Once a ConfigurationRecorder and a DeliveryChannel is set up, how would that fit into the rest of the CDK app, say in the case of, this app?

Skimming through the API, I don't see a property which takes the delivery channel as input, so what is the modeling of delivery channels, configuration records and rules?
Would there be 1 DeliveryChannel per app, or a one time set up per account per region?

[iliapolo]

@nija-at - The ConfigurationRecorder and DeliveryChannel are just enablers for the rules. You're right that in order to create rules, one doesn't need to refer to them, they just need to exist.

To clarify, the ConfigurationRecorder is just a specification of which resources will AWS even keep track of, and the DeliveryChannel is essentially which bucket to put the results in.

They are defined per account per region, but can be updated during their lifecycle. For example, I imagine you would normally start with minimum resource types to record (its charged per type), and as you add more rules, you might update the recorder with more resource types.

The current experience is that if you haven't done the initial setup, creating the rules will simply fail. If you have done it, but are defining rules for unrecorded resource types, you just won't have the data to actually employ those rules.

Im not entirely sure what the experience will look like because you don't actually reference any resources when you configure a rule, its just a lambda function that uses the aws.ConfigService object.

Once a ConfigurationRecorder and a DeliveryChannel is set up, how would that fit into the rest of the CDK app, say in the case of, this app?

If they are already setup, you don't strictly need them in the CDK app, unless you want to update them in the case i mentioned. Once we add support for those L2's, they will be defined as free floating resources in the app, and will be changed when your rules require additional configuration tweaks.

Does this answer you questions?

[nija-at]

unless you want to update them in the case i mentioned

If these are set up out of band (i.e., outside of the CDK app), why would they be updated as part of the CDK app?

they will be defined as free floating resources in the app

I'm not aware of this term. What does this mean?

Once we add support for those L2's

Should we even be providing L2s for these resources? In other words, should customers be defining these in their CDK apps?
On the other hand, perhaps we should model these as part of cdk bootstrap where if the app has 'AWS Config' rules, CDK sets up a recorder and delivery channel in the account + region that is being bootstrapped. Can the 'update' experience also be modeled here?

Overall, I agree with your assessment that we should hold off on graduation.
As a next step, we should design the correct user experience.

[iliapolo]

If these are set up out of band (i.e., outside of the CDK app), why would they be updated as part of the CDK app?

Well, you might need update them because you are adding rules in your CDK app that assumes specific resource types are being recorded. Thats exactly the case where you don't want to define them out of band, but inside the app.

I'm not aware of this term. What does this mean?

Ha yeah i made it up. I mean resources that are not connected to any other resources. Basically the ones who are not used as arguments in any other resource.

Should we even be providing L2s for these resources? In other words, should customers be defining these in their CDK apps?

Thats a good question we need to discuss.

Overall, I agree with your assessment that we should hold off on graduation.
As a next step, we should design the correct user experience.

👍

[ajaykarpur]

What is the timeline for adding ConfigurationRecorder support? I had tried to use AWS Config as follows to enable drift detection:

export function createDriftNotification(stack: cdk.Stack): void {
    // Create an AWS Config rule to detect stack drift
    const driftRule = new config.CloudFormationStackDriftDetectionCheck(stack, "Drift", {
        ownStackOnly: true,
        configRuleName: `${stack.stackName}Drift`,
        maximumExecutionFrequency: MaximumExecutionFrequency.ONE_HOUR,
    });

    // Create a topic for drift compliance events and subscribe on-call email address
    const complianceTopic = new sns.Topic(stack, "DriftComplianceTopic");
    complianceTopic.addSubscription(new EmailSubscription(Constants.oncallEmailAddress));

    // Send notification on drift compliance changes
    driftRule.onComplianceChange("DriftComplianceChange", {
        target: new targets.SnsTopic(complianceTopic),
    });
}

But I ran into the previously-mentioned issue wherein creating the rule fails.

It is certainly possible to create the ConfigurationRecorder out-of-band, but doing this through the AWS Console forces you to also create additional resources out-of-band (an S3 bucket and an IAM role). This defeats the purpose of setting up a drift detection check, since the idea is to prevent any out-of-band modification of resources. Our goal is for the CDK app to be the sole source of truth for the resources in the AWS account.

[pgollucci]

@jogold @MrArnoldPalmer @nija-at @ccfife

I'd like to kick-off a discussion on whether or not we should go ahead with graduating this module.

The problem is that in order to use the AWS config, one needs to first setup a ConfigurationRecorder and a DeliveryChannel. Before creating those resources, config rules cannot be created. The existing CDK constructs do not support creating them, and therefore, users have to either:

1. Create those resources using the L1 objects, as attempted here. This is a rather complex setup, and as the issue states, might be buggy and not fully supported.
2. Create those resources manually in the aws console prior to using CDK. This process is also quite complex and prone to user errors.

Neither of these methods provide a slick CDK experience that is in par with our other modules. When we graduate modules, we should consider the developer experience as a whole, not just with existing constructs.

I'd like your thoughts on this to decide if we push back on graduation until we provide support for ConfigurationRecorder and DeliveryChannel. My personal take is that we should put the breaks and invest our effort to adding those necessary L2 constructs.

Agree so much.

[pgollucci]

unless you want to update them in the case i mentioned

If these are set up out of band (i.e., outside of the CDK app), why would they be updated as part of the CDK app?

they will be defined as _free floating_ resources in the app

I'm not aware of this term. What does this mean?

Once we add support for those L2's

Should we even be providing L2s for these resources? In other words, should customers be defining these in their CDK apps?
On the other hand, perhaps we should model these as part of cdk bootstrap where if the app has 'AWS Config' rules, CDK sets up a recorder and delivery channel in the account + region that is being bootstrapped. Can the 'update' experience also be modeled here?

Overall, I agree with your assessment that we should hold off on graduation.
As a next step, we should design the correct user experience.

I like the bootstrap idea. Really it should a bunch of stuff like that including ServiceLinked Roles.

[ajaykarpur]

What is the timeline for adding ConfigurationRecorder support?

@iliapolo @nija-at Any updates on this?

[iliapolo]

@ericzbeard Can you provide some details on this?

[ericzbeard]

@shivlaks has been looking at this, and we agree that setting up the prerequisites is going to be important for the developer experience. But it should be possible for us to graduate the rest of the library and then go back and add this later without any breaking changes. Config is in line behind CloudFront and RDS for the team to tackle as a group, so we should be able to show some progress here by the end of Q3. We will post a project board and milestones as soon as we finalize the plan.