Aws-cdk: RFC: What's the right way to think about CDK with CI/CD?

[mikestopcontinues]

I'm in the process of building a CI/CD pipeline in CDK, and I've been going in circles about how to organize the deploy strategy.

At present, I'm trying to fuse all my architecture into a monorepo so that changes to the backend and frontend deploy together. I've got the following stacks in my CDK App:

• CommonStack
  
  
  
  • CodeCommit repo
  
  
  • CertificateManager certificate
  
  
• [Dev|Staging|Prod]ResourceStack
  
  
  
  • Shared S3 buckets
  
  
  • Shared dynamo tables
  
  
  • Shared event-driven lambda funcs
  
  
  • Lambda API nested stacks (containing funcs, api gateways, cloudfront deploys)
  
  
  • NextJS app nested stacks (containing buckets, s3-deployment api gateways, cloudfronts)
  
  
• [Dev|Staging|Prod]PipelineStack
  
  
  
  • CodePipeline
  
  
  • CodeBuild builds
  
  
  • CloudFormation deploy actions
  
  

I've been divided on whether I should:

1. Have the pipeline deploy the resource stack AND changes to itself
2. Have the pipeline only deploy the resource stack, then handle pipeline changes manually
3. Split the resource stacks by how often the resources change

On a related note, I've been trying to reason about the best way to handle deployment failure. Naturally, I can and will try to prevent errors with unit tests, but if I'm CDing infra, what's the best way to handle rollbacks? Is it even possible? And in either case, how do I leverage CodeDeploy rollbacks with my code bound up in CDK?

[skinny85]

Hey @mikestopcontinues ,

as it turns out, we have an entire RFC dedicated to this topic :).

Take a look at this document, and let us know what you think!

Thanks,
Adam

[mikestopcontinues]

Hi Adam,

Actually, between this doc and some further experimentation, I was able to cover everything I was trying for. Thanks!

[skinny85]

That's great @mikestopcontinues !

Can you share anything about your experiments? I'm curious what your conclusions were 🙂.

Thanks,
Adam

[mikestopcontinues]

Sorry for the long delay on this. I got sidetracked solving some other issues that came up refactoring my services into a monorepo, and was just able to close the loop today.

The solution was just to have my [Env]PipelineStacks depend on my [Env]ResourceStacks and my CommonResourceStack. Then the deploy step in my pipeline simply deploys [Env]PipelineStack (with deps).

My naive implementation has two downsides:

1. Changes to the pipeline itself leverage restartExecutionOnUpdate resulting in longer deploys. When it becomes too annoying, I think I'll refactor to have the PipelineStack deploy separately from the ResourceStack.
2. CDK/cloudformation does great with only changing the stuff that changed, but I still have to rebuild all my services to ensure all the files are in place within the CI/CD pipeline. The solution here is to lean on lerna's diff functionality to only rebuild the apps that changed. My lingering question is whether to cache the build directories in codebuild or to split the cdk stack by app as a means of doing the app-specific builds. The latter seems like the better long term solution, but in either case, it's not enough of a pain now to refactor.

Anyway, thanks again for you help on this. CDK is pretty darn cool.

[dsmrt]

I know this is closed but we've been running in circles trying to figure out how to handle CICD with CodeBuild /CodePipeline and CDK. This document is very helpful but I would love more, like a webinar or demo walking through some of these recommendations.

Currently our biggest hurdle is dealing with assets (lambda functions, layers, and nested stacks). Nested stacks has stopped us in our tracks. I assume resolving the following issues will help stream line the process here:
https://github.com/aws/aws-cdk/issues/3463
https://github.com/aws/aws-cdk/issues/1312

Also, due to assets being our largest hurdle, I'm very interested in the cdk-assets tool. This might be a broken link: https://github.com/aws/aws-cdk-rfcs/blob/master/text/cdk-assets.md

CDK has been a lifesaver so thank you for all of your hard work!

[skinny85]

Hey @dsmrt ,

yes, we're aware of these issues, and working hard to improve the situation with assets. Stay tuned - we should have an early access release of this in the not-too-distant future.

[mikestopcontinues]

@dsmrt In the meantime, I can try to help if you give me a better description of your problem. I'm using nested stacks and lambda assets without an issue.

[dsmrt]

Thanks @mikestopcontinues, here's a little more info.

We are still in the development stage, but this was my plan. We have multiple stacks and some of those stacks have nested stacks. (Currently, we are manually deploying these cdk stacks ... we are trying to move away from this).

My pipeline line looks like this:
CodeCommit branch (source) > CodeBuild (build) > CodePipeline (deploy)

Source
Contains my cdk source code (typescript).

Build

1. I package all of the lambda assets and copy them to s3 (with a custom script). Those assets are referenced in the cdk code by the s3 paths. This packaging step is temporary until the following is resolved: https://github.com/aws/aws-cdk-rfcs/issues/49
2. Then I run cdk synth to build all of the Cloudformation templates, which are passed on as deploy input artifacts.

Deploy
The deployment stage runs the Cloudformation deployment actions, one per stack (using the CloudFormationCreateUpdateStackAction action).

My issue is, since the nested templates aren't yet copied to s3, the Cloudformation actions won't see them, correct? Due to the nested stacks being an asset, synth doesn't deploy them to s3. It seems like I'd have to do more scripting to get those files in s3 and update the paths with the recently built templates, but only after cdk synth. Am I understanding this correctly?

Any input is greatly appreciated!

[mikestopcontinues]

@dsmrt I'm not sure if this is going to help much, given any particulars on your end, but the main difference I see between our setups is that I'm not uploading any assets to s3 beforehand. Is there any way you can avoid doing that?

In my setup, the codebuild step first bundles all my resources (mostly funcs, but some static files), then runs cdk synth with those assets in place. So I can reference local files:

   new Function(stack, envStack.envName(id), {
    code: Code.fromAsset(path.resolve(`<pathToFuncBundle>`)),
  })

In this method, synth collects all the necessary files into the .cdk-out directory, so my buildspec can look like this (stripped down):

version: 0.2

phases: 
  install: 
    runtime-versions:
      nodejs: latest

  build:
    commands: 
      - lerna run --scope '@arc/{funcs,admin,hub,view}' build
      - lerna run --scope '@arc/cdk' build

artifacts: 
  base-directory: infra/cdk/.cdk-out
  files: 
    - '**/*'

This way, when the complete artifact package gets copied over for CloudFormationCreateUpdateStackAction and it does its thing.

I use stack dependencies pretty extensively, as you saw in the thread. The basic structure is DevPipelineStack -> DevResourceStack -> CommonStack. Inside my ResourceStack, I use nested stacks. But since the pipeline stack doesn't explicitly reference anything from the others, I set:

  pipelineStack.addDependency(commonStack);
  pipelineStack.addDependency(resourceStack);

The benefit of this is that my pipelines can deploy (Dev|Staging|Prod)PipelineStack, but when I need to, I can manually just the resource stack... something I couldn't figure out with NestedStack.

One last thing, the only way I was able to get anything working myself was to start from a "shell" pipeline stack, then progressively add resources to the resource stack. Anyway, I hope that helps. Let me know how it goes!

[dsmrt]

@mikestopcontinues, this definitely helps but I feel like there's some missing magic here that you or CDK is doing.

The assets built in infra/cdk/.cdk.out are never explicitly copied to s3? How are they referenced within the templates correctly if they aren't in s3? Like with the nested stacks, The template URL needs to be a http url on s3. How does that template get to the correct location? Seems like you could make the artifact bucket the same as the cdk bootstrap bucket or you could overwrite the parameters somehow.

Btw, we have very similar setups which is nice to see! We are using yarn workspaces (instead of lerna for packages (functions, layers, cdk/infra code, etc).

[mikestopcontinues]

It is copied to s3 in the background using an Artifact between codebuild and CloudFormationCreateUpdateStackAction. Here's my unedited code for stringing them together. Since .cdk-out collects all my code resources and my templates, referencing that dir in buildspec is enough.

  stack.pipeline = new Pipeline(stack, envName('Pipeline'), {
    pipelineName: envName('Pipeline'),
    artifactBucket: stack.pipelineBucket,
    restartExecutionOnUpdate: true,
  });

  stack.pipeline.addStage({
    stageName: 'Source',
    actions: [
      new CodeCommitSourceAction({
        actionName: 'CodeCommit',
        repository: stack.repo,
        branch,
        output: stack.sourceArtifact,
      }),
    ],
  });

  stack.pipeline.addStage({
    stageName: 'Build',
    actions: [
      new CodeBuildAction({
        actionName: 'StackBuild',
        project: stack.stackBuild,
        input: stack.sourceArtifact,
        outputs: [stack.stackBuildArtifact],
      }),
    ],
  });

  stack.pipeline.addStage({
    stageName: 'Deploy',
    actions: [
      new CloudFormationCreateUpdateStackAction({
        actionName: 'StackDeploy',
        templatePath: stack.stackBuildArtifact.atPath(pipeStackTpl),
        stackName: pipeStackName,
        adminPermissions: true,
      }),
    ],
  });

[kei3po]

I know this issue has been closed but i am still struggling with the deploy execution of cloudformation. how are you executing the templates without the s3 template parameters for assets (artifactHashParameter, s3BucketParameter, s3KeyParameter)?

[mikestopcontinues]

There's a new pipelines module that solves the asset problem. At the time I wrote the above, I didn't have any assets to deploy, just infra.