Aws-cdk: [pipelines] Change to a single-source, single-build pipeline, deploy+approve model

[rix0rrr]

Realign the API model of the library more towards what's described in Automating Safe, Hands-off deployments.

This means we explicitly give up a bunch of freedom that CodePipeline allows, in order to make a simpler application deployment model. We'll reject adding multiple source stages, multiple builds, and many types of actions in weird locations people might want to insert into the pipeline; if they want to do that, they should drop down to building a complete CodePipeline pipeline.

This means we do the following things:

• Create wrapper classes for GitHubSourceAction and others which have fewer required parameters. We'll only ever have one source action, so no need to specify Artifacts.
• Same for Build Actions.
• Add the concept of an AppDeployment (name TBD); one pipeline stage can have 1 or more deploy AppDeployments; this is in order to do waves where we do multiple apps in parallel in one stage.
• An AppDeployment consists of two parts, that execute exactly in this order
  
  
  
  • 1 or more stack deployments, potentially with some ordering constraints between them.
  
  
  • 0 or more approval actions, that together form a sort of "approval workflow". See the builders library article as well as the internal deployment concepts.
  
  
• There can be some passing of information between the stack deployments and the approval workflow (like the stack outputs now), but it only needs to be very limited in scope.
• The concept of an approval workflow step which can render to a CodeBuild project and a CodeBuild Action, or maybe a Lambda and a Lambda Action, a manual verification, etc.
  
  
  
  • Should allow cross-account and cross-region actions
  
  
• Design mostly to interfaces so we can swap out backends
• There is a use case for "additional actions" between Prepare and Execute. This would be an extension to the Builders Library API as it does not have this concept (see https://github.com/aws/aws-cdk/issues/11333)

---

I think the "action" model (with hooks for users to insert approvals) needs to look somewhat like this:

The "application approvals" should probably be configurable in direct style, it's okay if the changeset approvals require implementing an interface of some sort (not ideal, but acceptable).

---

This might entail making changes to the CodePipeline L2 library; but only in order to make implementing CDK Pipelines easier, NOT to improve the API of the CodePipeline library. Some candidates:

• Move the codepipeline_actions.Action base class to the codepipeline library in order to make it easier to implement Actions.
• Make some Action properties like RunOrder and StackOutputs mutable, so we can change them after having created Actions.

---

I'm looking for compelling use cases that would require us to support multiple sources and/or multiple builds. Please supply them in this ticket if you have them.

On the deployment front, CDK pipelines will never do anything more than deploying CDK apps using CloudFormation, and running some limited set of validations.

[straygar]

Hey!

Great proposal! Coming from the Amazon world myself, I would love to see CDK Pipelines come closer to this model. :)

One use-case I have for multiple sources-builds is deploying a CDK app with a Java Lambda, given:

• a TypeScript CDK application living in repo A
• a Java Lambda code project living in repo B

I would find it nice to have separate source/build triggers for each one of those packages in the pipeline, unless there is a cleaner way to do so. The B repo build in particular could be a code-build project, which produces a JAR. This JAR artifact can then be replicated to an S3 bucket in each stage/account the pipeline deploys to.

Maybe there's a more idiomatic CDK solution to replicate a JAR artifact across multiple stages and using that as the Lambda code source, but I was unable to find a good example of this.

[hoegertn]

Is there a special reason you are having two repos? Imho the CDK way is to have the Java lambda inside the same repo and reference it as an asset. It will then be built once and the jar file will be used in all stages.

[straygar]

@hoegertn "The CDK way" - this is the first time I've heard of this, to be honest. Are there docs defining this as a best practice?

But I guess my reasoning would be separation of concerns - it'd be great to avoid dumping code of different languages, environments and build systems (Java, Scala, Python, TypeScript) in the same codebase. That sounds a bit messy to me (just a personal preference), but maybe I'm not familiar with some tooling that would make it nicer. (but part of the reason is historic - this Lambda repo was never a part of a CDK application or any CI/CD process, which I'm trying to change).

So your preferred project structure would be:

• lib - CDK infrastructure
• src - source code for all ECS services, Lambdas, Glue jobs etc.
  
  
  
  • python
  
  
  • scala
  
  
  • java
  
  

In this case, I suppose build-ing the package would do everything - produce any JARs, Python .eggs, .zips; and then CDK would pick those artifacts up as assets and deploy them?

[hoegertn]

I have written a blog post with a sample: https://taimos.de/blog/build-a-basic-serverless-application-using-aws-cdk

I will try to write something more into this ticket later today.

[straygar]

@hoegertn Thanks! I took a brief look. It seems that like with most CDK examples, it assumes my Lambda will be in TypeScript, which is much more straightforward to build, package and deploy from a TypeScript CDK project, than a Java lambda.

Alternatively, I guess you could have multiple projects in the same repo, if you wanted to:

• infrastructure - only TypeScript CDK stuff
• data-pipeline - some Python Glue jobs, with their own build system
• beer-service - some Java ECS service, with its own build system

And the infra package would build the other 2 projects prior to deploying. I don't know, to me it feels more natural to allow replicating a CodePipeline artifact to multiple stages. :/

In any case, looking forward for any tips you might have. :)

[corymhall]

@straygar this blog post goes into more detail around how you can build, package, and deploy with the CDK. It uses Golang as an example, but the same concepts can be applied to other languages.

https://aws.amazon.com/blogs/devops/building-apps-with-aws-cdk/

[nbaillie]

@rix0rrr , @hoegertn ,
Good to see the outline of the direction of travel above. Pipelines is a great addition to CDK.

Currently we have a pattern that can includes some db schema migrations before and some testing after deployment of stages.

Would it be the intention that in this mode of working migrations could occur in "_'additional actions' between Prepare and Execute_" or would they be before and separate to the AppDeployment within the stage?

For the integration testing we are using jest that runs after the stage is deployed, so hopping that this will still be supported.

Right now for both migration and testing we are using ShellScriptAction, would there be a place for something similar in this new overall approach?
Just this part that made me want to ask "_We'll reject ... many types of custom actions people might want to insert into the pipeline_"

so support something per stage like:
[Migrate -> (Prepare -> Deploy) -> IntegrationTest]
or:
[(Prepare -> Migrate -> Deploy) -> IntegrationTest]

The approval ideas described will address other challenges we have around visibility and control.

[hoegertn]

Is there a special reason you are not doing the DB migrations within your app deployment by using CFN custom resources for example?

[nbaillie]

@hoegertn , Not so much a special reason, but right now we use yarn script to run a typeorm migration and trigger via a shell command in ShellScriptAction. Ideally would be good to keep this however... one of the main reasons i asked is to try and ensure that we are looking at all options, and options that will have a synergy with the project direction.
Would be good to see what the overall idea for this type of thing looks like.

[rix0rrr]

@nbaillie to be clear:

"We'll reject ... many types of custom actions people might want to insert into the pipeline"

This was intended to be about locations in the pipeline, not specifically about the action types themselves. Although we won't offer out-of-the-box support for all types of actions, we won't prevent you from adding custom ones. I've updated the comment to reflect that.

But they will have to be in one of a couple of predefined positions:

• Before the app
• Between prepare and execute
• After the app

Right now for both migration and testing we are using ShellScriptAction, would there be a place for something similar in this new overall approach?

Something like ShellScriptAction will definitely still exist afterwards.

[mrpackethead]

"I'm looking for compelling use cases that would require us to support multiple sources and/or multiple builds. Please supply them in this ticket if you have them."

Sorry i'm coming to the party rather late, - This description is copied from a slack chat, sorry if its overly wordy.

I'm working on a project where I'm using cdk pipelines to do a multi-accoutn deployment for dev/test/prod. As part of the stack, i'm deploying containers on ECS.. I have to be able to support container image building, and placement into ECR, when a Github repo is updated. There is multiple images, each with its own github repo. When any of the images are updated through their respective codebuild, I need to do a green blue reployement of that image through the various stages.

As far as i could tell/discover, cdk pipelines dont' support multiple sources.. ( please, if they do, let me know how! ).. so, i had to find a way around this.. I thought the solution, I came up with, to work around this limitation was interesting, and i'd not seen it used before ( it may well have been of course ).

What I did was to add a few extra lines into the buildspec that is used for the image build.

  post_build:
    commands: |
      echo Build completed on $(date)
      echo Pushing the Docker image...
      docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      aws ssm put-parameter --name "$INFRA_PIPELINE/$IMAGE_NAME/CurrentTag" --type "String" --value "$IMAGE_TAG" --overwrite
      aws codepipeline start-pipeline-execution --name $INFRA_PIPELINE

I'm putting the image tag ( which is a commit_id hash ) in a SSM parameter, and then having the code kick the code exectution off.
This in effect gets around the limitations of only having a single source for the cdk pipeline

The code and dockerfiles for the various containers come from different repos because the code/repos are owned by different business units, and in one case a different organization entirely. They do allow us to read the repos and to pull from the on commits. ( we have created codestar connections to them ).
While it would be nice and simple for all the assets for the project to live in a single place, its never going to happen.
This is the first time i've needed to use a 'multi-source' pipeline ( this is my sixth project now with cdk pipelines ) . I'm not sure this is a corner case either, though i can only see the world from my world.

[mrpackethead]

"This means we explicitly give up a bunch of freedom that CodePipeline allows, in order to make a simpler application deployment model. We'll reject adding multiple source stages, multiple builds, and many types of actions in weird locations people might want to insert into the pipeline; if they want to do that, they should drop down to building a complete CodePipeline pipeline."

I read @clareliguori Blog and couldn't find anywhere specifically in that article that said there should only be a single source.?

Edit: Asked Clare what she thinks