Aws-cdk: [cdk-pipelines] Pipeline template grows very fast reaching the 460,800 bytes limit from CloudFormation

[JFox]

:question: General Issue

The Question

After adding around 14 stacks (25 counting nested stacks) to a single Pipeline Application (Stage) the pipeline references 27 assets and the template grows above CloudFormation's 460,800 bytes limit, blocking the pipeline and making it unusable.

Is there a way to avoid this by compressing the generated template or by splitting it somehow? I'm facing this on a staging stage and production is way bigger

My suspicion is that the assets make the template grow too fast as each of them generates a build action with the whole chain of roles, policies, etc.

Environment

• CLI Version : 1.51.0 (build 8c2d53c)
• Framework Version: 1.51.0 (build 8c2d53c)
• Node.js Version: v12.18.0
• OS : aws/codebuild/standard:4.0
• Language (Version): TypeScript (3.9.5)

[njlynch]

This sounds related to #9066, but possibly broader in scope.

[srethira]

I ran into same exact problem. Here is the github issue I created - https://github.com/aws/aws-cdk/issues/9237. Is there any workaround for this problem, until further fix to it ?

[njlynch]

Hi @JFox (and @srethira) --

I just merged a change (#9243) that should _help_ with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.

Thanks!

[JFox]

Thank you so much for taking care of this and for doing it so quickly.

I would keep a very close eye and give it a try as soon as it is released and get back to you with specific numbers.

[srethira]

@njlynch Thanks for providing the fix in a short period of time. Do you know when this will be released?

[njlynch]

@njlynch Thanks for providing the fix in a short period of time. Do you know when this will be released?

Soon! We typically release once a week.

[seawatts]

Hi @JFox (and @srethira) --

I just merged a change (#9243) that should _help_ with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.

Thanks!

I've got about 10 stacks with around 30 - 40 assets, with more to come. So we'll see. 🤞 I'll keep you updated.
I was thinking I could just create multiple pipelines / apps, however, because I have a shared stack that all the other stacks rely on, that solution doesn't work.

[seawatts]

Just tried out the latest fix (#9243) with v1.55.0 But now I am seeing a new error 😦

Maximum policy size of 10240 bytes exceeded for role PipelineStack-PipelineAssetsRole9B011B83-1UOLXJQTUXXXX

cc: @njlynch

[njlynch]

@seawatts - Thanks for the quick turn-around and feedback. We'll investigate the new issue with the policy size.

[JFox]

As a first test, we went by deploying the new version it to a fresh new account and it seems to work now, we are not getting the policy limit mentioned by @seawatts. However I won't totally discard it. I will post back once we run it in the pipeline that is nearly below the 460,800 bytes limit

[JFox]

I correct myself, we also hit the Maximum policy size of 10240 bytes exceeded

[srethira]

Hi @JFox (and @srethira) --
I just merged a change (#9243) that should _help_ with this issue. Given your description (14 stacks, 25 including nested stacks, and 27 assets), I'm not 100% sure that it will be enough, but it should significantly help. (With even just 10 assets, this change cut the template size in half). The fix will go out in the next release; once it does, I would appreciate any feedback on if it was sufficient to solve the problem for you, or if more work is necessary.
Thanks!

I've got about 10 stacks with around 30 - 40 assets, with more to come. So we'll see. 🤞 I'll keep you updated.
I was thinking I could just create multiple pipelines / apps, however, because I have a shared stack that all the other stacks rely on, that solution doesn't work.

That's my thought process to overcome to issue. Unfortunately, I'm in the same boat as you're on with stack dependency.

[njlynch]

9333 has been merged and will go out in the next release. I can create a pipeline with 50 assets successfully with this new approach. After that, we run into issues with limits on the number of actions in a single Pipeline stage. #9353 has been filed to track support for > 50 assets.

[JFox]

9333 has been merged and will go out in the next release. I can create a pipeline with 50 assets successfully with this new approach. After that, we run into issues with limits on the number of actions in a single Pipeline stage. #9353 has been filed to track support for > 50 assets.

Great news! Thanks again for getting it done so fast

[nonken]

Super awesome!!! 🙏🏻

[njlynch]

CDK 1.56.0 was released on Friday; please do let us know if this solved the issue for you.

[srethira]

@njlynch I'm possibly running into #9353

71/101 | 4:35:41 PM | UPDATE_FAILED | AWS::CodePipeline::Pipeline | Pipeline/Pipeline (Pipeline9850B417) Pipeline stage 'Assets' has too many actions. There can only be up to 50 actions in a pipeline stage (Service: AWSCodePipeline; Status Code: 400; Error Code: InvalidStageDeclarationException; Request ID: 9304128e-3fa6-4793-8079-2525422cd668)

[JFox]

We deployed version 1.56.0 successfully with 31 assets being generated in the Assets stage. I guess this will buy us enough time for the root cause to be addressed. Thanks @njlynch

[njlynch]

Thanks for confirming, @JFox. Closing this out and tracking the remaining scaling issue (50 assets/actions) in #9353.