Aws-cdk: [pipelines] Add stage to strip assets out of cloud assembly before deploying to CloudFormation

[MamishIo]

To avoid the CodePipeline artifact size limit in CloudFormation deploy actions, the pipeline should generate an intermediate artifact which is the cloud assembly but with asset files removed, and use this as the input for the deploy actions.

Use Case

Regardless of the source provider used, CFN deploy actions have an input artifact size limit of 256MB. The CDK pipeline uses the initial cloud assembly, containing all asset files, all the way through to the CFN action inputs, even though the stacks don't require them (as far as I understand the asset system, all assets are published and linked to CFN parameters by this point).

For builds that produce large/multiple assets totalling over 256MB, this causes CodePipeline limit errors in the deployment stages. Assemblies up to 1GB or 5GB (depending on the source provider) could be produced with this change.

Specific example: monorepos used to build many related services that are all deployed as separate containers/functions/etc.

Proposed Solution

Add an extra pipeline stage after asset publishing and before application stage deployment, which runs a CodeBuild action to load the cloud assembly, strip out asset files, and generate a new artifact containing only the CFN templates and any data necessary for CFN. The CFN actions should use this new artifact as their input.

Other

• This is currently exaggerated by the lack of a de-dupe option when deploying multiple application stages using the same assets - I believe feature request #9627 will reduce code size substantially.

• Overall code size can be reduced by using Lambda layers, but this adds build and deploy complexity compared to using standalone code assets.

  • [] :wave: I may be able to implement this feature request
  • [] :warning: This feature might incur a breaking change
    (:warning: assumes I haven't overlooked some need for the CFN actions to have direct access to asset files)

---

This is a :rocket: Feature Request

[seawatts]

I'm also seeing this error and becoming very close to hitting the limit 253MB with 36 lambdas, 1 docker container, and two application stages, Staging and Prod.

[seawatts]

@MamishIo any progress here?

[jonathan-kosgei]

Ran into this too while deploying to multiple regions, worked for 2 regions, got the limit on 3.

[jonathan-kosgei]

@MamishIo is there any workaround for this?

[rix0rrr]

There is no easy workaround as of yet.

[rix0rrr]

What could work is producing 2 cloud artifacts from the synth step (one with the assets, one without) and then using property overrides to switch between them for the different actions.

[jonathan-kosgei]

@rix0rrr Is there any timeline when this might be fixed? We're not able to use pipelines for a multi-region setup because of this.

[rix0rrr]

There is no timeline as of yet.

[rix0rrr]

Another workaround you could try is postprocessing the .json files in a post-build script in your Synth step and dedupe the assets yourself.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[maxdumas]

I'm not sure if this issue is completely fixed by #11008, as it is reported. #11008 only fixes issues of assets being needlessly duplicated—it doesn't do anything to solve the issues with assets needlessly moving forward in the pipeline and potentially hitting size limits. I'm currently encountering this issue, as my assets include several large Docker images with large build context dependencies. As a result, the CloudAssembly artifact hits 4.6GB in size by the time it goes forward into the CFN deployment stage.

@rix0rrr

[vibe]

I'm not sure if this issue is completely fixed by #11008, as it is reported. #11008 only fixes issues of assets being needlessly duplicated—it doesn't do anything to solve the issues with assets needlessly moving forward in the pipeline and potentially hitting size limits. I'm currently encountering this issue, as my assets include several large Docker images with large build context dependencies. As a result, the CloudAssembly artifact hits 4.6GB in size by the time it goes forward into the CFN deployment stage.

Also faced this issue recently. Even after some optimization, I'm uncomfortably close to the 256MB limit.