Aws-cdk: [pipelines] Asset stage can't support more than 50 assets.

[njlynch]

When attempting to deploy a pipeline with more than 50 assets, the deploy fails due to the Asset stage having more than 50 actions.

Reproduction Steps

Deploy a pipeline with > 50 assets.

    for (let i = 0; i < 32; i++) {
      const websiteBucket = new s3.Bucket(this, `WebsiteBucket${i}`, {
        websiteIndexDocument: 'index.html',
      });

      new s3deploy.BucketDeployment(this, `DeployWebsite${i}`, {
        sources: [s3deploy.Source.asset('./test-file-asset', { assetHash: `file-asset-${i}` })],
        destinationBucket: websiteBucket,
      });

      new ecr_assets.DockerImageAsset(this, `DockerAsset${i}`, {
        directory: './test-docker-asset',
        extraHash: `docker-asset-${i}`,
      });
    }

Error Log

On deploy, the following error is hit:

Pipeline stage 'Assets' has too many actions. There can only be up to 50 actions in a pipeline stage (Service: AWSCodePipeline; Status Code: 400; Error Code: InvalidStageDeclarationException; Request ID: xxx)

Other

The approach will likely be to dynamically create new asset publishing stages to the pipeline when > 50 assets are created. This will require API support in CodePipelines, as currently stages can't be inserted, only added to the end of the pipeline.

---

This is :bug: Bug Report

[nonken]

I am running into this as well in a serverless setup since I use one asset per Lambda to keep function size minimal.

Right now my workaround is to have a few lambdas handle several routes but I’d rather keep a 1:1 mapping.

As an addition, it might be helpful to highlight the potential financial impact for this. I am assuming it is just build minutes but haven’t dug in. Don’t want to run I to a surprise bill 😅

[MousaZeidBaker]

This also limits CDK apps to have maximum 25 stacks since two actions (one prepare & one deploy) are created in the deploy stage for each stack. Our app have already reached this limit. One possible workaround, to at least increase the limit to 50 stacks, is to put the prepare & deploy actions into different pipeline stages.

[seawatts]

@rix0rrr Is there a plan of how this will potentially be implemented yet? I'm coming pretty close to this limit, 48 assets, so if there is anyway I can help push this through that would be great!

[alexpulver]

AWS CodePipeline has (at the time of this writing) a non-adjustable quota of 50 total stages per pipeline and 50 total actions per stage. High number of artifacts can drive a pipeline to exceed 50 stages, hitting the non-adjustable quota.

What about using AWS Step Functions to execute AWS CodeBuild projects using a Map state? CodePipeline has a native integration with Step Functions, so no custom code would be required for such invocation. Step Functions has a native integration with CodeBuild, so no custom code would be required here as well.

The flow would be:
CDK Pipelines Assets stage => Step Functions action => Map state with artifact IDs (potentially with concurrency) => CodeBuild project => cdk-assets command.

The Map state can hold IDs of the artifacts (managed by pipelines module), for example, and set them as environment variables for the CodeBuild project (using environmentVariablesOverride parameter). The build would use cdk-assets to uploads these artifacts. Map state supports concurrency, which can further speed up processing. Step Functions now supports payload sizes up to 256KB, which should support processing large number of artifacts in a single CodePipeline stage and action. The number of artifacts processed depends on what will be the actual parameters passed to CodeBuild from Map state array.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[JFox]

@njlynch Thanks for the fix! However, the same issue exists when a single stage has more than 50 actions as well, either by having more than 25 stacks (1 prepare action + 1 deploy action) or less stacks + manual approvals. Is there a plan to fix that limit as well?