Aws-cdk: (cdk-pipelines): Long CloudFormation prepare step causes S3 Request Expired error

[straygar]

Recently, when trying to deploy a change that had a large-ish change-set, the Prepare step for a stack in one of my stages took a particularly long time (~12 minutes). After this time, the Prepare step failed with an S3 Request Expired error (see below).

I was able to work around this issue by splitting my change into multiple smaller commits. I'm not really sure what's happening during this prepare step (I assume it's creating the CFN change-set and uploading it to S3).

Reproduction Steps

1. Make a change that results in a large changeset (~50 CW alarms, with complex metrics containing nested maths expressions in my case).
2. Push the change.

What did you expect to happen?

A successful deployment of the stack.

What actually happened?

The stack spent ~12 minutes in the CloudFormation Prepare step in the first application stage, after which it failed with the following error:

Action execution failed
S3 error: Request has expired For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: f974d950-7771-496a-a9d6-d544848fd316)

Environment

• CDK CLI Version : 1.76.0
• Framework Version: 1.76.0
• Node.js Version: 14.13.0
• OS : Ubuntu 20.04
• Language (Version): TypeScript 3.9.7

---

This is :bug: Bug Report

[rix0rrr]

That is unfortunate. I unfortunately feel this might be outside our control, and purely a function of the interaction between CodePipeline and CloudFormation.

I will try to pass this information on to the CodePipeline team, but I can predict they're going to need more information:

• 12 minutes duration of Prepare seems excessively long to me. Are those times you see regularly, or was this a one-time event where CloudFormation was being slower than usual?
• Would it be possible for you to go into CloudTrail and retrieve the exact API call that produced this error? It would probably help.

[straygar]

@rix0rrr Thanks for the reply! Yeah, it's something I'm able to reproduce consistently. I'm not even sure if this is caused by the change-set being large, since it's hard to get visibility on this issue.

Maybe these would help internal investigation (from the latest failure):

• CodePipeline execution ID: 6232c31d-76db-490a-b2c4-4f99ca292a55
• Failed pipeline action execution ID: c302af56-c7b6-4e40-8c53-34d0823d1491
• S3 request ID: 4629fcd5-3c18-4f1b-8522-c0ff40f82f78

Weirdly enough, I can't find the CFN API calls to create/execute the changeset in CloudTrail for the failing stack, only the 2 stacks that were deploying in parallel to it.

[rix0rrr]

I've linked CodePipeline team to this ticket. That's all I can do, unfortunately.

[TheRealAmazonKendra]

Hi there, I'm from the CodePipeline team. Can you provide the region and time of the executions you provided above?

[straygar]

@TheRealAmazonKendra Hi Kendra!

Sure, this was in eu-north-1. The execution started on 2020-12-08 14:10 UTC.

[TheRealAmazonKendra]

Thanks for the information. The action is owned by CloudFormation so I've provided them with some additional context from our logs and reassigned the ticket to them.

[straygar]

@TheRealAmazonKendra Thanks! Please let me know, when the issue is resolved, or at least some kind of workaround becomes available. :)

[rix0rrr]

Going to close this ticket as it's not actionable for CDK. The CloudFormation investigators can still use this ticket to respond, though.

If weeks go by without response, you can try re-engaging through Customer Support or this GitHub project: https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap

Internal reference: D19336056

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.