Aws-cdk: (cdk-pipelines): Docker rate limit, guidance needed or auth mechanism

[arpowers]

Currently can't deploy because of a rate limiting issue with Docker. You may have seen this in other issues, but the problem here is I have not hit this limit myself (only this single deploy) so for some reason, I believe Docker is flagging all of Codebuild as one "anonymous" source.

Anyway, we need a way to authenticate when using DockerImageAsset from @aws-cdk/aws-ecr-assets or other guidance on how to circumvent this

Error message:

Step 1/9 : FROM node:14-alpine AS build
--
40 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
41 | error  : [100%] fail: docker build --tag cdkasset-d0160342c2de47434404b7f88eaefd92e63e83f7178db2ff9a88cc4e0b62f5b2 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
42 | Failure: Error: docker build --tag cdkasset-d0160342c2de47434404b7f88eaefd92e63e83f7178db2ff9a88cc4e0b62f5b2 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Proposed Solution

We need a way to authenticate to associate our accounts with the pull instead of anon.

[erudisch]

maybe this is helpful:

https://github.com/aws/aws-cdk/issues/10999

[arpowers]

@rix0rrr I know you guys are probably busy with the conference this week, but I can't deploy code due to this problem...

From the output, looks like you are logging in with an AWS user and that is getting rate limited. Not just for me but for everyone.

here's further info:

CDK Code:

/**
     * Build an image with the entire monorepo to work with
     */
    const dockerImage = new DockerImageAsset(this, baseId("DockerImage"), {
      directory: ".",
      file: `./docker/ci.Dockerfile`,
    })

    const image = ecs.ContainerImage.fromDockerImageAsset(dockerImage)

Terminal / Codebuild Output:

[Container] 2020/11/30 17:43:08 Running command cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1"
--
28 | verbose: Loaded manifest from assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json: 2 assets found
29 | verbose: Applied selection: 1 assets selected.
30 | info   : [0%] start: Publishing c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1
31 | verbose: Assume arn:aws:iam::433775104113:role/cdk-hnb659fds-image-publishing-role-433775104113-us-east-1
32 | verbose: [0%] check: Check 433775104113.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-433775104113-us-east-1:c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
33 | verbose: [0%] debug: docker login --username AWS --password-stdin https://433775104113.dkr.ecr.us-east-1.amazonaws.com
34 | verbose: [0%] debug: docker inspect cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
35 | verbose: [0%] build: Building Docker image at /codebuild/output/src177682768/src/asset.c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36
36 | verbose: [0%] debug: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile .
37 | Sending build context to Docker daemon  1.198MB
38 |  
39 | Step 1/9 : FROM node:14-alpine AS build
40 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
41 | error  : [100%] fail: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
42 | Failure: Error: docker build --tag cdkasset-c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36 --file ./docker/ci.Dockerfile . exited with error code 1: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
43 | at ChildProcess.<anonymous> (/usr/local/lib/node_modules/cdk-assets/lib/private/shell.js:46:24)
44 | at Object.onceWrapper (events.js:422:26)
45 | at ChildProcess.emit (events.js:315:20)
46 | at ChildProcess.EventEmitter.emit (domain.js:482:12)
47 | at maybeClose (internal/child_process.js:1021:16)
48 | at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
49 |  
50 | [Container] 2020/11/30 17:43:09 Command did not exit successfully cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1" exit status 1
51 | [Container] 2020/11/30 17:43:09 Phase complete: BUILD State: FAILED
52 | [Container] 2020/11/30 17:43:09 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk-assets --path "assembly-PipelineStack-Pre/PipelineStackPreService574433AC.assets.json" --verbose publish "c1dd1c8474c338cb046b5097f5c51b8d8b00b1c1c54618aa198354cebe8d7a36:433775104113-us-east-1". Reason: exit status 1
53 | [Container] 2020/11/30 17:43:09 Entering phase POST_BUILD

[arpowers]

again today

[JorisLimousinKaizen]

Same problem for us, this is a big issue.

[straygar]

@JorisLimousinKaizen @arkon Not a fix, but have you tried the workaround in the thread @erudisch mentioned?

https://github.com/aws/aws-cdk/issues/10999#issuecomment-716478132

[JorisLimousinKaizen]

Haven't tried it no, it has been working again since a few days, not sure if it has been fixed though.

EDIT: Got rate-limited again. :(

[JorisLimousinKaizen]

Just got the following answer from the AWS support in case it helps, will try it myself soon:

From reading your correspondence I understand that you are currently facing an issue in your CDK pipelines, where builds are failing with the following error:

---

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

---

This is a known issue that occurs now that DockerHub have introduced pull rate limiting. To resolve the error that you receive when throttling happens, you must configure CodeBuild to authenticate the layer pulls using your DockerHub account credentials. Full steps to resolve this issue can be found at the following knowledge center article [1]. However, to summarize;

• Store your DockerHub credentials with AWS Secrets Manager
• Resolve throttling that happens in the PROVISIONING phase
• Resolve throttling that happens in other phases
• Configure CodeBuild to retrieve the secret

If you have any follow-up questions or concerns, please feel free to contact me, I'm happy to help. Please note, my working hours are 9AM -5PM Monday - Friday, Irish Standard Time as I am based in Dublin, Ireland.

[1] How do I resolve the “error pulling image configuration: toomanyrequests” error when I use Docker images in CodeBuild? - https://aws.amazon.com/premiumsupport/knowledge-center/codebuild-docker-pull-image-error

[rix0rrr]

Until we have a proper solution for this, you can try the ECR Mirror construct we have developed here: https://github.com/awslabs/aws-delivlib#ecr-mirror