Aws-cdk: [lambda-nodejs] doc issue - Too Many Requests Docker Hub rate limit existed

[ookangzheng]

While building lambda-nodejs I encounter this issue serveral time today while using codebuild.

Error Log

Environment

• CLI Version : 1.54.0
• Framework Version: 1.54.0
• Node.js Version: v14.5.0
• Language (Version): Typescript 3.9.6

---

This is :bug: Bug Report

[j0k3r]

FWIW we got the same issue in AWS CodeBuild when using a custom image from Docker Hub. Using dedicated credentials to authenticate the request solved the issue for us.

As of now, without authenticating, it seems to be fixed too.
Maybe AWS (globally) hit the rate limit on Docker Hub 🤷‍♂️

[ookangzheng]

Our workaround will be using buildspec.yml login to a docker hub account to bypass rate limit.

[eladb]

@jogold let's add a documentation snippet to the bundling README with some guidance?

[jogold]

This looks like a temporary issue in CodeBuild not related to the aws-lambda-nodejs module? Can you confirm this @ookangzheng?

[ookangzheng]

What do you mean "temporay issue" in CodeBuild?
I agree, it maybe not related to aws-lambda-nodejs module, I hope maybe some how add a WARNING that let other users know it may need to login to docker hub everytime during CodeBuild.

[jogold]

I'm never logged in to Docker hub in CodeBuild and I don't have the issue. Anything specific about your setup? Are you pulling other images? With the Docker caching mechanism the image used by aws-lambda-nodejs should not be pulled more than once per CI run (and can be cached across runs)

[ookangzheng]

Currently we have this in our Codebuild, during development stage, we often run 3-4 times every hour.

1. A docker image from docker hub: docker:dind
2. Will pull down Lambda typescript files in Codebuild container
3. Do cdk deploy will compile with aws-lambda-nodejs parcel v2 on Codebuild container.
4. No cache.

[j0k3r]

Have you tried https://aws.amazon.com/fr/blogs/devops/how-to-use-docker-images-from-a-private-registry-in-aws-codebuild-for-your-build-environment/ ?

[ookangzheng]

Nope

[nbaillie]

i have seen this issue today. with "@aws-cdk/aws-lambda-nodejs": "1.74.0",

stderr: Sending build context to Docker daemon   2.56kB
--
341 |  
342 | Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-nodejs12.x
343 | Step 2/9 : FROM $IMAGE
344 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
345 | Subprocess exited with error 1
346 | Error: Subprocess exited with error 1

@jogold was working before with "@aws-cdk/aws-lambda-nodejs": "1.71.0", and when i rolled back it worked again.
I tried to run with "1.74.0" around 4-5 times over a few hours it failed, and when i rolled back to "1.71.0" worked then next failed run.

What is the recommend way to login to docker-hub within cdk-pipeline and using aws-lambda-nodejs?

im gessing it is affeced by this perhaps: Docker rate-limits

[jogold]

im gessing it is affeced by this perhaps: Docker rate-limits

Yes that's it... see also https://aws.amazon.com/blogs/containers/advice-for-customers-dealing-with-docker-hub-rate-limits-and-a-coming-soon-announcement/ and the Coming soon part at the end.

[ookangzheng]

We migrate from Docker hub to Amazon ECR though because of parcel build going to fetch from Docker Hub, therefore we will log in to Docker Hub at Codebuild.

buildspec.yml example

env:
  variables:
  secrets-manager:
    DOCKER_HUB_USERNAME: docker-hub-credential:username
    DOCKER_HUB_PASSWORD: docker-hub-credential:password

phases:
  install:
    commands:
      - docker login --username ${DOCKER_HUB_USERNAME} --password ${DOCKER_HUB_PASSWORD}
      - npm install -g --unsafe-perm=true esbuild ## Recommend change from Parcel to esbuild 

package.json

{
"xxx": "xxx"
"scripts": {
    "build": "esbuild ./src/lambda.ts --bundle --platform=node --target=node12 --outdir=dist --external:aws-sdk --minify"
}
}