Amplify-console: Custom Build Image: Cannot pull container error Too Many Request

[karice]

I've been experiencing an issue today with building using a custom image. The error is telling me the Docker Hub is running into a request limitation. The limit is 100 pull requests per 6 hours fo anonymous access. We definitely haven't gotten close to 100. I can also still pull the image to my computer. We have a few other builds AWS Console which are using the same custom image without issues.

I've seen this error popup a few times randomly. I can usually make it past the error by redeploying but that's not the case today.

[hsspain]

@karice We are looking into this and will let you know when we can resolve this for you.

[lorengordon]

fwiw, pretty sure this is a duplicate: https://github.com/aws-amplify/amplify-console/issues/837

[karice]

837 is asking about adding authentication for the request to docker hub to avoid the rate limit which is 100 requests per 6 hours for anonymous access. We haven't even come close to hitting this limit. We were able to push a different build using the same image a few times but this one is hung up on this error and won't go thru.

[lorengordon]

837 is about exactly this error, I authored it, so I am quite certain. Any solution would be fine, having some way to set authentication is just one possible solution.

It's not _you_ hitting the limit. It's the amplify build service, and the IP that docker hub sees. Which is probably a small-ish pool of IPs across all amplify jobs.

[djensen47]

I'm periodically running into this issue. Given all the other bugs have been closed as a duplicate of this, I just want to make sure this issues does not get lost. Given the new Docker rate limits, this should likely be addressed ASAP. This is like a P0 or a P1 at worst.

[djensen47]

Scratch that, this is now a regular issue.

I use Amplify because of it's simplicity, I don't have to set up Github Actions + S3 + CloudFront; it really is great that way. But despite this issue being open for nearly 3 months and constant warning from Docker that rate limiting will and did happen, nothing is done. What am I paying for if I cannot deploy or constantly have to hit the "Redeploy" button until it is successful?

It took 6 build before it worked. Next time it could be 20 or 50 or 2.

There's not even a work around? If you don't want folks using this service, then please announce it's end of life so we can move on and find a better solution like Digital Ocean or Render.

[asendia]

Scratch that, this is now a regular issue.

I use Amplify because of it's simplicity, I don't have to set up Github Actions + S3 + CloudFront; it really is great that way. But despite this issue being open for nearly 3 months and constant warning from Docker that rate limiting will and did happen, nothing is done. What am I paying for if I cannot deploy or constantly have to hit the "Redeploy" button until it is successful?

It took 6 build before it worked. Next time it could be 20 or 50 or 2.

There's not even a work around? If you don't want folks using this service, then please announce it's end of life so we can move on and find a better solution like Digital Ocean or Render.

CMIIW but there is workaround by using your own image stored in AWS ECR. It probably will cost you < $1/mo, see here for more detailed cost breakdown.

[djensen47]

Okay, thanks for the update. But based on the follow-up response below looks like it's still an issue.

[karice]

@asendia Amplify Console does not allow you to pull custom images from ECR for some reason.Only docker hub. https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html

@div5yesh AWS is planning on releasing a public image hosting service to work around Docker Hub limits. No hard release date yet and don't know when they will change Amplify Console to work with it or ECR. It might be better to find a new CI service. https://aws.amazon.com/blogs/containers/advice-for-customers-dealing-with-docker-hub-rate-limits-and-a-coming-soon-announcement/

[emilio-martinez]

We face this issue every week now. Is there a projected fix for this? It seems like a relatively simple use case.

[kevinold]

I faced this all day yesterday literally not able to run a build without waiting at least 10 minutes to trigger another build. I ended up dropping back to a AmazonLinux container with additional scripts to add what I need from the Docker image

[hyungjunk]

Still experiencing this issue and can do nothing to solve it. Manually redeploying on web console had no luck which did work a few days ago.

[lucashenning]

+1 I've created an AWS ticket regarding this issue. They recommend pushing a custom image to ECR and using it as a build image. The AWS support referenced the following links (incl. this thread) in their response:

[1] Custom Build Image: Cannot pull container error Too Many Request - https://github.com/aws-amplify/amplify-console/issues/1009

[2] Data transfer (number of pulls allowed per 6 hours) - https://www.docker.com/pricing

[3] Pushing an image - https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html

[4] Pulling an image - https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-pull-ecr-image.html

[5] Have amplify build containers support ECR containers & custom container sizes - https://github.com/aws-amplify/amplify-console/issues/493

[6] https://github.com/aws-amplify/amplify-console/issues/254#issuecomment-556550722

[emilio-martinez]

I was able to make Amplify provision with a custom image by publishing a new image version in Docker hub... but can't have our team doing this every time though! Lol

We tried adding items as scripts during preBuild, but it's just too slow of a deploy, so this has quickly turned into a major blocker 😕 we may have to migrate elsewhere, at least temporarily, until this is fixed

[axelson]

Is there a guide to using ECR with Amplify? https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html doesn't contain any information on using an ECR-based image.