Copilot-cli: CodeBuild buildspec ECR docker login

[sirbully]

Hello! I was wondering if anyone from the team might have encountered this before. I set up the pipeline using copilot pipeline init and it generated the pipeline.yml and buildspec.yml files. Eventually, we got a toomanyrequests error since Docker Hub implemented a pull rate limit last year. We decided to move the image to ECR to avoid this. But I'm having issues with authentication inside the buildspec.yml file

Scenario:

1. I have an image in a private ECR repository that I created manually
2. I have a Dockerfile to pull that image
3. CodeBuild should build the new image from Dockerfile

I get the following error in CodeBuild:

Step 1/8 : FROM xxx.dkr.ecr.{region}.amazonaws.com/{image}:{tag}

pull access denied for xxx.dkr.ecr.{region}.amazonaws.com/{image}, repository does not exist or may require 'docker login'

I tried putting the command $(aws ecr get-login --no-include-email --region {region}) in the install phase and build phase and also right after the # Build images comment block in the generated buildspec.yml file.

I also tried aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin xxx.dkr.ecr.{region}.amazonaws.com but it didn't work either.

If I remove the login, I get a no basic auth credentials error instead. Sorry that it's not necessarily related to Copilot!

[efekarakus]

Heya @sirbully !

Sorry to hear you hit the dockerhub limit :( Here is a way to work around it:
https://github.com/copilot-example-voting-app/api/commit/7834a2c66cbb59744229f10a3ee1681b103ad8d6

1. I stored my Docker username and password in SSM Parameter Store with the param names DOCKER_USERNAME and DOCKER_PASSWD as SecureString parameters.
2. And then I updated my buildspec as shown above to get around it.

Hope this works!

For the ECR get-login route, my guess is that the command doesn't work because the BuildRole doesn't have permission to GetAuthorizationToken. One way to mitigate it would be to add the AmazonEC2ContainerRegistryReadOnly manged policy to the role. (https://docs.aws.amazon.com/AmazonECR/latest/userguide/security_iam_id-based-policy-examples.html)

[kohidave]

It might also be worth while to see if your base image is in the ECR Public repository: https://gallery.ecr.aws/

Lots of public images are there and the rate limits are much more generous :)

[sirbully]

Thanks for that suggestion @kohidave! I wasn't aware that this was a thing. I'm assuming that this doesn't require me to docker login inside the buildspec.yml file in order to pull the image right?

[sirbully]

Hey @efekarakus, this lead me to the right direction! I looked at the BuildRole that was automatically generated after pipeline init and dived deeper into the policies. I found this specific block in the CodeBuildPolicy which resolved the error:

"Condition": {
    "StringEquals": {
        "ecr:ResourceTag/copilot-application": "{app-name}"
    }
},

I had manually created the private repository in ECR and didn't add any tags. After this discovery, I added the copilot-application(key): {app-name}(value) tag to my repository and it started working.

It seems that logging in to my AWS ECR registry was already successful from the beginning. I looked back at my build logs and there was always a Login succeeded after $(aws get-login). I just didn't set up proper permission for CodeBuild to pull the image from the repository that I created since it had a condition.

Thank you both again @kohidave @efekarakus ✨

[sugarjig]

I just ran into the toomanyrequests error as well. I am building a custom image from an official PHP image in Docker Hub. I'm wondering, can this be worked around by caching the Docker images in the buildspec? Something like:

cache:
  paths:
    - '/var/lib/docker/**/*'

[bvtujo]

Hey @sugarjig it looks like Codebuild has a native Docker layer cache which requires no buildspec configuration after enabling. This is definitely something we should add to our pipeline Cloudformation, or optionally expose in pipeline.yml.

You can't currently take advantage of this without modifying your pipeline manually, but I'll create an issue to capture this feedback. Give it a +1 if it's important to you and that will help us prioritize!

[sugarjig]

@bvtujo I think it would also be useful to support custom local caching. A good example of this is if we use npm to run tests. We first need to run npm install to download dependencies. It can significantly speed up builds if can cache the /root/.npm/ directory between builds.

[mikelhamer]

You can't currently take advantage of this without modifying your pipeline manually,

@bvtujo, which pipeline settings could be manually tweaked to enable this?