Watchtower: Support for AWS's ECR

[shofetim]

Amazon Web Service's Elastic Container Registry changes it's authentication token every 12 hrs. So setting REPO_USER & REPO_PASS only work for that long...

I would like to be able to set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_REGION and then have watchtower regenerate (via AWS's API) the authentication token as needed.

Without this, it doesn't appear possible to use watchtower with ECR.

Thanks!

[chrisferry]

Also support for IAM Profile should be added so no access or secret key is needed.
👍

[gabormay]

FWIW I was able to use watchtower with ECR successfully using the following setup:

• Run watchtower with mapping the docker config.json as described under https://github.com/v2tec/watchtower#usage
• Have a cron job set up to run every 6 hours that does $(aws ecr get-login --region <whatever>) or equivalent (in my case I run it as a root user so I'm mapping /root/.docker/config.json above)
• The instance has permissions to access the given ECR repo set up in its IAM profile

[shadycuz]

Why can't you use https://github.com/awslabs/amazon-ecr-credential-helper ??? I used it on my machine (not with watchtower) and it's like ECR = Dockerhub.

Seem like it will work with watchtower.

[vcajes]

@gabormay I am trying to do what you said, but I am receiving this error:

time="2018-05-25T15:30:00Z" level=info msg="Unable to update container /app-frontend, err='Error response from daemon: pull access denied for 1111111111111.dkr.ecr.ap-southeast-1.amazonaws.com/app-frontend, repository does not exist or may require 'docker login''. Proceeding to next."

Do you know what can I do to have the watchover container being able to access to the ECR? Many thanks!!

[gabormay]

@vcajes Well, it's likely something wrong with your setup. Can you check the logs of the 'aws ecr get-login' command? Can you run it manually, see if it works? Did you check the repository address and region? Can you pull the image manually?
FWIW in the meanwhile I started using amazon-ecr-credential-helper as suggested by @shadycuz and it works like a charm, maybe give that a try if all else fails.

[vcajes]

@gabormay Yes, I can run it manually. The docker pull command works well and the aws login as well.

The only problem seems to be that the watchover container is not able to use the AWS credentials to pull the images. Do I need to pass AWS credentials of something to the watchover?

[gabormay]

@vcajes Well, I did everything according to https://github.com/v2tec/watchtower#usage, used the second variant with the config.json mapping and it worked like a charm.

[vcajes]

Thank you @gabormay ! I solved the problem and now is working. The problem was that I installed docker as root. Now installed with the ec2-user of the Amazon Linux AMI and working like charm. Thanks!!

[GreatEarl]

@gabormay Hi mate! I see that you have a working solution. Can you provide your config files as example? Looks that I missed something in my settings. I'm trying to avoid 12h re-login from EC2 (amazon linux) vm to ECR instance.

[GreatEarl]

Eventually I was able to make aws helper working, so I can manually do pulls form ECR, but still can't configure watchtower.
docker logs watchtower gives - no basic auth credentials, while manual pull works and ecr log get filled appropriately. Watchtower initialized by

docker run -d \
  --name watchtower \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower

regarding to this

supply registry authentication credentials with the environment variables REPO_USER and REPO_PASS

a bit confused what need to be set there - I use AWS EC2 and ECR.

[gabormay]

@GreatEarl I think you are still missing the -v /home/<user>/.docker/config.json:/config.json part.

[GreatEarl]

@gabormay thanks mate, already figured it out ))

[andyault]

if anyone is still having troubles with this, I was able to get watchtower to play nice by:

• adding the credential helper to the image myself
• adding { "credsStore": "ecr-login" } to /config.json (not included in my image)
  
  
  
  • note that I wasn't able to use "credHelpers" or any of the tricks seen here without getting "no basic auth credentials" again :(
  
  
• exposing my key id and secret key to watchtower by either env vars or the ~/.aws/credentials file
  
  
  
  • note that I wasn't able to get either of these to work until I made my watchtower-ecr start from alpine instead of from scratch
  
  

My final docker-compose.yml:

version: '3.2'
services:
  my-service:
    image: <id>.dkr.ecr.<region>.amazonaws.com/my-image:latest
  watchtower:
    image: andyault/watchtower-ecr:armhf-latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /path/to/docker-config.json:/config.json
    environment:
      AWS_ACCESS_KEY_ID: <blank or id>
      AWS_SECRET_ACCESS_KEY: <blank or key>
    command: --interval 30 --debug

[jakob-stoeck]

If you are on EC2 you don’t need to login for ECR. Just attach an IAM policy to the EC2 instance and afterwards use ECR without a login:

Add an IAM role with access to ECR: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html

Attach this new role to your EC2 instance: https://aws.amazon.com/blogs/security/easily-replace-or-attach-an-iam-role-to-an-existing-ec2-instance-by-using-the-ec2-console/

[fantapop]

@jakob-stoeck do you have a reference for this? I'm unable to get this working. I've given my ec2 instance the AmazonEC2ContainerRegistryPowerUser policy and when I'm on it and try to pull my containers, I get: Error response from daemon: .... no basic auth credentials.

[laszlojau]

@fantapop To make it work with IAM roles, in addition to the above, you have to mount your AWS credentials file in andyault/watchtower-ecr.

You can use a separate credentials file that refers to the role you have attached to the EC2 instance.

Example aws-credentials file

NOTE: Change the role arn to match the role that you have attached to the EC2 instance.

[profile EC2-ECR-pull]
role_arn = arn:aws:iam::<account-id>:role/<role_name>
credential_source = Ec2InstanceMetadata

You can mount it this way:

    volumes:
      - /path/to/aws-credentials:/root/.aws/credentials

[simskij]

There is actually a way of doing this, described in the docs, which works with the official image:
https://containrrr.github.io/watchtower/private-registries/#credential_helpers

I'd also like to encourage all future readers to never add credential files to your docker images.