Rancher: Feature Request: Support for AWS Elastic Container Registry (ECR)

[johnrengelman]

ECR is a bit different in that you have to call an AWS endpoint to request a temporary token for accessing the registry (I believe credentials last for 12 hours). Would be great to have built in support for this.

In the UI, I would provide an AWS Access Key and Secret Key, and Rancher would periodically request the token and login to the registry.

[johnrengelman]

Any chance this could be implemented soon?
If you could provide some pointers of where to start looking to add support, I can take a look at creating a PR.

[0ff]

@johnrengelman Do you know if there's a workaround to make it work right now, such as getting the token by hand and updating it manually? I would also like to run docker images stored in our private ECR, but I cannot seem to find a way to get Rancher authenticated.

Thanks in advance for your help!

[johnrengelman]

I suppose you can do a getAuthToken from ECR and put the username/password it gives you into Rancher. Don't see why that wouldn't work. It will only last 12 hours though.

[0ff]

Okay, that sounds like a short-term solution indeed, thanks!
My guess is that rancher doesn't support any other auth apart from username/password at all.
See https://github.com/rancher/ui/tree/c30662c7a4e771ab29d6083e98fe70e45455b921/app/settings/registry-new - this is where the current 3 options are implemented. All they do is provide a template for the Address, there is no such thing as logic applied to registries.

Maybe, for a mid-term solution, a simple cronjob would do which would simply update the registry with a new token every 10hrs or so. It could just use the existing API, no need to change anything within rancher.

[johnrengelman]

@0ff So I've given this a try the last couple of days this doesn't seem to work for me. I enter all the login information into Rancher, but it fails to pull a container image from ECR.

[johnrengelman]

Even manually running the login command doesn't seem to do the trick.

[jaygorrell]

+1 for a solution to this.

[johnrengelman]

This is what I've gone with for a solution. https://github.com/objectpartners/rancher-ecr-credentials
It's available on Docker Hub under objectpartners/rancher-ecr-credentials

[bbedwell]

+1

[jaygorrell]

@ibuildthecloud Is this feature currently on the roadmap by chance? I seem to recall it being mentioned in one of the meetups before ECR was released to the public.

[markmcnaughton]

I also would like to add my vote for this feature. We are currently using ECR and really want to go with Rancher. I hope the Rancher team can implement this feature otherwise we need to have another look at ECR alternatives. We didn't have great success with the ECR competitors though.

[ajagnanan]

+1

[ibuildthecloud]

Okay, I figured out what it will take to do this, let me see if there's a way we can get it in the schedule.

[jaygorrell]

Awesome, that's great to hear. I was the one that asked in yesterday's meetup and planned to get you some information today. The container above from @johnrengelman is working pretty well for me as a stop-gap at least.

[devlinrcg]

+1 on this. Like @markmcnaughton said, hoping to use AWS ECR with Rancher as an alternative.

[markmcnaughton]

Thank you Darren! I'm really blown away with Rancher and the work you guys
are doing.

On Fri, Feb 26, 2016 at 6:42 PM, Darren Shepherd [email protected]
wrote:

Okay, I figured out what it will take to do this, let me see if there's a
way we can get it in the schedule.

—
Reply to this email directly or view it on GitHub
https://github.com/rancher/rancher/issues/3126#issuecomment-189364372.

[adamkdean]

Would really like to see this as we're wanting to use ECR!

[kjones1876]

+1

[blandes]

+1
This gentlemen came up with this solution, maybe a possible community catalog candidate?
https://github.com/objectpartners/rancher-ecr-credentials

[jaygorrell]

Hmm, rancher-ecr-credentials seems to be broken now for me. I see there were a couple of commits to it yesterday.

@ibuildthecloud Do you know if the official feature will make it in for 1.0?

[johnrengelman]

@jaygorrell What are you seeing broken? Can you file issues over on that project, please?

[jaygorrell]

Sure, done!
https://github.com/objectpartners/rancher-ecr-credentials/issues/4

Edit: @johnrengelman Already fixed the problem I was running into. :)

[artburkart]

Will the feature in this issue or #4250 include support for IAM roles with ECR? Or is that beyond the scope?

[danawoodman]

@ibuildthecloud any status update on this?

[luispabon]

Any news on this?

[weitzj]

• Use https://github.com/objectpartners/rancher-ecr-credentials
• Make sure to use version 1.0.1 and not 1.0.0
• When you register your registry in Rancher, do not put a 'https' in front of the URL

[luispabon]

I have set this up ^ to no avail. Rancher really needs native support for ECR as part of its wider AWS integration effort, instead of third party hacks.

[artburkart]

https://github.com/objectpartners/rancher-ecr-credentials has been working really well for me. I needed to tweak it, so it could leverage IAM roles, but beyond that, it's been pretty smooth

[danawoodman]

Is there any plan to adding this into Rancher natively? Any updates?

[luispabon]

We've stopped considering rancher over this issue. The workarounds proposed
are unreliable at best.

On Mon, 1 Aug 2016, 18:07 Dana Woodman, [email protected] wrote:

Is there any plan to adding this into Rancher natively? Any updates?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/3126#issuecomment-236642994,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGF8V_oQEHaYeZt5IH9mfmIXFXdVScEEks5qbifKgaJpZM4G6AlO
.

Kind regards,
Luis Pabon

[orlando]

This works great for me. Thanks @johnrengelman

[jfaissolle]

Works great for me also with latest ECR Credential Updater from the Catalog.

[goffinf]

Required for us too. ECR is our strategic registry, without it, Rancher isn't much use to us.

When is this feature request going to be scheduled, it was raised over a year ago and @ibuildthecloud already said he knows whats needed to sort it. Plus there is a catalog item for dealing with token refresh already.

What do we have to do to get this on the list sometime soon ?

[deniseschannon]

@goffinf Any particular reason why you can't use the catalog item?

[luispabon]

In our case, it felt like a third party hack, which it is, not a first
class citizen. Plus we couldn't make it work at all.

On Tue, 20 Dec 2016, 03:54 Denise, [e-mail] wrote:

@goffinf https://github.com/goffinf Any particular reason why you can't
use the catalog item?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/3126#issuecomment-268148076,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGF8V63fVZbd9SVbmxHe1WexNc8ckHuLks5rJ1FYgaJpZM4G6AlO
.

>

Kind regards,
Luis Pabon

[goffinf]

@deniseschannon Configuring ECR as a custom registry does work (in the sense that once set up if is possible to pull images as normal), the thing that doesn't is the ability to update the time limited authentication token which means that every 12 (or less) you need to manually update the password field in the UI for that registry. Clearly that is not a sustainable solution. See this issue that refers : https://github.com/rancher/rancher/issues/7088

Fraser

[jfaissolle]

@goffinf I have been using the ecr-updater provided in the catalog without problem in production for several months. I have added the ecr url in the Rancher registries and given a Role able to access it to my host instances.

[goffinf]

@jfaissolle Thx, Not sure what paltform you are on (we're on AWS). I am waiting for our Cloud SysOps guys to set up the IAM instance profile so I can switch to this rather than using AWS as the user and the password from the generated token.

What do you put in the user field ? .. the arn for the role ?

[jfaissolle]

@goffinf We're also on AWS. In the registry config, I put AWS as username and a random email (our sysops email) in the email field. I do not think it is really important whatever the credentials you use (Role or Access Key).
In the ecr-updater config, I filled only the AWS region which is a very important parameter.
We have given the AmazonEC2ContainerRegistryReadOnly poliy to the IAM Role. Note that the role has to be given to the host instances, not the rancher server instances.
I hope this helps.

[orlando]

@luispabon it works pretty much out of the box, just make sure you are doing everything that the README.md says.

@goffinf you can use https://github.com/objectpartners/rancher-ecr-credentials. It worked for us without problems.

@luispabon

it felt like a third party hack

I mean, that's how Amazon ECR works (https://github.com/aws/amazon-ecs-agent/blob/v1.7.0/agent/engine/dockerauth/ecr.go#L49). In this case, you are just running a container to do that, instead of the scheduler do it for you.

[Dmitry1987]

@orlando did you use "rancher-ecr-credentials" with Cattle environment? I'm trying to make K8s env work, and thinking more about putting on each host worker node a cron entry with "aws ecr get-login --region us-west-2 | bash" that will run each N hours. What do you guys think?

[orlando]

@Dmitry1987 yeah, with the Cattle environment.

That should work, I'll use aws ecr get-login --region us-west-2 | xargs command instead

[Dmitry1987]

@orlando that is SO weird, i have this cronie that works well and I can "docker pull" from cli with no problem, any needed image from ECR. But Rancher when I deploy a new pod in its K8s environment, creates pod, and gets stuck on authorization stage vs ECR to pull the image! Then I ssh manually to that host where Rancher struggles to pull the needed image, do manually "docker pull ..." and it works, pod starts happily as soon as I complete the pull image :D , I don't get it ...
The instance itself has needed IAM role, so cron job always refreshes credentials and runs the "docker login" every time. So docker daemon itself is authorized to pull. I also see the needed credential is saved in '~/.docker/config.json' on the machine... how comes Rancher gets to "authorization needed, Back-off pulling image" status? totally weird

[Dmitry1987]

@ibuildthecloud I'm using rancher 1.2.1, what might be the issue to pull the image, while docker daemon on the worker node is already authorized to do so? does Rancher pulls image with golang binaries and needs separate authorization not related to docker daemon itself?

[goffinf]

@orlando @jfaissolle I'm pretty sure that I did follow the README instruction accurately. As you mentioned for the credentials helper the region is key, I used eu-west-1. For the custom registry config, I used AWS as the user and none as email (per its docs) ...

Address: 123456789012.dkr.ecr.eu-west-1.amazonaws.com
Email: none
Username: AWS
Password: the currently active pwd

I know the registry is configured ok because while the password is active, I am able to successfully launch stacks that pull images from ECR.

I also use the AWS credentials helper so don't need to run a cron job to refresh the local credentials, see https://aws.amazon.com/blogs/compute/authenticating-amazon-ecr-repositories-for-docker-cli-with-credential-helper/ and the links above.

The credentials I am using are associated with policies that have all privileges (the error that I see is not that the user doesn't have sufficient privilege but that the credentials have expired), although when I get back to work tomorrow I intend to switch to using an IAM instance profile to see if that changes anything.

However, like @Dmitry1987 , I have found that everything works fine from the command-line on the host itself. Since I config the AWS ECR credentials helper as part of the AMI build, I don't need to run a cron job to refresh the password as others have suggested, that's all taken care of transparently. I can successfully pull and push images to/from ECR with no problems whatsoever. It's only the integration with Rancher that fails. That would appear to be confirmed by the logs for the ecr-updater container (which shows that it does attempt to send the request, but for some reason it fails) ...

16/12/2016 18:14:22Updating ECR Credentials
16/12/2016 18:16:23Error updating ECR, RequestError: send request failed
16/12/2016 18:16:23caused by: Post https://ecr.eu-west-1.amazonaws.com/: dial tcp 54.239.33.135:443: i/o timeout

[Dmitry1987]

@goffinf I replied in that other ticket, but just now realized that your log about connection issue is from inside "objectpartners/rancher-ecr-credentials" container itself.. I recognize the "Updating ECR Credentials" line :) . Did you use latest version of it? because it worked for me today, in us-west-2.
you might have missed the fact you MUST provide it with IAM user access-key/secret-key when you run that container?
can you try not to launch it from catalog, but manually on your Rancher master (or any node that has access to master API) with similar command that I used?

I mean, don't skip those 3 env vars in docker run command: AWS_REGION=us-west-2 -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY

the keys should be IAM user, that you gave him permission to registry (i guess you used IAM role on all instances in hope that it will allow the needed access). it seems like those things (both rancher and helper tool) ignore the fact IAM Role associated with instance they run on.
that's just my guess, so can't help much further. but pretty sure that if you try the same steps i did, it will work, because we're in same exact scenario, just different AWS region.

Cheers :)

[Dmitry1987]

@for-all-others who stumble upon this from search:
i used this, and it worked:

docker run -d -e AWS_REGION=us-west-2 -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e CATTLE_URL=http://${RANCHER_URL}:8080 -e CATTLE_ACCESS_KEY=$ENV_API_ACCESS_KEY -e CATTLE_SECRET_KEY=$ENV_API_SECRET_KEY objectpartners/rancher-ecr-credentials:latest

1) ENV_API_SECRET_KEY and ENV_API_ACCESS_KEY are Rancher API keys that you generate in "web ui -> API -> advanced". not global API key of Rancher.
2) AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are IAM User keys (user needs ECR access policy of course)
3) CATTLE_URL rancher master URL and port, check with telnet or something that access enabled.

this was solution that worked for me,
hope it helps someone 👍

[luispabon]

This discussion proves the point I made earlier. It's a hack, not an
enterprise ready solution.

On Mon, 2 Jan 2017, 20:47 Dmitry, [e-mail] wrote:

@for-all-others who stumble upon this from search:
i used this, and it worked:

docker run -d -e AWS_REGION=us-west-2 -e
AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e
AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e CATTLE_URL=http://${RANCHER_URL}:8080
-e CATTLE_ACCESS_KEY=$ENV_API_ACCESS_KEY -e
CATTLE_SECRET_KEY=$ENV_API_SECRET_KEY
objectpartners/rancher-ecr-credentials:latest

1. ENV_API_SECRET_KEY and ENV_API_ACCESS_KEY are Rancher API keys that
   you generate in "web ui -> API -> advanced". not global API key of Rancher.
2. AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are IAM User keys
   (user needs ECR access policy of course)
3. CATTLE_URL rancher master URL and port, check with telnet or
   something that access enabled.

this was solution that worked for me,
hope it helps someone 👍

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/3126#issuecomment-270018629,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGF8V0e_hiIPdiH0mjIlgBcdfS4AnHJ5ks5rOWJJgaJpZM4G6AlO
.

>

Kind regards,
Luis Pabon

[tommy-donorschoose]

@Dmitry1987 were you able to make this work with a Rancher K8s env ?

docker run -d -e AWS_REGION=us-west-2 -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -e CATTLE_URL=http://${RANCHER_URL}:8080 -e CATTLE_ACCESS_KEY=$ENV_API_ACCESS_KEY -e CATTLE_SECRET_KEY=$ENV_API_SECRET_KEY objectpartners/rancher-ecr-credentials:latest

ENV_API_SECRET_KEY and ENV_API_ACCESS_KEY are Rancher API keys that you generate in "web ui -> API -> advanced". not global API key of Rancher.
AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID are IAM User keys (user needs ECR access policy of course)
CATTLE_URL rancher master URL and port, check with telnet or something that access enabled.

& if so, are you running the above .. on your rancher server or your rancher hosts? I see also that you said that Roles are not working?

I see that rancher-ecr-credentials is available in the catalog for a cattle environment but not for K8s - and that the ENVs that seem to correspond to the Rancher registry are namespaced 'cattle'

Maybe this won't work with k8s?

[superseb]

This should now be working as described in the docs: http://rancher.com/docs/rancher/v1.6/en/environments/registries/#using-amazons-ecr-registry

@tommy-donorschoose and everyone else, please create a new issue if you experience any issue following the docs