Dependabot-core: Support for AWS ECR

[greysteil]

_From @edkellena on February 1, 2018 9:21_

Dependabot is complaining that it can't authenticate to our ECR docker registry and we should add the login details to the dashboard. However AWS works differently and you need to run their CLI to generate your login creds (which are time-limited).
Something like ${aws ecr get-login --no-include-email --region $REGION}

An interesting problem for you to solve. We could add IAM credentials into the dashboard to allow you to do this??

_Copied from original issue: dependabot/feedback#81_

[greysteil]

Sounds sensible. I’m on holiday but will take a proper look when I’m back.

[fernandocarletti]

Hello. Is there any love for this issue?

[greysteil]

There is and in fact @nesl247 tweeted about it a few hours ago.

I'm keen on this but need to do some research on how it would work.

[nesl247]

@fernandocarletti and I had just been discussing it prior to this (both work for the same company).

I wonder if a possible quick solution would be to allow the registry configuration in dependabot to be accessible via an API. We could then run a cronjob that calls that API with updated credentials every 11 hours or whatever.

That is until a more permanent solution could come about. Is there an API for updating it?

[nesl247]

@greysteil I've configured our dependabot with a username and password that should work with AWS, however, it still doesn't appear to be finding any updates.

How does dependabot check to see what newer tags there are?

[greysteil]

@nesl247 I'm crunching on getting Terraform support out this evening, but am planning to focus on Docker tomorrow. Barring a sea of bugs I'll get this sorted then 🙂

[nesl247]

So I'm thinking, and hoping I'm wrong, but it may be that AWS does not fully support the registry API. This is hard for me to debug since I don't know how to. I have written a small script that I can execute that will actually connect to api.dependabot.com and provide a docker registry entry with a username and password that should work for pulling.

However, dependabot is not finding any new tags for one of our applications. In my case, the tags are short commit references like 62c0ade.

Sounds good @greysteil

[greysteil]

Ah, it may be that Dependabot just isn't picking those up because they're not version-like, and it wouldn't have a way to order it.

Have you had a look at dependabot-script, to step through the logic there? It uses dependabot-core in basically the same way that the main Dependabot updater does, and is what I use to debug these issues if/when I need to step through the logic.

[nesl247]

I wasn't aware of dependabot-script, good to know.

I'm not sure how we should do our tags then. I was so focused on getting dependabot connected to AWS I wasn't thinking about that. In my head, it was like "well, git knows this is newer than the last..".

[nesl247]

We're switching to tagged releases. We were moving very fast, and skipped thinking about tagging for dependency docker images for us. So we'll keep this ticket back on track which is the support for ECR.

[greysteil]

I was hoping to get to this yesterday but spent the day on a couple of Python issues. It’s top of my list for the next few days.

[greysteil]

This is done! Sorry it took a while. That changes are in https://github.com/dependabot/dependabot-core/commit/e92672c2cb10a59e7646583bff60827222ccf099, and to use it you'll need to give Dependabot API keys for you AWS account (scoped using IAM to have read access only to ECR, of course).

I'm going to update the front-end to make it clearer how to add credentials.

Thanks for the feature request!

[nesl247]

Hey that's awesome! We'll give it a shot.

[greysteil]

Frontend is added now (well, it's just some additional comments in the description of the fields under "config variables"). Happy to talk you through if not clear.

[nesl247]

I saw that after I commented and had hoped I got the comment changed in time, guess not 😄 .

[nesl247]

I tried it out, unfortunately it didn't work. I confirmed the keys work through the AWS cli though.

[greysteil]

Looking into it!

[greysteil]

Should be fixed in https://github.com/dependabot/dependabot-core/commit/f5445565c1e48abace204575c205047e2ee8c490. About to re-run.

[nesl247]

Unfortunately we have a new issue.

build(deps): bump docker-php from 7.2-0.3 to 7.2-9658143 in /docker/cart-sync

This should have used 7.2-0.3.1. Instead, it's changed to the commit hashes.

[nesl247]

Thanks for the ECR support, that part is great. I'm not sure if my above comment should be made into a new issue or not. I'm doing a new response instead of editng since I know how quickly you respond sometimes 😄

[greysteil]

Huh, that one is tricky! Did we just get unlucky here and have a commit sha version that happened to start with 7 digits (no letters)? If so I think the solution might be asking dependabot to ignore that specific version - I’ll make sure that’s possible for Docker now.

[nesl247]

It's quite possible. Most of our hashes are 7 digits though, so unless this is permanently fixed, we're back to not using dependabot for Docker, or we stop using tags, which we started because of our previous discussion.

[greysteil]

That's fine - I can put something in that stops it from switching from more traditional looking versions to 7 digit numbers. I'll do that now.

[nesl247]

Sorry, I think I sounded a bit snippy, not my intention.

I think as long as the current tag is a version, the logic should be to not switch to commit hashes, even if it is newer. I would think the only time to switch to commit hashes if it it is currently using latest, or it already uses a hash.

[greysteil]

No worries - I figured.

OK, the issue here is that 9658143 looks a lot like a (very high) version. Even worse, Dependabot is picking up the 7.2 part and thinking that's a version. Getting Dependabot to figure out what's going on here is just too tricky.

What I can do is https://github.com/dependabot/dependabot-core/pull/690, which I hope will work for you. Basically

• I've introduced a special case for pruning out suffices that look like commit SHAs
• I've prefixed all of your tags with php, so that the first part no longer looks like a version.

If you start pushing tags that are prefixed with php7.1 and php7.2, and using those tags in your Dockerfiles, then Dependabot will realise what's going on and do the right thing. Will that work? (You won't need to edit the told tags - once you start using the php... prefix in your Dockerfiles Dependabot will scope its search to just tags with that prefix.)

[greysteil]

Oh, hang on, I might be able to do one better. Give me a moment...

[greysteil]

Yep, have made it work without the need to change the prefix to be non-numeric. Deploying now.

[nesl247]

It's working beautifully. Thank you so much for the fast turnaround.

[greysteil]

No worries - all helps to make Dependabot better! 🎉

[nesl247]

One quick question, when using the api for updating private deps, do we have to specify docker-php for example as the dependency name, or the full namespace of the docker image? For example, something.something.aws.amazon.com/docker-php?

[greysteil]

Just docker-php (or whatever the name in the PR title you're seeing is bump <name> from ...). 🙂

[nesl247]

Awesome, thanks.