Dep: Http authentication for private repositories

[zevdg]

Spinning this item out of #174

I'm specifically looking for something along the lines of what composer does. Go get has always been painful to work with for private repositories for this reason. You generally have to manually clone the project to your gopath to get around this issue. Being able to configure dep with a personal access token and/or a git-like credential manager would make working with private repositories much easier.

Furthermore, Github Enterprise at my org is configured to require authentication for all pages, even "public" repositories. I'm not even sure if it can be configured otherwise. This makes go get fundamentally incompatible with our GHE instance without this feature or some kind of intermediate vanity url that doesn't require authentication.

[zevdg]

Oh, there are also some sorta sketchy workarounds for this that involve configuring git to rewrite http urls into ssh urls, but we can definitely improve on this.

[zacps]

I'll take a look at this on the weekend. The ways I can think of are:

• Replace with SSH
• Prompt (using requires letting git prompt from the terminal)
• Configuration (either through git config or our own)
• Require SSH for private repos (looks like other tools, like _cargo_, do this)

The composer approach (username and password in the URL) is depreciated (RFC 3986) so we should probably avoid this.

[sdboyer]

Thanks for spinning this out, and sorry it took me so long to respond here.

Almost all of the substantive change here is going to be in gps. dep's just going to be responsible for picking the information up, then injecting it into gps appropriately. Some things I can say up front about @zacps' suggested possible approaches:

Replace with SSH

In a sense, we do this already. When doing "source deduction" on an import root, we typically produce a set of multiple schemes that could be used for an import path. In gps parlance, these are known as maybeSources. These are attempted, in order. With github.com/sdboyer/gps, for example, we would try https, then ssh, then git, then http.

Prompt (using requires letting git prompt from the terminal)

This is a strong maybe. I opened sdboyer/gps#28 nearly a year ago after someone raised the point about wanting to be able to make SSH work without an agent.

The problem, as that issue notes, is that prompting the user is somewhat at odds with doing all this work in parallel (which we very, very much want to do for performance reasons).

Configuration (either through git config or our own)

It would be good to rely on standard configuration from the underlying tools as much as possible. Barring that, though, I'd say that configuration is a possible option...but I worry we'd pretty quickly find ourselves in a discussion about creating a dotfile for dep. I'd rather we try to avoid going there just yet.

Require SSH for private repos (looks like other tools, like cargo, do this)

As above, we sorta end up doing this automatically. For us, the problem is that source discovery operates so much by magical incantation string inference that we can't really know ahead of time what's private and what isn't.

The composer approach (username and password in the URL) is depreciated (RFC 3986) so we should probably avoid this.

Agreed, I would prefer to just not even have this on the table at all.

[mikijov]

Prompt (using requires letting git prompt from the terminal)

This is a strong maybe. I opened sdboyer/gps#28 nearly a year ago after someone raised the point about wanting to be able to make SSH work without an agent.

The problem, as that issue notes, is that prompting the user is somewhat at odds with doing all this work in parallel (which we very, very much want to do for performance reasons).

How about using authentication callback. When solver requires credentials, it would call the callback (if one is registered). dep or another front end can then return credendials from any available source. Moreover, text mode driver can ensure to serialize text prompts, and ask the user for one credential at a time. This would require calling solve() in async way, but that should be easy I think.

[sdboyer]

a callback is interesting - it's in keeping with a number of other tools that I'm aware of.

it might have to take a few different forms, though. e.g., ssh creds, https token vs user/pass, etc. but it would open up integration with system keychains, which is what we need.

let's spec it out.

[matjam]

so, git handles this. I've added instructions to the FAQ on how to configure Authentication with git so that it will work with dep.

[sdboyer]

closing this now, as the solution described in the docs is good enough for now

[mattnworb]

Regarding the FAQ entry and telling git how to authenticate to these private repositories:

This only works if dep is able to deduce that the package is a git repo, i.e. if it has .git in the name somewhere, right?

If I have

import "ghe.example.com/org/repo"

then, as best as I can tell, dep goes down the path of treating this path as a URL and fetching it with https://. i.e. the check at https://github.com/golang/dep/blob/83789e236d7ff64c82ee8392005455fc1ec1983b/internal/gps/deduce.go#L604-L605 fails and deduceRootPath proceeds to httpMetadataDeducer.deduce(..).

[sdboyer]

@mattnworb so, there's a few phases here. the httpMetadataDeducer.deduce(..) path is fetching the same HTML metadata over HTTP that go get relies on in order to serve vanity URLs - that all happens before any VCS backend is invoked.

unfortunately, this doesn't work very well with github enterprise (nor would it with github itself, actually - github only works because it's one of the well-known cases handled in deduceKnownPaths()) because GHE _only_ correctly reports this metadata for the package at the root of the repository.

by far the easiest solution here will be to add a .git suffix/infix to the appropriate place in your import paths. (unfortunately, source can't currently solve this for you, for the reasons explained in #860)

[mattnworb]

Thanks @sdboyer for the fast response. Is there a listing somewhere of all the ways to make dep think that a package can be accessed with git (if there are other ways besides adding .git to the path)?

[sdboyer]

@mattnworb yep, we follow the rules set down by go get. the only addition we make is special handling for gopkg.in, but that should never be noticeable, as behavior is identical to what gopkg.in establishes via its http metadata.

[juliendelannoy]

Hello, I had the same problem, and i could solved i by a little "hack":

git config --global url."https://[email protected]".insteadOf "https://github.com"

where $$GH_TOKEN is a secret env variable containing my personnal access token