Go: x/vgo: not working with GitHub Enterprise

[richardwilkes]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rich/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rich/code/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s9/9s3fk2mx5f936dkcp66ltwsc0000gn/T/go-build062786983=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

What did you do?

Attempted to build some code that had dependencies on one or more packages that come from our internal GitHub repo as well as from the public github.com using the experimental vgo tool. As this refers to an internal repo (and that is the basis of the problem), there is no way for me to give you code that reproduces the problem that you can actually try. Suffice to say, though, that even the simplest import causes the issue.

What did you expect to see?

I was hoping that GitHub Enterprise would work as well as normal GitHub.

What did you see instead?

vgo: resolving import "jaxf-github.fanatics.corp/forge/furnace"
findRepo: XML syntax error on line 25: unescaped < inside quoted string
vgo: import "jaxf-github.fanatics.corp/forge/ratelimiter" ->
import "jaxf-github.fanatics.corp/forge/furnace": unknown VCS, Repo: git,
https://jaxf-github.fanatics.corp/forge/furnace.git

[ALTree]

go get has the same issue: #17898

[richardwilkes]

Yes, I just realized that as well. This is still workable when using the normal go build, though, because we can (and do) manually checkout our internal repos into our $GOPATH/src tree. That doesn't seem to be an option with vgo, unless I've missed something.

[rsc]

I don't know much about GitHub Enterprise. Does it have similar APIs to public GitHub? It is possible this could work better with vgo than old go get. If the tag said something like githubenterprise to tell us it was that kind of server, we could use those APIs.

[richardwilkes]

Can you give me an example request? I'm happy to make it against something in our repo and give you the resulting output.

[mikkeloscar]

The APIs are the same (except for the base path) but vgo would have to know that something like https://github.mycompany.com is a Github Enterprise installation.

It would be great to think about this case from the beginning!

I opened a similar issue when dep was released: https://github.com/golang/dep/issues/174 but this is unfortunately still an issue preventing us from adopting dep.

[amnonbc]

This is an issue which has caused me a fair amount of pain.

See https://go-review.googlesource.com/c/go/+/37176 for a simple fix for the go get case in go-1.10

[sdboyer]

FWIW, i have been collecting a group of company GHE customers to try to influence GitHub to actually fix this problem. (it exists in GitHub itself, as well, but it's masked by having static rules that cover it)

[amnonbc]

Hi @sdboyer
I work for a GHE customer, and raised the issue a couple of months ago with them.
This is what they told me:

"[Thaddeus Cortez] and his team will work with Engineering to establish contact with Google's Go team to help them provide a solution on their end.
We've discussed various ways of fixing this on GitHub's side, but none of the potential solutions are ideal. The meta tag approach comes with the problem of information leakage which we consider a serious security risk. It's a trade-off we are not willing to make. And, a toggleable option in the web UI would require us to turn our static error pages into dynamic ones which would come with several different issues, most notably performance and UX implications as well as added complexity in terms of code maintenance.

We really think, the Go team needs to integrate support for other sources like GitHub Enterprise (or one of our competitor's products), and not just GitHub.com."

[cjs]

Does vgo have the same reliance on the meta tag returned by the ?go-get=1 query string that go get has?

[jameschensmith]

@amnonbc & @sdboyer,
I work for a GHE customer as well, and this was their response to us:

The "go-import" meta tag is used for installing Go applications from a URL, and is indeed supported on GitHub Enterprise, there is a caveat however:
 
To avoid leaking information about repositories when the appliance is in private mode, GitHub Enterprise doesn't return the meta tag unless you're authenticated. The ideal solution is for Go to be modified to allow authentication options, but for now, you can work around this by adding .git to your repository paths. For example:
import "[ghe-hostname]/bevns/HelloWorld.git"

Or for go get:
go get -v [ghe-hostname]/bevns/HelloWorld.git

Alternatively, another workaround is to manually clone the repository into your $GOPATH and then run go-install. For example:

git clone https://[ghe-hostname]/bevns/HelloWorld $GOPATH/src/[ghe-hostname]/bevns/HelloWorld
go install [ghe-hostname]/bevns/HelloWorld

Some other customers using Go have implemented a workaround where Go import statements refer to an internal server whose sole purpose is to direct go get to the proper repository URLs. The proxy server would respond to both HTTP and HTTPS and can return a meta tag that will point to your GitHub Enterprise instance.

For example, you could have an import statement of the form import my.ghe.domain/user/project, which should result in the following HTTP request:
https://my.ghe.domain/user/project?go-get=1

This should then return a meta tag that looks something like this:
<meta name="go-import" content="my.ghe.domain git https://my.ghe.domain/user/project.git">

That should then result in the correct repository being cloned.

They also added this:

Please also keep in mind that this proxy server would need to be a standalone server and should not be configured on your GitHub Enterprise server.
 
The best solution will come when the Go team accepts and merges https://github.com/golang/go/issues/17898. That will detect the headers sent from both the GitHub Enterprise appliance and GitHub.com, and use that for verification that a Git repository is being requested.
 
I would also like to mention that some alternative Go package managers such as https://glide.sh/ work with the Go toolchain and support private Git servers and repositories.

[amnonbc]

@james-r-smith yes we use the former hack as a workaround at the moment.
The problem is that it forces the users of your library to modify their import paths in a way that is not obvious (i.e. appending .git). This adds unnecessary friction, especially to new users.

[jameschensmith]

@amnonbc, I wholeheartedly agree.

[cjs]

With the GitHub Enterprise 2.13 release, on-premises GitHub Enterprise instances will now send the <meta> tags expected by go get for private instances and all repositories.

For instances that are in private mode, you will still need to manage authentication for the git pull phase of go get, but there are easy workarounds such as putting an authentication token in your .netrc or the insteadOf technique to force SSH as mentioned above.

x/vgo, from what I can tell currently only supports GitHub.com, but that's to be expected considering it is still in proposal status.

[rsc]

There's a header we can look for in the go-get response to understand that this is a GitHub Enterprise server and can be accessed using the usual GitHub code. We should do that.

[gopherbot]

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

[gopherbot]

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access

[coolaj86]

I see that this is marked as closed, but I've been unable to find an example of the solution.

I have private repos that will require API keys in deployment environments and I'm trying to figure out where I tell go.mod "when you see this import, construct this url with this token instead of whatever you would normally do".

So in the case of

go get awesome.io/go/sauce

The end result should be

git clone https://token:[email protected]/awesome/go-sauce.git

Can anyone give me an update on the best practice way to accomplish this? Something that doesn't require modifying an existing server would be preferred.

[dmitris]

@coolaj86 - could you use SSH keys instead of tokens (that's how I believe the authentication to GitHub Enterprise server normally works)? If so, you can add to your (and everyone's in your org 😭 )~/.gitconfig file:

[url "ssh://[email protected]"]
    insteadOf = https://git.awesome.com

if (with a bit of luck such as GHE >= 2.13 - https://enterprise.github.com/releases/2.13.0/notes) your Git server replies to requests from the go tool to https://git.awesome.com/awesome/go-sauce?go-get=1 with a meta tag <meta name="go-import" content="git.awesome.com/awesome/go-sauce git https://git.awesome.com/awesome/go-sauce.git", then the go tool will end up cloning or updating from ssh://[email protected]/awesome/go-sauce.git.

I really wish https://github.com/golang/go/issues/17898 and @matope's https://go-review.googlesource.com/c/go/+/37176 could be revisited and finished - so that it wouldn't be necessary for all users to "hack" their .gitconfig files with the insteadOf mantras before they can use go get and build with Go modules.

Now suppose you don't have yet such a wonderful server support (GHE < 2.13 etc.) - you can still make the go tol fetch your repos and use them in modules. You would need to add a replace section to go.mod where for each "real" package (module) listed in the require section that you want to replace:

require (
      git.awesome.com/awesome/go-sauce v1.0.0
)
replace (
     git.awesome.com/awesome/go-sauce => git.awesome.com/awesome/go-sauce.git v1.0.0
     // other module rewrites
)

• notice the .git suffix in the replacemend module - I believe it is necessary (to prevent go from sending ?go-get=1 requests that the server doesn't know how to handle).
• the insteadOf settings in ~/.gitconfig are still needed as above so that git would use the ssh keys for authentication

The replace pattern can also be applied to use custom git mirrors (let's assume opensource.git.awesome.com mirrors https://github.com/golang/net into its golang/net repo) with:

replace (
   golang.org/x/net => opensource.git.awesome.com/golang/net.git v0.0.0-20180911220305-26e67e76b6c3
)

• once again, notice the .git suffix in the replacement module which be not necessary with GHE >= 2.13 but I believe in all cases speeds up the checkouts/updates by avoiding an extra roundtrip with the ?go-git=1 request.

[dmitris]

another option is to use personal access tokens with a credential helper - then you don't need to put insteadOf in ~/.gitconfig -see https://help.github.com/enterprise/2.14/user/articles/creating-a-personal-access-token-for-the-command-line/ , https://help.github.com/enterprise/2.14/user/articles/updating-credentials-from-the-osx-keychain/ and https://help.github.com/enterprise/2.14/user/articles/caching-your-github-password-in-git/ (the last page has tabs for Mac, Windows, and Linux / All)

@coolaj86 - I think this would fit your use case:

• add the appropriate credential.helper setting in ~/.gitconfig as per https://help.github.com/enterprise/2.14/user/articles/caching-your-github-password-in-git/#platform-mac.
• follow the instructions in https://help.github.com/enterprise/2.14/user/articles/creating-a-personal-access-token-for-the-command-line/#using-a-token-on-the-command-line - enter your Username and the access token as "Password". They should be saved in the credential helper (such as osxkeychain) - verify that you can do git clone https://git.awesome.com/awesome/go-sauce.git without getting Username / Password prompt
• if awesome.io server returns the meta tag (<meta name="go-import" content="git.awesome.com/awesome/go-sauce git https://git.awesome.com/awesome/go-sauce.git"), you should be able to run go get awesome.io/go/sauce and get https://git.awesome.com/awesome/go-sauce.git checked out

Would be curious if that works for you.

[yulrizka]

Since I'm using jenkins which stored the username & password/access-token, my workaround is using a GIT_ASKPASS wrapper script

git-askpass-wrapper.sh

#! /bin/sh
case "$1" in
  Username*)
    echo $GIT_USERNAME
    ;;
  Password*)
    echo $GIT_PASSWORD
    ;;
  *)
    echo "unknown argument: '$1'"
esac

$ export GIT_USERNAME=some-user
$ export GIT_PASSWORD=some-password
$ GIT_ASKPASS=the/path/to/git-askpass-wrapper.sh go [command here]

It works because when git require credentials, it will call the wrapper script. You can modified the script to your need (for example, if you have different creds for different repo).