Hub: Unknown CA Authority for Github Enterprise

[lztst]

Hi,

I have set the local hub.host but when I try to do a 'hub pull-request', I am getting this:

Post https:myhost/mypath: x509: certificate signed by unknown authority

I tried adding the certificate manually to global git setup, like:

git config --global --add http.sslCert path-to-pem
git config --global --add http.sslVerify false

They don't seem to work.
My regular git works fine.

What else can I try?

Thanks.

[mislav]

Hi, thanks for reporting.

Your GitHub Enterprise host seems to be using a self-signed certificate that you need to explicitly approve somehow. Doing the steps you tried above might solve it for git push/pull, but hub doesn't communicate with GitHub over any of the git protocols. Instead, it does regular HTTPS requests to GitHub API. These HTTPS requests are subject to certificate verification and that's why your unverified host fails.

If you tell us which OS you're on, which hub version do you use, and how did you obtain hub (precompiled binary or compiled yourself?), we could try help you investigate how to add custom root certificates.

@jingweno Do you have an idea where Go net/http draws it root certificates from? Is it bound to system OpenSSL? On OS X, that would mean that you could simply import a new root certificate to your Keychain and configure it so it's always trusted by default.

[lztst]

Hello,

Thanks a lot for the help.

I am on MacOS, and I got hub from the binaries downloads section here.

Also, just mention that I do not need to any of that with regular Mac git, without doing the extra config mentioned above. All I did is setup the Enterprise github account to include my certificate.
I'd say that hub should somehow be able to do the same, if it is to work with Enterprise github, unsure though.

Thanks again.

[mislav]

Going back to git transfer protocol, this piece of git config might be dangerous:

# not recommended
git config --global --add http.sslVerify false

You're turning off SSL certificate verification for all git operations across all hosts. That means when you try to pull git changes from any host, including GitHub.com, the operation will be performed insecurely (that is, the identity of the host will never be verified). I would recommend removing that configuration, and first exploring how you can set http.sslCert to point to a custom certificate that makes it possible for you to git clone/pull/push to your Enterprise instance. Then, when that's sorted out, let's figure out how we can enable the same for hub operations (which depend on the GitHub HTTPS API).

[lztst]

Yes, I guessed that's what that would be doing.
I tried to set that option if it had an effect on hub, but it seems it didn't.

Regular git was working for me with custom certificate on github Enterprise.
I never needed the http.sslCert option for that either, I tried to set it if it solved the hub issue, but it did not.

[mislav]

I tried to set that option if it had an effect on hub, but it seems it didn't.

Almost none of the git options have any effect on hub, because they are separate programs and communicate with GitHub differently. I understand how this is confusing.

Regular git was working for me with custom certificate on github Enterprise. I never needed the http.sslCert option for that either

You never needed that option because you disabled cert verification with http.sslVerify false. In my above comment I encouraged you to disable that configuration (remove it from ~/.gitconfig or set it back to true) and see if git pull/push still work for you. This is not related to hub yet, but it's an important first step to understand that we're at least able to configure the custom root cert for git protocol.

[lztst]

I never had those two options in regular git usage, and git always worked fine.
I added them to see if it had an effect on hub.

[owenthereal]

@mislav:

Do you have an idea where Go net/http draws it root certificates from? Is it bound to system OpenSSL? On OS X, that would mean that you could simply import a new root certificate to your Keychain and configure it so it's always trusted by default.

Go uses the root cert from the host if not specified. We always verify cert in current implementation. We could make it optional though by setting InsecureSkipVerify to false based on a environment variable or the http.sslVerify git config (nicer).

[mislav]

I never had those two options in regular git usage, and git always worked fine.

Ok, good to know.

When your company set up the GitHub Enterprise instance on the custom domain, did you ever have to set up a custom SSL certificate to access it in the web browser?

[logc]

I have the exact same issue, on Mac OSX 10.10, using hub installed via Macports

$ port installed | grep hub
hub @2.2.0_0 (active)

The certificate is signed by my company's root CA, and it is stored in the OS X Keychain; i.e. if I browse the site on a web browser I see a correct, green, verified connection. But I can't tell hub to look into the Keychain for the certificate. Or is there a way?

[mislav]

It seems that Go should look up certificates in Keychain on OS X. Which Go version is the Macports hub compiled with?

Unfortunately, if it doesn't work, I don't know what else you could try. Maybe we need a fix in hub's own source.

[logc]

It's go @1.4.2_0. Thanks for looking into this!

[mislav]

That is a very recent version of Go. I can't tell why a root cert isn't being discovered from OS X Keychain.

[christianmlong]

I'm having this same problem.

hub create
<asks for username and password>
Post https://<GH Enterprise URL>.com/api/v3/authorizations: x509: certificate signed by unknown authority

Go version
go1.4.2 darwin/amd64

hub version
git version 2.4.5
hub version 2.2.1-gca7939b

OS X 10.10.3
Gray lock icon in browser location bar.

2 factor is on for my regular Github account and for my Github Enterprise account.

I was able to use the gist tool to access this same instance of Github Enterprise.

I was able to use hub to create a repo on my regular Github account.

[mislav]

@christianmlong Do you remember where you installed the certificate for your Enterprise repo system-wide so you can access the site from your web browser and other programs?

[lethjakman]

Same issue, same versions as @christianmlong. I've installed the certificates to my keychan and I still get issues.

[stevehedrick]

I'm having the same issue. Based on the source of golang @ https://github.com/golang/go/blob/master/src/crypto/x509/root_darwin.go It looks like the only place that Go checks for certs is in the System Root Certificates keychain. It should be checking the System and the Login keychains for certs as well, but isn't.

It's not possible from the Keychain Access gui to add certificates to this keychain. I've tried playing with the command line security tool, but so far no luck.

[mislav]

@stevehedrick Awesome detective work. Do you think we could/should replicate this logic to look up the cert in Login keychains from hub ourselves? Or should we add hub configuration to specify the location of custom CA bundle on disk?

[mrkwjc]

Hi!
I have very similar problem on Windows XP and not Enterprise git:

Post https://api.github.com/authorizations: x509: certificate signed by unknown authority

I have:

git version 2.7.1.windows.2
hub version 2.2.3

Regular git works fine, however i had to configure it with:

git config --system http.sslcainfo "C:\Program Files\Git\mingw32\ssl\certs\ca-bundle.crt"

[bmabey]

@mislav FYI, the git-lfs project has solved a similar issue. I imagine the solution for hub would be similar: https://github.com/github/git-lfs/issues/308#issuecomment-197964308

[mislav]

Thanks for that reference! That's really useful.

This is a really interesting starting point for OS X, for example. Thanks for doing this hard work in git-lfs, @sinbad!

[sinbad]

No problem - you actually want to use this as a starting point: https://github.com/github/git-lfs/commit/cb889413872b9f94e172fe15fe4b005e84c33624#diff-9938c5ad789d4db49e660fe24b343282, the commit above was a WIP. HTH!

[skeet70]

Patch that makes go look for certs in system.keychain is landing in 1.8.

[tconroy]

Hi -- running into this issue at the moment, wondering what the solution is on the user's end? I'm using Hub 2.2.3 on OSX, installed via Homebrew.

I'm able to pull/push to our enterprise config over HTTPS and SSH. But if I try and open a PR with Hub, I get prompted for my login and password, then hit with an error:

Post https://github.<our domain>.com/api/v3/authorizations: x509: certificate signed by unknown authority

I'm able to log in to our site through the web just fine, and the green SSL lock appears in the nav bar. Unsure what to do here so guidance is appreciated. :)

[mislav]

@tconroy Unfortunately, there is no user workaround right now. We need to replicate the git-lfs functionality (linked above) into hub first.

[tconroy]

@mislav I see -- very unfortunate. :( Do you have any rough/approximate ETA for when this feature will be fixed? honestly the primary draw of hub for me is the ability to do tasks such as open PR's from the command line.

[seanmalloy]

Looks like most people have been trying this on OSX. I have also not been able to get this to work on RHEL 7.

Thanks.

[joedj]

I am having this same issue, and out GitHub Enterprise cert is signed by a real, trusted external root CA:

Post https://github.ourcompany.com/api/v3/authorizations: x509: certificate signed by unknown authority

The cert chain is:
Entrust Root Certificate Authority—G2 (System Roots keychain)
-- Entrust Certificate Authority ‐ L1K (login keychain)
-- Company GitHub Enterprise cert

~|⇒ hub --version
git version 2.6.4 (Apple Git-63)
hub version 2.2.3

[mislav]

@joedj That's strange. I would have hoped that Go discovers the certificates stored in your system or login keychains. In any case, the version that you're running is really old and I encourage you to upgrade to at least v2.2.9. It might not solve your certificate problem, but it fixes a lot of other bugs.

[joedj]

Thanks @mislav

Unfortunately I can confirm that v2.2.9 does not resolve this issue.

For reference, it looks like the relevant golang issue is https://github.com/golang/go/issues/14514 (which has been closed).

There is apparently a workaround for El Capitan users, which I have not tried yet:

1. Disable System Integrity Protection
2. Add the new CA to SystemRootCertificates.keychain

[blaster151]

I am on Windows 10 and am running into this same issue when attempting to connect to an enterprise GitHub account. (I notice that https is NOT enabled when I navigate there in the browser.)

[kannanekanath]

I get the same error "x509: certificate signed by unknown authority".

I use the git config --global --add http.sslVerify false option and the git command line works for me. Is it in anyway possible to get hub to skip the ssl verification too?

I understand that there is a security vulnerability here however given that git already has this feature we are only asking hub to do exactly what git is doing (and not weaken the security model).

[rage28]

Any updates on this issue?

[jsok]

Yes this is quite a blocker for GitHub Enterprise usage!

The most debugging I've been able to do so far is:

> export GODEBUG="x509roots=1"
> hub browse
crypto/x509: 11 certs have a trust policy
crypto/x509: ran security verify-cert 0 times
Get https://github.example.com/api/v3/user: x509: certificate signed by unknown authority

My corp CA is in my System Keychain and set to Always Trust.

Upgrading to hub v2.3.0 appears to have fixed this issue.

[rage28]

@jsok I was able to resolve this myself and am able to use hub with github enterprise. You will have to compile hub yourself with Golang with CGO disabled. You can follow the following steps:

• Install go with CGO disabled. If go is already installed you can verify this by checking the output of the command go env and search for 'CGO_ENABLED' flag. If it's "0" then disabled (obviously). You can install golang without CGO using brew command brew install go --without-cgo
• Clone hub and compile the standalone binary
• Import the certificate for your github enterprise into the "System" keychain
• Add your github enterprise URL into .gitconfig by executing git config --global --add hub.host my.example.org

You should be able to use hub with your github enterprise now :)

Update

As confirmed by @jsok upgrading to 2.3.0, importing the certificate to _System keychain_ and setting it to _Always Trust_ resolves the issue. The above mentioned method might not be needed and as @mislav said it's a just a workaround so use it only if you know what you are doing

[mislav]

@rage28 Thank you for sharing your workaround!

Changing a local installation of go is indeed a nuclear option that I wouldn't suggest to everyone unless they really know what they are doing, since that will affect their other go programs as well. I would much rather see us add code to hub that allows reading an extra certificate from a file or something.

[bezhermoso]

I'm also experiencing this issue working with Github Enterprise:

• Fails with same certificate error with hub pr ..., hub issue ..., hub ci-status, hub release
• hub pull-request works.

[wkoszek]

Same issue. GitHub Enterprise, valid cert, getting the warning on hub checkout <pr_url>

[egyptiankarim]

I can confirm that running the Homebrew-installed hub (2.5.0) on High Sierra will work against Enterprise instances using my enterprise CA when the _System_ Keychain is set to explicitly "Always Trust" that CA. That is, I am now able to run hub commands pointing at an Enterprise repository with no complaints about the certificates I'm using.

Disclaimer: In my case, it's an enterprise CA that I'm explicitly trusting, so I consider this slightly less risky than explicitly trusting a self-signed certificate, but everyone can/should do that calculation for themselves.

[forresthopkinsa]

This is still failing for me on MacOS Mojave 10.14.1 and Hub 2.6.0 using a private enterprise CA. The cert is trusted in my system keychain, but Hub continues to stop me at every turn with x509: certificate signed by unknown authority. Major showstopper.

[espoelstra]

@forresthopkinsa There was a bug a for a while in Go (not sure if it is fixed, https://github.com/golang/go/issues/24652) where it has issues accessing the certificates from System or SystemRoot, one workaround is to toggle the trust to Never Trusted and then back to Always Trusted, this triggers an update of the cert in the keychain which sometimes allows Go to correctly find/use them. Another workaround is to copy the certificate to the login keychain associated with your user and give it full trust there.

Open Keychain Access and navigate to the certificate in question under System and then drag and drop it onto login to initiate the copy, at which point you need to provide sudo capable credentials. Once that is done, double click the certificate to view it, then expand the Trust arrow and give it System-wide Always Trust and you should be able to talk using golang tools like hub without getting this error.

Let me know if this works for you, currently I'm on 10.13.6, but I imagine I'll find out soon enough when they push the Mojave upgrade to my machine.

[jijojv]

Ran into this on Mojave, brew upgrade go and hub resolved it.