Git: New curl behaviour favours http.sslCAInfo config (if set) when using SChannel backend

[shiftkey]

• [x] I was not able to find an open or closed issue matching what I'm seeing

This is a problem that GitHub Desktop encountered while upgrading to MinGit 2.17.1 (2), and seems to be related to how a recent version of curl will use config entries in new and unexpected ways.

Setup

Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

This is related to MinGit and bundling it in GitHub Desktop. For the purposes of this demo, I've extracted it to a folder on Desktop:

C:\Users\shiftkey\Desktop\MinGit-2.17.1.2-64-bit\cmd>.\git --version --build-option
git version 2.17.1.windows.2
cpu: x86_64
built from commit: a60968cf435951d9411fc0f980a2e362d5cccea2
sizeof-long: 4

Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.17134.81]

What options did you set as part of the installation? Or did you choose the defaults?

install-options.txt isn't created by MinGit

Any other interesting things about your environment that might be related to the issue you're seeing?

The setup for reproducing these seems to be related to these moving parts:

• A vanilla MinGit install has http.sslCAInfo set ([citation]) to a path relative to where it's unpacked.

• Desktop sets http.sslBackend=schannel in it's system config ([citation]) as part of the default install.

• Git 2.17.1(2) comes with curl v7.60.0, which now supports passing a traditional CA bundle when using the SChannel backend. @niik wrote up a good summary here about the change in behaviour: https://github.com/desktop/desktop/issues/4817#issuecomment-393241303

Details

Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other**

Vanilla CMD

What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.**

> cd ~/Desktop/MinGit-2.17.1.2-64-bit/
> .\cmd\git config --system http.sslbackend schannel
> SET GIT_TRACE=1
> SET GIT_CURL_VERBOSE=1
> .\cmd\git clone https://github.com/desktop/desktop.git
18:47:07.947050 exec_cmd.c:236          trace: resolved executable dir: C:/Users/shiftkey/Desktop/MinGit-2.17.1.2-64-bit/mingw64/bin
18:47:07.947050 git.c:344               trace: built-in: git clone https://github.com/desktop/desktop.git
Cloning into 'desktop'...
18:47:07.993976 run-command.c:640       trace: run_command: git remote-https origin https://github.com/desktop/desktop.git
18:47:07.993976 exec_cmd.c:236          trace: resolved executable dir: C:/Users/shiftkey/Desktop/MinGit-2.17.1.2-64-bit/mingw64/bin
18:47:07.993976 git.c:576               trace: exec: git-remote-https origin https://github.com/desktop/desktop.git
18:47:07.993976 run-command.c:640       trace: run_command: git-remote-https origin https://github.com/desktop/desktop.git
18:47:08.009550 exec_cmd.c:236          trace: resolved executable dir: C:/Users/shiftkey/Desktop/MinGit-2.17.1.2-64-bit/mingw64/bin
* Couldn't find host github.com in the _netrc file; using defaults
*   Trying 192.30.255.112...
* TCP_NODELAY set
* Connected to github.com (192.30.255.112) port 443 (#0)
* schannel: SSL/TLS connection with github.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 175 bytes...
* schannel: sent initial handshake data: sent 175 bytes
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: encrypted data got 3539
* schannel: encrypted data buffer: offset 3539 length 4096
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 4096
* schannel: SSL/TLS handshake complete
* schannel: added 158 certificate(s) from CA file 'C:/Users/shiftkey/Desktop/MinGit-2.17.1.2-64-bit/mingw64/ssl/certs/ca-bundle.crt'
* schannel: connection hostname (github.com) validated against certificate name (github.com)
* schannel: SSL/TLS connection with github.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET /desktop/desktop.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.17.1.windows.2
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
...

It's this line that I want to call out:

* schannel: added 158 certificate(s) from CA file 'C:/Users/shiftkey/Desktop/MinGit-2.17.1.2-64-bit/mingw64/ssl/certs/ca-bundle.crt'

This is the new behaviour that seems to be key. If I do the same steps with 2.17.0 I get this output:

> cd ~/Desktop/MinGit-2.17.0-64-bit/
> .\cmd\git config --system http.sslbackend schannel
> SET GIT_TRACE=1
> SET GIT_CURL_VERBOSE=1
> .\cmd\git clone https://github.com/desktop/desktop.git
C:\Users\shiftkey\Desktop\MinGit-2.17.0-64-bit>.\cmd\git clone https://github.com/desktop/desktop.git
19:27:54.403071 git.c:344               trace: built-in: git clone https://github.com/desktop/desktop.git
Cloning into 'desktop'...
19:27:54.465572 run-command.c:640       trace: run_command: git remote-https origin https://github.com/desktop/desktop.git
19:27:54.481195 git.c:576               trace: exec: git-remote-https origin https://github.com/desktop/desktop.git
19:27:54.481195 run-command.c:640       trace: run_command: git-remote-https origin https://github.com/desktop/desktop.git
* STATE: INIT => CONNECT handle 0x33d6770; line 1404 (connection #-5000)
* Couldn't find host github.com in the _netrc file; using defaults
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => WAITRESOLVE handle 0x33d6770; line 1440 (connection #0)
*   Trying 192.30.255.112...
* TCP_NODELAY set
* STATE: WAITRESOLVE => WAITCONNECT handle 0x33d6770; line 1521 (connection #0)
* Connected to github.com (192.30.255.112) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x33d6770; line 1573 (connection #0)
* Marked for [keep alive]: HTTP default
* schannel: SSL/TLS connection with github.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 175 bytes...
* schannel: sent initial handshake data: sent 175 bytes
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x33d6770; line 1587 (connection #0)
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: encrypted data got 3539
* schannel: encrypted data buffer: offset 3539 length 4096
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with github.com port 443 (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with github.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
* STATE: PROTOCONNECT => DO handle 0x33d6770; line 1608 (connection #0)
> GET /desktop/desktop.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.17.0.windows.1
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
...

What did you expect to occur after running these commands?

Previous versions of Git, when configured to use the SChannel backend, would skip the configured http.sslCAInfo file and instead use certificates installed into Windows. Two suggestions come to mind:

• The old behaviour: Git should not use the provided certificate bundle (I've always thought they were associated with the OpenSSL backend) and verify certificates using the existing SChannel APIs .
• Gracefully degrade: Git should fall back to SChannel APIs if the provided certificate bundle is not able to validate the certificate chain

What actually happened instead?

Various issues happened in the wild:

• https://github.com/desktop/desktop/issues/4828 - Connecting to a GitHub Enterprise host did not resolve correctly because it chose to use the default certificate bundle which did not contain valid certificates created by the company:

We shipped a fix for this in https://github.com/desktop/dugite-native/pull/98 which removes the system http.sslCAInfo config (if set in MinGit).

• https://github.com/desktop/desktop/issues/4817 - A http.sslCAInfo entry defined in the special %PROGRAMDATA%\Git\config config file was inherited into our environment:

This requires manually cleaning up that file - I couldn't find a recent version of Git for Windows that adds this.

• https://github.com/desktop/desktop/issues/4832 - the http.sslCAInfo defined in the special %PROGRAMDATA%\Git\config config file was inherited into our environment, but pointed to a absolute path rooted at / instead of C:\, which looked for a non-existent file in the MinGit distribution

Same thing as above - I couldn't find a recent version of Git for Windows that was doing this. Maybe you can recall how long ago this was?

If the problem was occurring with a specific repository, can you provide the URL to that repository to help us with testing?

This was only noticeable from servers whose certificates aren't in the configured bundle. I don't have a URL handy to point to currently.

[qq15870939565]

Manually delete C:\ProgramData\Git\config

[qq15870939565]

[http]
sslCAInfo = d:/Program Files (x86)/Git/mingw32/ssl/certs/ca-bundle.crt

this is OK

[shiftkey]

Manually delete C:\ProgramData\Git\config

This is a shared config file that all Git distributions should respect on a Windows machine. Removing entries arbitrarily is something that terrifies me.

>

[http]
sslCAInfo = d:/Program Files (x86)/Git/mingw32/ssl/certs/ca-bundle.crt

this is OK

This assumes 32-bit Git for Windows is installed at that location. I need to support users who may _or may not_ have an existing Git for Windows install. I want to understand this change to curl and how to properly support it when switching between backends.

[qq15870939565]

The best way is to find out where git is installed and have the correct configuration C:\ProgramData\Git\config
[http]
sslCAInfo = d:/Program Files (x86)/Git/mingw32/ssl/certs/ca-bundle.crt

[qq15870939565]

cover shared config -sslCAInfo

[shiftkey]

@qq15870939565 please read the issue. I want to avoid http.sslCAInfo completely because I set http.sslBackend=schannel and I have users who add custom certificates into the relevant Windows Certificate Stores (the same place that browsers will also look for custom certificates). Getting users to manage this certificate bundle on top of that isn't something I want to have to do.

[qq15870939565]

sorry!I understand that I don't have a previous version. I installed a 32-bit version of git. Today I upgraded to 64-bit. I updated git after updating GitHub Desktop. I can only provide so much information.

[whoisj]

We have nearly a dozen newly opened issues on Developer Community due to this issue. I consider this a high priority item.

Thanks @shiftkey for your deep dive work.

For the record git config --unset-all http.sslCAInfo is the best option if you need http.sslBackend=schannel.

(adding a link to the libcurl PR referenced in the GitHub desktop issue, so it'll be easier for the libcurl contributors to track things: https://github.com/curl/curl/pull/1325)

[dscho]

For the record: I discussed this with @whoisj and consider the following changes:

• introduce a config setting to heed sslCAinfo for schannel mode, and then
• in schannel mode, ignore the sslCAinfo by default

How does that sound, @shiftkey? Would that solve the issues you encountered?

[shiftkey]

I think that works for us, ping @niik @iamwillshepherd for a 👍

[dscho]

Please see https://github.com/dscho/git/commit/ffd4963bde17c11dfc2580d258a315964fa21378 for my proposed change.

cc @whoisj

FWIW I started the process to build and publish v2.18.0-rc2 with this change, hopefully there are no hiccups and this thing will be available at https://github.com/git-for-windows/git/releases/tag/v2.18.0-rc2.windows.1 in half an hour or so, could you please give it a whirl? (Git for Windows' prereleases include MinGit and PortableGit to allow for extensive testing...)

[whoisj]

@dscho I've downloaded the portable-64 package, I'll start playing with it now and update this thread with results.

[whoisj]

... GitHub lost my previous post. Not sure what just happened. 😞 Oh well, not writing all of that again. Here's the short version.

I have http.sslbackend=schannel set with httpsslcainfo={path} set (twice).

$>  git config --list | findstr /I http
http.sslcainfo=C:/ProgramData/Git/usr/ssl/certs/ca-bundle.crt
http.sslbackend=schannel
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt

Here's the output from v2.17.1(2) successful fetch, where we see the ca-bundle.crt file get read and 158 certificates added.

$> git version
git version 2.17.1.windows.2
$> git fetch origin
10:15:33.429188 http.c:709              == Info: Couldn't find host dev-x.visualstudio.com in the _netrc file; using defaults
10:15:33.541162 http.c:709              == Info:   Trying 13.107.6.175...
10:15:33.541162 http.c:709              == Info: TCP_NODELAY set
10:15:33.614188 http.c:709              == Info: Connected to dev-x.visualstudio.com (13.107.6.175) port 443 (#0)
10:15:33.614188 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 1/3)
10:15:33.614188 http.c:709              == Info: schannel: checking server certificate revocation
10:15:33.616188 http.c:709              == Info: schannel: sending initial handshake data: sending 187 bytes...
10:15:33.616188 http.c:709              == Info: schannel: sent initial handshake data: sent 187 bytes
10:15:33.616188 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.616188 http.c:709              == Info: schannel: failed to receive handshake, need more data
10:15:33.693189 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.693189 http.c:709              == Info: schannel: encrypted data got 4096
10:15:33.693189 http.c:709              == Info: schannel: encrypted data buffer: offset 4096 length 4096
10:15:33.694186 http.c:709              == Info: schannel: received incomplete message, need more data
10:15:33.694186 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.694186 http.c:709              == Info: schannel: encrypted data got 1024
10:15:33.694186 http.c:709              == Info: schannel: encrypted data buffer: offset 5120 length 5120
10:15:33.694186 http.c:709              == Info: schannel: received incomplete message, need more data
10:15:33.694186 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.694186 http.c:709              == Info: schannel: encrypted data got 1024
10:15:33.694186 http.c:709              == Info: schannel: encrypted data buffer: offset 6144 length 6144
10:15:33.694186 http.c:709              == Info: schannel: received incomplete message, need more data
10:15:33.694186 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.694186 http.c:709              == Info: schannel: encrypted data got 1024
10:15:33.694186 http.c:709              == Info: schannel: encrypted data buffer: offset 7168 length 7168
10:15:33.694186 http.c:709              == Info: schannel: received incomplete message, need more data
10:15:33.694186 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.694186 http.c:709              == Info: schannel: encrypted data got 452
10:15:33.694186 http.c:709              == Info: schannel: encrypted data buffer: offset 7620 length 8192
10:15:33.696162 http.c:709              == Info: schannel: sending next handshake data: sending 93 bytes...
10:15:33.769187 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 2/3)
10:15:33.769187 http.c:709              == Info: schannel: encrypted data got 51
10:15:33.769187 http.c:709              == Info: schannel: encrypted data buffer: offset 51 length 8192
10:15:33.771164 http.c:709              == Info: schannel: SSL/TLS handshake complete
10:15:33.775187 http.c:709              == Info: schannel: added 158 certificate(s) from CA file 'C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt'
10:15:33.797187 http.c:709              == Info: schannel: connection hostname (dev-x.visualstudio.com) did not match against certificate name (visualstudio.com)
10:15:33.797187 http.c:709              == Info: schannel: connection hostname (dev-x.visualstudio.com) validated against certificate name (*.visualstudio.com)
10:15:33.803187 http.c:709              == Info: schannel: SSL/TLS connection with dev-x.visualstudio.com port 443 (step 3/3)
10:15:33.803187 http.c:709              == Info: schannel: stored credential handle in session cache

Here's the output from v2.18.0(rc2) successful fetch, where we see the ca-bundle.crt file get noticed but no certificate adds reported.

$> D:\Users\whoisj\Downloads\PortableGit\cmd\git.exe version
git version 2.18.0.rc2.windows.1
$> D:\Users\whoisj\Downloads\PortableGit\cmd\git.exe fetch origin
10:17:29.517372 http.c:724              == Info: Couldn't find host dev-x.visualstudio.com in the _netrc file; using defaults
10:17:29.599350 http.c:724              == Info:   Trying 13.107.6.175...
10:17:29.600347 http.c:724              == Info: TCP_NODELAY set
10:17:29.673372 http.c:724              == Info: Connected to dev-x.visualstudio.com (13.107.6.175) port 443 (#0)
10:17:29.778372 http.c:724              == Info: ALPN, offering http/1.1
10:17:29.778372 http.c:724              == Info: Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
10:17:29.785373 http.c:724              == Info: successfully set certificate verify locations:
10:17:29.785373 http.c:724              == Info:   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
10:17:29.786363 http.c:724              == Info: TLSv1.2 (OUT), TLS header, Certificate Status (22):

[dscho]

@whoisj short version: it works? ;-)

[whoisj]

@whoisj short version: it works? ;-)

Yes. I think so. Is that the output you expected?

[dscho]

Is that the output you expected?

Pretty much ;-)

[dscho]

Is that the output you expected?

Pretty much ;-)

I am actually a bit concerned because the bundle is still reported. But since it does not actually add the certificates, I imagine that the output comes from cURL's initialization, and my new code to disable CAfile works as intended (and is executed after that log message).

[dscho]

Read: I would appreciate more testing from the GitHub Desktop side... @shiftkey @niik @iAmWillShepherd hint, hint, strong hint

[shiftkey]

@dscho you've caught me during a really busy week, but I'll have a chance to test this out and get feedback to you on Monday

[dscho]

you've caught me during a really busy week, but I'll have a chance to test this out and get feedback to you on Monday

No worries @shiftkey! I know you just moved intercontinentally, I just hoped that your team would pick up some slack (as my team does when I'm on vacation...).

[dscho]

@niik @iAmWillShepherd you are really very, very silent. Not nice.

[niik]

For the record: I discussed this with @whoisj and consider the following changes:

introduce a config setting to heed sslCAinfo for schannel mode, and then
in schannel mode, ignore the sslCAinfo by default
I think that works for us, ping @niik @iAmWillShepherd for a 👍

That sounds like exactly what we'd want. 👍

I am actually a bit concerned because the bundle is still reported. But since it does not actually add the certificates, I imagine that the output comes from cURL's initialization, and my new code to disable CAfile works as intended (and is executed after that log message).

I believe that's curl built-in auto discovery of the CA bundle path right? As long as it doesn't try to read the file at all this should be safe. We had a few users who discovered that their http.sslCAInfo was set to a nonexistent path.

@niik @iAmWillShepherd you are really very, very silent. Not nice.

I'm, truly, grateful for your hard work on resolving this. That being said I would kindly ask to not have intent assigned in not responding to a Thursday morning ping before the weekend. I ensure you, not being nice is rarely my motive behind not answering a ping :heart:

[whoisj]

That being said I would kindly ask to not have intent assigned in not responding to a Thursday morning ping before the weekend. I ensure you, not being nice is rarely my motive behind not answering a ping

I believe by "not nice", the intended meaning isn't "rude" or "cruel", but simply a lack of being "nice". :man_shrugging:. ... and wait, your week ends on Thursday? Does that imply a 4-day work week? 🤤

Thanks for responding - I guess we ought to test with empty and invalid (missing, inaccessible, etc) ca-bundle.crt files

[dscho]

@niik hopefully I did not cause any hard feelings. I just wanted to make sure that I have a good bug fix ready for the imminent Git release (http://tinyurl.com/gitCal still says that it is intended to be released yesterday, but that clearly did not happen). In light of that, maybe you can forgive me for being so harsh?

[iAmWillShepherd]

@dscho sorry for the silence, I did not get this until Friday afternoon and decided to push it to this week when I had the time. Thanks for getting the fix in, if I'm able to complete my work tomorrow, I'd be able to test this out. Of course, my team mates may get to it before me, but I want to assure you we're not trying to be mean.

[dscho]

Well, I'll release Git for Windows v2.18.0 shortly after Git v2.18.0, with my current fix. If it does not work for anybody else, it's their responsibility to fix it now ;-)