Git-lfs: LFS incompatible to cURL+winssl

[hsattler]

Please see https://github.com/git-for-windows/git/issues/1158

Essentially, choosing the "Windows Secure Channel" option in Git-for-windows installer 2.12.2.windows.2 does not work for the LFS shipped in that installer (version 2.0.2).

[dscho]

I really do not like seeing links on top of anything else. Like, the readers' time is worthless, go hunt for yourself.

So let me summarize the problem here, to spare the good Git LFS maintainers a good chunk of their time.

Background: Windows has a certificate store, and native facilities for SSL/TLS ("Secure Channel"). It makes it easy to deliver company-wide "self-signed" certificates using native Windows administration tools. For portabilitiy, however, many projects choose to stick with OpenSSL, which does not support Windows' Certificate Store and therefore insists on using a "ca-bundle.crt" file.

This is burdensome to Windows-based administrators, therefore Git for Windows introduced an installer option recently to use the Secure Channel instead of OpenSSL for the HTTPS transport.

Obviously, this choice does not extend to Git LFS.

What @hsattler meant to ask is: is the Git LFS project aware of Windows' certificate store? Would it be possible to support that natively, rather than requiring users to add their custom certificates to the "ca-bundle.crt" file?

[hsattler]

@dscho: Sorry for the inconvenience. I though that the full discussion in the other bug report might be valuable. Thanks for extending the description.

[ttaylorr]

What @hsattler meant to ask is: is the Git LFS project aware of Windows' certificate store? Would it be possible to support that natively, rather than requiring users to add their custom certificates to the "ca-bundle.crt" file?

@dscho thanks for explaining this to me. I work on a macOS machine usually, so am usually quite unfamiliar with these sorts of issues on Windows. That being said, as you mentioned, LFS does not currently implement any knowledge of the Windows certificate store, though that or something like it would certainly be possible.

I do not think that this is on any of the core team's individual roadmaps right now, so in the meantime adding your own ca-bundle.crt would be the only workaround.

That being said, I am more to happy to help a contributor through the process of submitting a pull request against LFS, provided that they could help me through the Windows parts 😄 .

[dscho]

@ttaylorr thanks for the clarification!

May I ask for more enlightenment? Specifically, I would love to know how the HTTPS transport works, i.e. what libraries are at play. I just would like to get a better idea how involved a fix would be.

Thanks!

[shiftkey]

@dscho @ttaylorr yes, this scenario is also relevant to my interests. Happy to help get to the bottom of this...

[shiftkey]

This file looks relevant to our interests: https://github.com/git-lfs/git-lfs/blob/c2d242c0954476a858acbe0c5f546658d45ea531/lfsapi/certs.go

[shiftkey]

@hsattler I'd like to see a trace output (I think TRACERX_TRACE is enough, but @ttaylorr can correct me) on how the Git LFS command fails for your setup. This will help us understand where to dig into the Git LFS internals better (and quicker) and maybe provide a workaround.

[hsattler]

@shiftkey How do I use TRACERX_TRACE with git-lfs?
An export TRACERX_TRACE=d:/test/tracerx.txt or export TRACERX_TRACE=1 does not give anything when running git lfs clone http://server/username/repository.git.

[shiftkey]

@hsattler weird, that's what the source seems to suggest. I'll let others chime in here about troubleshooting, I'm probably missing something.

[technoweenie]

TRACERX is the default key prefix, but Git LFS uses GIT_TRACE (which also works with Git too).

[hsattler]

Same as in the git-for-windows bug report, then:
trace git-lfs: tq: sending batch of size 78
trace git-lfs: api: batch 78 files
trace git-lfs: HTTP: POST https://xxxxx/yyyyy/zzzzz.git/info/lfs/objects/batch

POST /yyyyy/zzzzz.git/info/lfs/objects/batch HTTP/1.1
Host: xxxxx
Accept: application/vnd.git-lfs+json; charset=utf-8
Content-Length: 6902
Content-Type: application/vnd.git-lfs+json; charset=utf-8
User-Agent: git-lfs/2.0.2 (GitHub; windows amd64; go 1.8; git 85e2aec4)

trace git-lfs: api error: Post https://xxxxx/yyyyy/zzzzz.git/info/lfs/objects/batch: x509: certificate signed by unknown authority

Git LFS: (0 of 578 files) 0 B / 796.48 MB
batch response: Post https://xxxxx/yyyyy/zzzzz.git/info/lfs/objects/batch: x509: certificate signed by unknown authority

[whoisj]

Which libcurl binary does Lfs consume? I assume it's imported via Go - though I suppose Go has written its own networking stack (seems highly unlikely). If my memory serves me, the libcurl-*dll needs to be re-built with the SChannel support enabled.

I do believe the ABI remains unchanged, so this could be a "free win".

[shiftkey]

@whoisj this comment has thrown me too, so I'm not sure which approach is being used by the Go networking stack.

[whoisj]

@shiftkey well, let's assume for a moment that Go-lang does rely on libcurl. Then that means it very well could be correct that it does support WinSsl and the Windows Certificate Store, but only if the correctly compiled version of libcurl is available.

There are quite a few reasons why it would be true for one user and not another.

• Accidental changes in build options between versions of Lfs
• One user has libcurl-4.dll in their %windir%\System32 folder (Windows will load libraries from here first).
  
  
  
  • Unrealized differences in 32- and 64-bit versions of Go-lang and/or libcurl.
  
  
• Environment variable switches which enable or disable libcurl or golang features
• etc...

Those of us who have become accustomed to Microsoft and/or Apple product often forget how overly configurable (even by accident) many operating systems and run-time environment can be.

[dscho]

Okay, I wanted to give the Go experts a chance to speak with competence, but it seems that @ttaylorr and @technoweenie would like amateurs such as myself to take over. sigh

Caveat emptor: I am a real amateur when it comes to Go. So take everything that is written below with a huge mountain range of salt.

So it would appear that the HTTPS proxy handling (which I used as an entry point for my git greping) is handled exclusively in lfsapi/lfsapi.go, and that file imports net/http: https://github.com/git-lfs/git-lfs/blob/master/lfsapi/lfsapi.go#L7

So let's look if we can extract some kind of information from the documentation of said Go package https://golang.org/pkg/net/http/

Nah, it does not really have any good information about certificates, except if you want to serve via HTTPS, which we don't. :frowning_face:

Okay, let's try a web search for "go" and "https"... lots of hits... lots of talk about serving. :frowning_face:

Ah, one hit with Go's source! https://golang.org/src/crypto/tls/. Let's see. The closest I guess is https://golang.org/src/crypto/tls/handshake_client.go. Let's look inside. Ah, it references other stuff inside other Go packages, such as crypto/x509.

So it looks as if Go indeed has its own HTTPS transport, and does not use cURL.

Frankly, I would have been surprised.

Now I am at a dead end: I do not speak Go, I just guess from common constructs what it does, and I really do not want to dive into Go's own source code, just to help with an issue that is only vaguely related to Git for Windows (and there are many dozens of Git for Windows issues waiting for my attention, I should probably not indulge in this side track, but whatevs).

But hey, I remember that somebody used Git for Windows' bundled certificates that were actually intended for use with OpenSSL. So how does Git LFS access the certificates we specify via our cURL config?

Let's search for "cert" in Git LFS' source code.

A ha!

https://github.com/git-lfs/git-lfs/blob/ac9ad7d8175a181fdd2a6ffab2bf07f9c3938080/lfsapi/certs_windows.go

But that does not make sense, either. That code has an explicit comment stating:

// golang already supports Windows Certificate Store for self-signed certs

Whoa. If that were true, how on earth can @hsattler run into the described problems?

Back to the web search (wow, I already spent most of my time allotted for bug fixing today just on this issue, I really wish a Go expert would have done this analysis, can't have taken that long...).

Ah. There seems to be a ticket in the Docker project (which is written in Go): https://github.com/moby/moby/issues/24806. Linking to https://github.com/moby/moby/issues/12756. Ah, so if you specify a custom CA, it won't use the Windows Certificate store.

That, or that Go has problems: https://github.com/golang/go/issues/13335 (see also the list toward the bottom of related tickets).

Oh, wait. If it is actually because we specified a custom CA, then it is most likely because there is a reference in the Git config that points to the ca-bundle.crt, and Git LFS uses our config, even if Git for Windows does not use it.

@hsattler could you try to remove the http.sslcainfo setting (in my setup, it is in C:\ProgramData\Git\config) and test again whether Git LFS picks up the certificate from the Windows Credential Store?

[hsattler]

@dscho Thanks a lot for your detailed analysis. I will try your suggestion once I am back in the office. Might be some days, though.

[technoweenie]

Thanks @dscho, that's a great analysis. This isn't my area of expertise, so I spent some time yesterday tracking the PR where the ssl handling code was introduced (https://github.com/git-lfs/git-lfs/pull/1067), and how the root CAs are added.

1. The HTTP Transport configures the TLS client here.
2. That attempts to load the root ca's from gitconfig, and then any platform specific method here.
3. The pool is updated from git config settings here. Is it missing some git config or env values?

[dscho]

@technoweenie I do not think that any git config or env values are missing. If I read the situation correctly, then the problem is triggered by our reading of the git config: it seems that at least one report suggests that Go's CA handling will skip the system CA (which would be the Windows Certificate Store on Windows) if you specify one CA (which http.sslcainfo does).

Again, this is just a guess, but i am fairly certain that we are now on to something, and I await @hsattler 's feedback eagerly.

[wasilukm]

I have confirmed that removing http.sslcainfo from git configuration forced LFS to use winssl, so I could clone my repo successfully.

[technoweenie]

@wasilukm Thanks for that confirmation :+1: Does that mean _both_ Git and Git LFS work with winssl without having to set http.sslcainfo?

[dscho]

@technoweenie it does mean that. But it also means that having set http.sslcainfo causes the bug in Git LFS that it ignores the Windows Certificate Store.

If https://github.com/golang/go/issues/16736 ever gets addressed in Go, this bug may be fixed. In the meantime, I am uncertain what to do, as I am lousy at Go (as I have mentioned previously).

[wasilukm]

@technoweenie Yes, they work with winssl without http.sslcainfo set in the configuration. It is working already on several machines in my company.

[hsattler]

Without http.sslcainfo set, it also works here. So git lfs does not really have a problem as it can easily switch between the two certificate stores/files.

After all, selecting a certificate file may also mean to to ONLY accept those, not additionally.

Maybe this bug should be closed and there should be a solution in https://github.com/git-for-windows/git/issues/1158 only?

[whoisj]

It's not a Git for Windows bug to have http.sslcainfo set and ignore it in favor of WinSSL.

I can imagine several setups where a user has more than one instance of Git established, one with OpenSSL + http.cainfo and the other with Secure Channel via WinSSL - are you telling me that you'd consider that a bug? 😮

[hsattler]

@whoisj: It actually is a Git for Windows bug to have http.sslcainfo set when it is not used as shown by this bug report. If that is not considered bad, then a line-up of git lfs with the other git tools is needed.

At the moment you cannot have your described behaviour as git for windows support either WinSSL or OpenSSL, not both with the same installation:

1. cURL with WinSSL -> http.sslcainfo should not be set by installer.
2. cURL with OpenSSL -> http.sslcainfo must be set by installer.

=> git lfs will automatically work with either WinSSL or OpenSSL.

If git for windows could ship both cURL versions and switch based on http.sslcainfo being set, then it is always well-defined behaviour and would be in-line with current git lfs behaviour.

[whoisj]

@hsattler a system can contain more than a single installation of Git for Windows. One of those might be setup to use OpenSSL and another WinSSL - and the user is absolutely free to add a http.sslcainfo to their git config --global, git config --local, or git config --file as they see fit.

Therefore, I assert it is not a bug but a fact of life and the existence of a http.sslcainfo record in the git-config absolutely should not be inferred to mean that the user expects it to be used in-place of Secure Channel / WinSSL.

[dscho]

In indicated in https://github.com/git-for-windows/git/issues/1158#issuecomment-308224678 how to go about fixing it. Yes, it is an easy project. So easy, in fact, that I am happy to leave it to you (and instead go back to problems much too challenging to delegate).

@whoisj you are correct that there can be multiple Gits for Windows installed. That is the reason why the http.sslcainfo entry is not configured in C:\ProgramData\Git\config anymore. It is "system-wide" to the respective Git for Windows. Hence it is fair game to be removed if the user configured Secure Channel as the secure transport.

[whoisj]

the http.sslcainfo entry is not configured in C:\ProgramData\Git\config anymore

That's great new going forward., However users with pre-existing configuration entries will likely retain the entry. There's going to be a lot of those users. Therefore, I stand by my original assertion that "it is not a bug but a fact of life and the existence of a http.sslcainfo record in the git-config absolutely should not be inferred to mean that the user expects it to be used in-place of Secure Channel / WinSSL".