Git-lfs: GIT lfs 2.9.0 gives stream error: stream ID 39; HTTP_1_1_REQUIRED on Azure Devops servers

[YawJatah]

Hi guys,
We have been having a core issue with git lfs and Azure Devops Servers (and also it's precedent Team Foundation Server 2018). The issue steams from when a binary file is introduced in to the repo before being tracked by git lfs. before 2.9.0 git lfs will just hang on the problematic binary without any feedback. The issue is related to fix #3806, so we updated to 2.9.0 to see if our issue will be resolved. Unfortunately with 2.9.0 we can't even do git lfs on an Azure DevOps Server because right away you run in to stream error: stream ID 39; HTTP_1_1_REQUIRED. This is a possible regression introducted because of fix #3793 which was suppose to add HTTP/2 support.
From my understanding IIS 10 has a deep support for HTTP/2 (and apparent can't be turned of in recent version of Windows Server), and since hosted Azure DevOps Servers run off off IIS they support HTTP/2 by default for ssl connections and will fall back to HTTP/1.1 if the client doesn't support it by usse the error HTTP_1_1_REQUIRED.

From the logs (GIT_TRACE=1), git lfs starts the connection as HTTP/1.1, is it possible that the implementation of support of HTTP/2 isn't complete (hence there might be a mis match in between the git lfs client and server when it comes to agreeing on what protocol to use).

From the trace log git lfs does a:
POST /url_to_git_repo/project.git/info/lfs/objects/batch HTTP/1.1.

Any assistance will be greatly appreciated.

We can't really use git 2.8.0 and prior, because of the hang, and can't use 2.9.0 because of the HTTP/2.0 issue.

As a side note: it feels like git lfs needs a lot of testing on a self hosted TFS/Azure Devop servers as they seem to be lots of little blocking issues that you don't encounter on a GitHub server :S. You guys are doing a great job 👍

[YawJatah]

Update:
We were able to disable HTTP/2 on our Windows Server (not ideal because now all the IIS hosted webservices are running with HTTP/1.1) by using the registery change described here: https://forums.iis.net/t/1235335.aspx?HTTP+2+connections+with+iOS+or+Android+devices+not+working+. And this allows us properly connect. But we are running into an issue where one binary file keeps giving:
No credentials are available in the security package
during the git lfs fetch, and causes the lfs operaiton to abort.

[bk2204]

Hey,

Sorry to hear you're having trouble. We use Go's built-in HTTP/2 support, which presumably works. The Bitbucket folks use HTTP/2, from what I can tell, and the prerelease passed their tests, so 2.9.0 should be at least minimally functional using HTTP/2.

I don't know if Go's HTTP/2 support does or doesn't handle HTTP_1_1_REQUIRED gracefully, though. From what Google tells me, this appears to show up with IIS 10 when you're using client certificates; is that the case here?

We could add a knob to disable HTTP/2 support if it doesn't work with IIS (or more likely honor the http.version setting that Git already has).

As for the "No credentials are available in the security package", we've seen that in #3829, and it likely has something to do with NTLM, but since none of the Git LFS maintainers use NTLM and the code was contributed entirely by third parties, we have no clue what's going on. We'll be happy to accept a patch which improves things, though.

[bk2204]

Ah, reading the IIS documentation, this is going to be because you're using NTLM, which isn't support with HTTP/2.

I'll try to spend some time implementing the http.version option, hopefully over the next few days, and see if we can roll out a 2.9.1 with that option in.

[YawJatah]

@bk2204 Thank you for your follow-up, and for shedding light on this issue. Indeed we use NTLM (without client certificates). The strange thing is that git lfs and TFS (now Azure devops), has worked (we have been using it since early 2018, not without it's problems though); the issues (such as No credentials are available in the security package) arises when a file is tracked after it has already been committed. After that it seems like the TFS/Azure Devops implementation of git lfs server doesn't play nice with git lfs clients.

What will be the best alternative to NTLM? (Right now our Azure server supports NTLM and Negotiate)
Thanks again for you time and assistance (I am not familiar with Go, so can't pitch in on the dev side, but glad to help with testing and logs).

Cheers.

[bk2204]

The only recent change to NTLM support was in #3617, which came in with 2.8.0, and which tries to use NTLM/SSPI login first and then fall back to prompting for credentials. That was from @AJH16, who might know more about what's going on.

The message "No credentials are available in the security package" comes from Windows, but I don't know if that's from the client side or server side. If it's from the client side, it might suffice to find that error code and treat it as an error to be retried, which could fix the problem if it's a problem with a lack of SSPI credentials. Unfortunately, I don't know the type of the error that would be received, so it would be impossibile for me to extract the proper error code from it.

As for alternatives, the Negotiate mechanism is actually a meta-mechanism that handles both NTLM and Kerberos. Git LFS doesn't support Kerberos at the moment (although as a Kerberos user, I intend to fix that), so if you don't want to use NTLM, you'll need to use Basic authentication.

[bk2204]

I have a guess about what might be the problem here. Can you build #3881 and see if that works for you when pushing?

[YawJatah]

@bk2204 , Thanks again for the follow-up and clarifications.

We ended up enabling basic authentication (In conjunction with Negotiate), and now we can sync the git lfs files that were problematic.
Basically we had to turn off HTTP/2 and enable basic authentication (which is not desired from a security point of view, and the fact that i breaks PAT authentication, but for now since we use vpn and https we can secure the server that way).

The conclusion we can come up is that the current implementation of git lfs NTLM authentication, is missing some error handling.
Maybe instead of aborting when we receive 401: No credentials are available in the security package, git lfs should retry the same way it retries when it receives other 401 during the negotiate process?
Maybe @AJH16 can confirm?

Thanks again for looking into this. I know alot of Azure DevOps users now use the cloud services instead of rolling our their own hosted server, so this issue is a bit niche, but I think resolving this issue will allow for more adoption of git lfs for projects that need to have binary in the repo (basically all video game development), and need to keep their repo off of the cloud.

Cheers,

[bk2204]

The conclusion we can come up is that the current implementation of git lfs NTLM authentication, is missing some error handling.
Maybe instead of aborting when we receive 401: No credentials are available in the security package, git lfs should retry the same way it retries when it receives other 401 during the negotiate process?

Yeah, that's what #3881 tries to do, assuming the problem is on the client side (which I'm still unclear about). I've reached out to the other person who was experiencing that error and hopefully someone will be able to confirm that it does in fact fix the problem.

As for the HTTP/2 issue, I plan to spend some time implementing http.version so you can force a downgrade if you want without needing to turn off HTTP/2 on the server side. It will likely be next week before I can get to that, though.

[YawJatah]

@bk2204 greatly appreciated for taking the time to look into this. Let me know if there is a build you want us to test and it will be a pleasure to assist.

For now we will use our workaround (HTTP/2 off, basic authentication activated).

Thanks

[AJH16]

It's been a while so it's a bit foggy in my memory by this point, but I tried to minimize touching the NTLM flow itself with my changes. The big thing I did was just wrap up the credentials such that we could do a later prompt for credentials and that way we can attempt to use integrated credentials first before falling back to prompting for NTLM credentials. I've not been able to reproduce any of the problems described here so far on any of our Azure Devops Server instances.

[cloud586]

I ran into the same issue where I was getting the error HTTP_1_1_REQUIRE. What I found was disabling HTTP/2 on the Azure DevOps server did work but was not the solution. My local client was using an outdated version of git. When I updated to version 2.23 I was no longer receiving the issue and was able to commit to LFS.

[hhyyrylainen]

I didn't personally get this or a related issue, but I got a few people reporting that they can't use git lfs version 2.9.0 to checkout my repository. My git lfs server is using HTTP/2 so I'm thinking it is related. The server has a nginx remote proxy in front so I hope there isn't anything wrong with nginx's HTTP/2 implementation.

I'll ask if they can check if updating their git version helps.

[bk2204]

We'll be dong a 2.9.1 release on Monday with the http.version option, which can be used to force the use of HTTP/1.1 if necessary.

[danstur]

@bk2204 Just to summarize the problem to make sure I understood it correctly: If communicating with a server that supports HTTP/2 git lfs will since 2.9.0 use HTTP/2 instead of HTTP 1.1, which then causes problems because NTLM (not sure why Kerberos isn't used, but let's ignore that for now) is not supported with HTTP/2.

It's rather hard to find information on this, but is standard git also using HTTP/2 by now (2.24 windows)? (I'll look into this myself, but given missing HTTP/2 support with fiddler I'll have to check this out with wireshark). Because if that is actually working having to disable HTTP/2 completely would be a bit sad and an additional option to explicitly set the protocol version for git lfs would be great.

[bk2204]

@bk2204 Just to summarize the problem to make sure I understood it correctly: If communicating with a server that supports HTTP/2 git lfs will since 2.9.0 use HTTP/2 instead of HTTP 1.1, which then causes problems because NTLM (not sure why Kerberos isn't used, but let's ignore that for now) is not supported with HTTP/2.

That's correct. It isn't that we don't support HTTP/2 with NTLM, but that IIS doesn't support HTTP/2 with NTLM. Apache appears to support SPNEGO (NTLM and Kerberos) auth with HTTP/2 just fine.

We don't support Kerberos because nobody has yet added support for it. I hope to work on that soon, because I have a (Linux) Kerberos environment at home which I can use for testing.

It's rather hard to find information on this, but is standard git also using HTTP/2 by now (2.24 windows)? (I'll look into this myself, but given missing HTTP/2 support with fiddler I'll have to check this out with wireshark). Because if that is actually working having to disable HTTP/2 completely would be a bit sad and an additional option to explicitly set the protocol version for git lfs would be great.

Git is using HTTP/2 if libcurl supports it, which it does on my Linux box. I appreciate that a separate option would be nice, but we generally prefer to use the same options for both if possible. Note that Git doesn't benefit that significantly from HTTP/2 with the smart protocol since it doesn't make concurrent requests, so the impact on Git from using HTTP/1.1 is minimal.

[danstur]

That's correct. It isn't that we don't support HTTP/2 with NTLM, but that IIS doesn't support HTTP/2 with NTLM. Apache appears to support SPNEGO (NTLM and Kerberos) auth with HTTP/2 just fine.

We don't support Kerberos because nobody has yet added support for it. I hope to work on that soon, because I have a (Linux) Kerberos environment at home which I can use for testing.

Wasn't sure if that wasn't a misconfiguration on our part, glad to hear that that's not the case. If there's an issue for that, I wouldn't mind testing this in our dev environment, if you can use additional testers with IIS on Windows Server 2016 and Windows 10 clients.

Git is using HTTP/2 if libcurl supports it, which it does on my Linux box. I appreciate that a separate option would be nice, but we generally prefer to use the same options for both if possible. Note that Git doesn't benefit that significantly from HTTP/2 with the smart protocol since it doesn't make concurrent requests, so the impact on Git from using HTTP/1.1 is minimal.

Fair point, so it seems fine to simply set the option altogether and forget about it until Kerberos support comes along and then check if IIS supports HTTP/2 and Kerberos together.

Thanks for the hard work 👍

[galsi]

After updating to v 2.9.1 - push to to AzureDevops Server is working with LFS
Thanks for you help

[bk2204]

Just so you know, there's a PR (#3941) that implements Kerberos support. Theoretically it should work on Windows, but I have no Windows systems using Kerberos, so testing would be appreciated. From my reading of the IIS documentation, neither Kerberos nor NTLM are supported with HTTP/2, although as I mentioned above, this is not intrinsically a limitation of HTTP/2.

Since the issue originally mentioned here (that users couldn't use NTLM with a self-hosted Azure Devops server) has been fixed (by allowing a downgrade to HTTP/1.1), I'm going to close this issue.

[itsho]

Hi, I still have the same issue with both v2.9.1 and v2.9.2.

To temporarily bypass it, I had to edit my gitconfig and add to [http] section the following line:
version=HTTP/1.1

[jmlundberg]

@itsho @bk2204. Thank you. We have this issue with git 2.24.1.windows.2 (but not 2.21.0)
It seems fixed by

git config --global --unset http.version HTTP/1.1
git config --global --add http.version HTTP/1.1

[andyraf]

It's unclear whether there is an actual fix, or if this is just a workaround.

Using git version 2.25.0.windows.1, we hit this issue trying to push a 139MB file using GIT LFS ao an Azure server. The push failed consistently (despite fiddling with the various timeouts), but finally succeeded with the change to set:

git config http.version HTTP/1.1

Is that the final solution for all Azure users, or is there a fix forthcoming so long term people don't need to know this workaround?

[AJH16]

I finally did have a repo reproduce this issue.  I'm hoping to dig further this week or next, but have a considerably busy schedule now than I used to, so I'm not exactly sure when I'll be able to look.On Mar 31, 2020 8:45 PM, Andrew Raffman [e-mail] wrote:
It's unclear whether there is an actual fix, or if this is just a workaround.
Using git version 2.25.0.windows.1, we hit this issue trying to push a 139MB file using GIT LFS ao an Azure server. The push failed consistently (despite fiddling with the various timeouts), but finally succeeded with the change to set:
git config http.version HTTP/1.1

Is that the final solution for all Azure users, or is there a fix forthcoming so long term people don't need to know this workaround?

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[bk2204]

Setting the HTTP version to 1.1 is the final solution for Azure DevOps users unless Microsoft decides to make HTTP/2 work with Kerberos and NTLM. Apache does work fine in that configuration, so it is possible to do; I suggest you reach out to Microsoft and report a bug with them.

You can also use Basic authentication if you prefer that, in which case Azure DevOps should work.

[AJH16]

Ah, so my guess from yesterday was right about the problem then.  That's unfortunate.On Apr 1, 2020 9:49 AM, "brian m. carlson" [e-mail] wrote:
Setting the HTTP version to 1.1 is the final solution for Azure DevOps users unless Microsoft decides to make HTTP/2 work with Kerberos and NTLM. Apache does work fine in that configuration, so it is possible to do; I suggest you reach out to Microsoft and report a bug with them.
You can also use Basic authentication if you prefer that, in which case Azure DevOps should work.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[YawJatah]

Hi Guys, sorry that we haven't follow-up with this issue in a while.
I am not 100% certain about the full implementation of Azure DevOps git LFS, but there are several issues at play.

For starters the work around:
git config http.version HTTP/1.1

will only work if you don't have binary that was commit to git before you did git lfs install.
If that happens then you must use basic authentication to be able to access git lfs on Azure DevOps.

The default entry in the .git config file for git lfs (if git lfs install was run before a binary commit) is:

[lfs "https://my.domain.com/tfs/collation/_git/Project.git/info/lfs"]
    access = ntlm

When there is binary in your repo before git lfs install (and after that binary is being tracked byt git lfs)

then you must change that entry to (also basic authentication must be activated on your IIS server)

[lfs "https://my.domain.com/tfs/collation/_git/Project.git/info/lfs"]
    access = basic

When you run git command to pull (and git lfs gets called),

a new entry automatically added to the config file

[lfs "https://tfs.illuminiastudios.com/tfs/Game%20Dev/BOAA/_git/BOAA/info/lfs/objects/"]
    access = ntlm

This will new entry seems to be used for all files that were committed before being tracked by git lfs.
you must change the acces to

[lfs "https://tfs.illuminiastudios.com/tfs/Game%20Dev/BOAA/_git/BOAA/info/lfs/objects/"]
    access = basic

This is still not enough,
you must add the following entry for git lfs:

[lfs]
    tlstimeout = 300
    activitytimeout = 60
    dialtimeout = 600444
    concurrenttransfers = 10
    downloadtransfers = basic
    uploadtransfers = basic

The two must important entries are downloadtransfer and uploadtransfer, (the others are optional), must be set to basic.

After these changes the repo will work. This means that when cloning (same for pull if a file is later being tracked by git lfs) you must disable smudge by using:
GIT_LFS_SKIP_SMUDGE=1 GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 GCM_TRACE=1 git clone url_to_git repo_folder

So this seems to be a quirk with the implementation of git lfs server on Azure DevOps server.
Those two endpoints for git lfs seems to be causing the odd behaviors on the client side .

The second part of the problem (and the major one), is for those that are using Visual Studio as their git client (Visual Code doesn't seem to suffer from this issue).
Visual Studio seems to have a very hard time with git lfs (I wrongly presumed that it simply called git commands but there seems to be more at play). Since visual studio 2019, visual studio seems to interact with git in such a way that git lfs is broken. Basically all the binaries being tracked by git lfs aren't uploaded. So git pushes the commit but git lfs fails to upload the binarys. So then you have corrupted references on the server.
When this happens the only ways is to recommit the missing binary using the command line (or any other git client that properly invokes git via the command line like Visual Code).

For us this was one of the main problems, because once this happens the Azure repo will forever be in a state where you need to connect using basic authentication work around(even after you recommit the culprit binary).