Git-credential-manager-for-windows: GCM Latest vesion (v1.10.1) does not work with TFS Git

[evil-shrike]

I had difficulties to get GCM working with TFS Git. It works fine with Bitbucket/Github for example but not with TFS. See details here - https://developercommunity.visualstudio.com/content/problem/88612/git-clone-authentication-failed.html

But old version v1.2.2 works fine. I installed it separately (via install.cmd).

git version 2.13.3.windows.1 in both cases.

I didn't test other versions between 1.2.2 and 1.10.1. So I only can tell that v1.2.2 works, v1.10.1 does not.

[lafe]

Same here. The previously installed version was 1.8.1. I upgraded to the most recent one (1.12.0) and it failed as well. Downgrade to 1.2.2 works. The issue is also not isolated to a single computer or user account but occurs on other computers (with or without domain membership) and with other users as well.
The issue only occured after I changed my Windows password, so with previously stored credentials it worked. We are currently using TFS 2017 Update 2 RC 2 (v. 15.117.26615.0).

Just investigated the issue further:
1.7 is the last version that works. Starting with 1.8 the error occurs assuming that no credentials are in the Windows Credential Manager. When 1.7 adds the Credentials to the Credential Manager, I can update to 1.12 and it will continue to run. If I remove the credentials from the Credential Manager, it will stop working (downgrade to 1.7, entering credentials again and upgrade to 1.12 works again).

[whoisj]

I had to really go back in time to find when 1.2.2 was released (Mar 2016), and at that time there was no support for NTLM. The NTLM support was added in 1.8.0 (and fixed/updated in subsequent releases).

This leads be to believe that the TFS servers in use are designed to require username + password and to not support NTLM / Active Directory. In this case, use the following configuration option:

git config --global credential.msts.authority basic

This will hint the GCM to not allow NTLM and use basic authentication even when the server broadcasts that NTLM is available.

[Tankatronic]

@whoisj I tried that option it still does not work with TFS. Also can confirm that version 1.2.2 works for TFS.

I was using version 1.8.0 before and it had worked. However my password expired and even after clearing my credential store I still could not get credential manager to reprompt me for credentials. I uninstalled 1.8.0 and installed the latest version, which did not work as stated above, then uninstalled and installed 1.2.2 and it worked without any further configuration.

[whoisj]

@Tankatronic could you enable tracing and see what is reported?

C:\Src\repo> SET GIT_TRACE=1

C:\Src\repo> SET GCM_TRACE=1

C:\Src\repo> git fetch --all

[Tankatronic]

Sure thing, I will get to it by early next week!

On Thu, Aug 31, 2017 at 2:35 PM J Wyman [e-mail] wrote:

@Tankatronic https://github.com/tankatronic could you enable tracing
and see what is reported?

C:\Src\repo> SET GIT_TRACE=1

C:\Src\repo> SET GCM_TRACE=1

C:\Src\repo> git fetch --all

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/473#issuecomment-326398953,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AD7M9Ft0Kx2Vu21B6vjAZMhyPT3i-CB3ks5sdwsYgaJpZM4OsSZC
.

[whoisj]

@Tankatronic any chance you had a test to reproduce the issue with the tracing enabled? I'm likely coming into some available time to spend on the GCM and am trying to prioritize issues to work on.

Thanks!

[Tankatronic]

Sorry @whoisj! I was busy and totally forgot.

I went ahead and uninstalled my previous version and installed the latest version, it's working now however.. These are the steps I took

1. Uninstalled gcm v 1.2.2
2. Executed git fetch --all on repository. Was prompted for credentials. Entering them and executing command again prompted for credentials again.
3. Installed gcm v 1.12.0
4. Executed git fetch --all on repository. Was prompted for credentails.
5. Executed git config --list --global. No credential.helper manager key/value pair.
6. Executed git config --global credential.msts.authority basic. Still asking for credentials. Entering them again and doing another git fetch --all prompted credentials again.
7. Executed git config --unset --global credential.msts.authority basic
8. Executed git config --global --credential.helper manager
9. Executed git fetch --all. Successful.

[whoisj]

Ahh I see. So you need to force the GCM into basic mode to capture credentials. Once captured, the credentials work.

The issue is, most likely, that since v1.2.2 the GCM has "learned" about NTML and prefers it to basic authentication when the server advertises that it supports it. The credentials for NTLM are always stored as ":" (empty username + empty password).

Since the GCM will always at least try to make existing credentials work, it doesn't get to the "try with NTLM" part because the cached credentials work.

Sounds like: your TFS server has NTLM enabled and this is advertised, but for whatever reason it doesn't really work and you need to just use basic.

In this case, there's not a ton the GCM can do, other than allow you to override the authentication type via git config --global credential.msts.authority basic.

Unless you've any ideas, I'm not sure there's much I can do here. Removing NTLM support would break hundreds (thousands?) of users who rely on it daily, and since there's no standardized probe for "does NTLM work" across Git hosts, there's no reasonable way for the GCM to validate the NTLM credentials besides handing them to Git.

[Tankatronic]

TFS 2017 removed the option to select NTLM as an authentication method in the administrator console. You have to run a command on the command line to set TFS to use NTLM. Currently ours is set up to use Negotiate (Kerberos), just verified that on our console.

This morning once I uninstalled GCM 1.2.2, it unset the credential.manager helper variable in my global settings. Installing GCM 1.12.0 did not reset this variable as I had to set it again in step 8. I have had this issue with GCM for a long time, at least at work, as it never sets that credential.helper manager variable in any level of my settings. So I think what was happening, at least this morning, is git was trying to use its default method to prompt for credentials and GCM was not involved until step 8. I verified this by running the git fetch commands again and I was prompted each time for my credentials as I would expect git to do without GCM. Once I reset the credential.helper manager config, it worked without prompting me for credentials.

As of this writing I just discovered why as well. My credentials were already present in my credential store. So once I renabled the credential helper, GCM was able to find my stored credentials. I deleted my credentials, and now I am back at my original issue where Authentication is failing. Here are the logs you requested:

git fetch --all
13:37:34.171103 git.c:349 trace: built-in: git 'fetch' '--all'
Fetching origin
13:37:34.173103 run-command.c:336 trace: run_command: 'fetch' '--append' 'origin'
13:37:34.256111 git.c:349 trace: built-in: git 'fetch' '--append' 'origin'
13:37:34.270113 run-command.c:336 trace: run_command: 'git-remote-https' 'origin' 'https://tfs.[redacted].hq/[redacted]'
13:37:34.686154 run-command.c:336 trace: run_command: 'git credential-manager get'
13:37:34.830169 git.c:563 trace: exec: 'git-credential-manager' 'get'
13:37:34.830169 run-command.c:336 trace: run_command: 'git-credential-manager' 'get'
13:37:34.942180 ...\Common.cs:524 trace: [Main] git-credential-manager (v1.12.0) 'get'
13:37:35.005186 ...\Where.cs:239 trace: [FindGitInstallations] found 1 Git installation(s).
13:37:35.009186 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 62 entries.
13:37:35.010187 ...\Where.cs:239 trace: [FindGitInstallations] found 1 Git installation(s).
13:37:35.011187 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 62 entries.
13:37:35.029188 ...\Common.cs:74 trace: [CreateAuthentication] detecting authority type for 'https://tfs.[redacted].hq/'.
13:37:35.038189 ...uthentication.cs:139 trace: [GetAuthentication] not github.com, authentication creation aborted.
13:37:35.038189 ...\Common.cs:169 trace: [CreateAuthentication] authority for 'https://tfs.[redacted].hq/' is basic with NTLM=Auto.
13:37:35.253211 ...uthentication.cs:122 trace: [AcquireCredentials] 'https://tfs.[redacted].hq/' supports NTLM, sending NTLM credentials instead
13:37:35.253211 ...\Common.cs:556 trace: [QueryCredentials] credentials found.
13:37:35.282214 run-command.c:336 trace: run_command: 'git credential-manager erase'
13:37:35.387224 git.c:563 trace: exec: 'git-credential-manager' 'erase'
13:37:35.387224 run-command.c:336 trace: run_command: 'git-credential-manager' 'erase'
13:37:35.499235 ...\Common.cs:524 trace: [Main] git-credential-manager (v1.12.0) 'erase'
13:37:35.563242 ...\Where.cs:239 trace: [FindGitInstallations] found 1 Git installation(s).
13:37:35.567242 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 62 entries.
13:37:35.568242 ...\Where.cs:239 trace: [FindGitInstallations] found 1 Git installation(s).
13:37:35.569242 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 62 entries.
13:37:35.582244 ...\Common.cs:74 trace: [CreateAuthentication] detecting authority type for 'https://tfs.[redacted].hq/'.
13:37:35.591245 ...uthentication.cs:139 trace: [GetAuthentication] not github.com, authentication creation aborted.
13:37:35.591245 ...\Common.cs:169 trace: [CreateAuthentication] authority for 'https://tfs.[redacted].hq/' is basic with NTLM=Auto.
13:37:35.592245 ...\Common.cs:189 trace: [DeleteCredentials] deleting basic credentials for 'https://tfs.[redacted].hq/'.
13:37:35.593245 ...aseSecureStore.cs:50 trace: [Delete] credentials not found for 'git:https://tfs.[redacted].hq'.
fatal: Authentication failed for 'https://tfs.[redacted].hq/[redacted]/'
error: Could not fetch origin
13:37:35.615247 run-command.c:336 trace: run_command: 'gc' '--auto'
13:37:35.696255 git.c:349 trace: built-in: git 'gc' '--auto'

It seems like it is unable to create my credentials, and it is defaulting to NTLM when our TFS is not configured to use NTLM.

[whoisj]

The correct command would be git config --global credential.helper manager, careful in your response you had helper and manager flipped. 😏

Secondly, you'll need to force the GCM into basic mode via cd {worktree_dir} && git config credential.authority basic. If your server is advertising NTLM (and I can see that it is via this line 13:37:35.253211 ...uthentication.cs:122 trace: [AcquireCredentials] 'https://tfs.[redacted].hq/' supports NTLM, sending NTLM credentials instead) but you need to use basic, then you'll need to force basic authentication on.

[Tankatronic]

Setting git config --global credential.authority basic worked. I was prompted for my credentials and it created/saved them properly. Question though, in your original response to OP you said to set git config --global credential.msts.authority basic, which is what I originally did that did not work. What was the msts portion for?

[whoisj]

What was the msts portion for?

The OP's original question was actually on https://developer.visualstudio.com/ and the URL "http://msts:8080" was specifically mentioned. The "msts" was in reference to the URL because the GCM will perform best matching against any git-configuration options it knows how to read.

In your case, your server wasn't likely named msts, thus having that in the configuration name caused the GCM to assume that the option was for some other server. The simple credential.{option} entries affect all hosts, while the credential.{domain}.{option} entries are limited to operations which match {domain} and they also override any general options for that match domain only.

Does that answer your question? Apologies for the confusion. If you have suggestions on how to make that clearer I'd love to hear them. Thanks.

[Tankatronic]

That does clear it up thanks. I just saw the documentation over these options and had I have read that, I would have known!

[grbsystems]

I've just run through this exact issue when using an out-of-the-box basic install of TFS 2018 RC1. A default install of Git for Windows with a default install of TFS 2018 RC1 should work together without additional configuration.

I suspect the issue is the default setup of TFS 2018 RC1 and the way that it has it's authentication set up, but I'd like the Devs here to comment before I raise an issue,

Thanks

J.

[whoisj]

I've asked one of the developer leads for TFS to give me additional information. I'll reply here (or he will) once I've got something useful to share.