Git-credential-manager-for-windows: GCM keeps credentials.lock files in Apache Temp and git-credential-manager hangs

[acoberlin]

• [x] I was not able to find an [open] or [closed] issue matching what I'm seeing

  Setup

• Which version of Git for Windows are you using? 32-bit or 64-bit? Include the
  output of git version as well.

git 2.9.0 with GCM 1.4.0

$ git --version
git version 2.9.0.windows.1

• Which version of Windows are you running? 32-bit or 64-bit?

Windows Server 2008 R2 Datacenter 64-Bit

• What options did you set as part of the installation? Or did you choose the
  defaults?

Installed git for windows and then GCM 1.4.0

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Jenkins is installed on this Server with Apache 8.0.28 64Bit and JDK 1.8.0_74. Jenkins uses c:\Program Files\Git\bin\git.exe for polling our local installation of GitBlit. Around 90 repositories are polled every 15 Minutes. Most of the time it works fine, but sometimes polling of some project hangs and git-credential-manager.exe just waits for the timeout configured in Jenkins (1o Minutes) to be killed. During the next polling after 15 Minutes the same project could be successfully polled so that an authorization issue can be ruled out.
Looking in the Apaches tmp folder we found a git[randomnumber].credentials.lock file for every SCM query. Every hour there are 360 files created. In older Versions this never happened.

I hope this gives some information about this issue. If anything is missing I am more than happy to help and provide further information.

[whoisj]

The GCM doesn't write any .lock files to disk. While it may be involved in some manner, the information provided doesn't help triage the issue.

Can you enable git config --global credential.writelog true and then report back the logs (recorded in the .git/ folder)? Additionally, which host are you seeing this with? GitHub, VSTS, BitBucket, other?

Thanks

[acoberlin]

I could narrow down the Problem:

since Jenkins creates a .gitcredentials file for every query there might be some issues to that.
So I created a global .gitcredentials file which jenkins is using now. So far no credendials.lock files anymore in temp.

But the problem that git-credential-manager hangs still persists. But I found a workaround: Since Jenkins is running 10 simultaneous SCM queries in parallel I have reduced this number to 1 (!).
Now there are no git-credential-manager.exe that are hanging or stalled any more. But as a tradeoff the SCM polling now requires about 2 Minutes of time.

I have enabled the credential.writelog but I am not able to find any logfiles. Where exactly should they be regarding a 64-Bit Windows installation of Git under C:\Program Files\Git?

Thank you very much for your comment & help

[whoisj]

The GCM log files are written into the .git/ folder at the root of the repository's working directory.

The GCM would likely only hang if it were attempting to interact with a user (in the advent of the authentication attempt failing). Have you configured the GCM to be build agent friendly? https://github.com/Microsoft/Git-Credential-Manager-for-Windows#build-agents

[acoberlin]

Yes I did that

Here is my config:

git config -l

core.symlinks=false
core.autocrlf=true
core.fscache=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
help.format=html
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
sendemail.smtpserver=/bin/msmtp.exe
diff.astextplain.textconv=astextplain
rebase.autosquash=true
credential.helper=manager
credential.writelog=true
credential.interactive=never

Still same behaviour: When I set the simultaneous polls to 10 max sometimes GCM hangs. Its arbitrary on which jenkins job.

Our Jenkins polls every 15 minutes. Here's the credentials.log of a project that hangs:

Log Start (2016-07-06 14:16:30Z)
Microsoft Git Credential Manager for Windows version 1.4.0
Program::Store
   targetUri = https://gitserver.domain.local/
Program::CreateAuthentication
   detecting authority type
   authority is basic

The hanging GCM started at 14:30:00Z - nothing appears in the logfile. I have 3 hanging queries as of now. None of them logs anything... Even when the process is killed by jenkins after a timout of 10 Minutes. As you can see the credentials were correct and last SCM query was successful.

[whoisj]

Hmm... that's rather interesting. It looks like the process got as far as to identify the host as basic, after a request to store credentials. This would normally be a Program::Get followed by a Program::Store.

Can you reproduce this without Jenkins in the mix?

Sorry to ask you for additional information but I'm wholly perplexed here.

Another option would be for you to use Windows Task Manager to create a dump of one of the hung GCM processes then make that file available for me to diagnose.

Thanks for the report.

[acoberlin]

Unfortunately I cannot reproduce it without Jenkins because I cannot simply create 90 SCM polls with 10 simultaneous requests without jenkins or any other tool. And since there are only in 50% of the time 3 out of 90 hanging queries I think it is very unlikely that I can reproduce it without Jenkins.

But I could reproduce it on our Jenkins reference-system, that has the same settings and installed software.

I created 3 full and mini-dumps. I am not sure an how to make them available for you since I dont have any access to public ftp or anything similar in my company. I could e-mail them to you though (3MB minidump / 60MB full dump).

[whoisj]

The full dump would be ideal.

I am not sure how to acquire the dumps from you. Let me ask around, there has to be method for this.

[acoberlin]

You should have an e-mail with a wetransfer-link in your mailbox. I have uploaded three mini and full dumps there. Thank you for your efforts and help!

[whoisj]

Yes, I did get the email and the dumps are downloading now.

Thank you very much, I'll update this thread when I have more information for you.

[whoisj]

OK, I've looked at the dumps and there's a logic flaw in the GCM that I'll correct ASAP.

Here's what is happening: you're deadlocking here, because the conditional here is flawed.

I'll get a PR up to fix this asap, thank you very much for all of the assistance.

[acoberlin]

Awesome!
I assumed that there must be some deadlocking with dialogs involved - but I couldn't find out anything...
Thank you for your help and efforts. Looking forward to GCM 1.5.0 ;)

[acoberlin]

Unfortunately the issue still happens when miltiple SCM queries in parallel are done. I can provide you with another full and minidump. Setup is now:
Jenkins 1.656, Windows 2008R2, Git 2.9.2 with GCM 1.5.0 - using CMD for git (not MinTTY).

[whoisj]

Dumps are always welcome.

(reopening)

[acoberlin]

Hi j,

I have sent you another dump via wetransfer.com
Sorry it took so long.

[whoisj]

Thank you and not a problem. I'll try to look at them today.

[whoisj]

Looks as if you're still hitting the modal dialog issue.

Any chance your Git config files are not quite setup the way you think they are?

Regardless, the new environmental variable override feature will solve this for you.

You can now
SET GCM_MODAL_PROMPT False
SET GCM_INTERACTIVE Never
SET GCM_VALIDATE False

and not have to worry about the GCM finding the correct configuration.

Mind you, SET is for the current console, use SETX for permanent changes.

[acoberlin]

Sorry for the delay. I have Set the Environment Variables (in System context):

C:\Windows\system32>set
GCM_INTERACTIVE=Never
GCM_MODAL_PROMPT=False
GCM_VALIDATE=False

And my Git-Config is as following:

C:\Windows\system32>git config -l
core.symlinks=false
core.autocrlf=input
core.fscache=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
help.format=html
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
sendemail.smtpserver=/bin/msmtp.exe
diff.astextplain.textconv=astextplain
rebase.autosquash=true
credential.helper=manager
credential.interactive=never

Entries in gitconfig have been made in C:\Program Files\Git\mingw64\etc\gitconfig as global config File.

Jenkins simultaneous SCM queries: 10

With these settings I still experience the same Problem - or even worse: now the credentials are erased and none of the git requests works anymore. Logfile:

Log Start (2016-08-01 06:38:05Z)
Microsoft Git Credential Manager for Windows version 1.5.0
Program::Erase
   targetUri = https://gitserver.domain.local/
Program::CreateAuthentication
   detecting authority type
   authority is basic
   deleting basic credentials

Setting everything back to "normal" (1 simultaneous SCM query in jenkins and readding Credentials once) works fine - no hanging processes. I could provide another Dump files, if needed.

[whoisj]

@acoberlin the Environment Variable functionality won't be available until v1.6 is released, did you recompile the GCM for your testing?

[whoisj]

@acoberlin please retest with v1.6.0 - thanks

[acoberlin]

Hi whoisj,

sorry - it took me a whole to test.
At first things looked good. But it only worked for three days. Then it began again to stuck.
The reason is: somehow GCM seems to loose the credentials. If I do a git request with credentials while some other request hangs, all the other requests worke fine then.
If not then none of the requests will be processed correctly. I could reproduce this behaviour twice.
I have set GIT_ASKPASS=never - still hanging processes.

Maybe GCM is completely right to ask for credentials simply because there are none to find anymore when this occurs. Unfortunately I cannot pinpoint the exact time/number of requests when it looses the credentials. But it is certainly a starting point.

Workaround is still to have only one concurrent git request simultaneously.

[whoisj]

@acoberlin since v1.7.0 the GCM has better logging available.

I recommend that you setx GIT_TRACE c:\.trace\git.log and setx GCM_TRACE c:\.trace\git.log. The combined logs might give you enough information to finally root cause and resolve this issue.

[grofit]

Not sure if this is related but I am trying to remove my credentials from someone elses computer and I am using the latest git for windows (with I assume the latest credentials manager) however when I run:

git credential-manager delete <some-url>

It just hangs, even if I go to the credentials manager in windows and remove it from there, it just comes back in if I try to do anything with that repo again.

[whoisj]

@grofit that's a separate bug, and thanks for the report!

[whoisj]

-- issue clean up - @acoberlin to re-open if still an issue --