git fetch performance regression 2.12.2.windows.2 -> 2.13.0.windows.1

[AdrianIssott]

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

The slow version is:

$ git --version --build-options

git version 2.13.0.windows.1
built from commit: eba7af3dbb4c846c6303c5f64102acee696c9ab0
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.15063]

• What options did you set as part of the installation? Or did you choose the
  defaults?

For both versions of git I choose the defaults. Here are the results for 2.13.0.windows.1:

> type "C:\Program Files\Git\etc\install-options.txt"

Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Not that I'm aware of.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Admin CMD.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

* Delete the git workspace
* git clone <URL to a large repo>
* git fetch --tags origin +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*

• What did you expect to occur after running these commands?

The git fetch to take approximately as long when running either version of git.

• What actually happened instead?

git 2.13 was significantly slower than git 2.12. The fetch pulled down some 60,000 references in a couple of minutes on git 2.12 whilst for 2.13 I gave up waiting after two hours. The issue reproduces reliably for me.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

Sorry no.

[dscho]

• If the problem was occurring with a specific repository, can you provide the URL to that repository to help us with testing?

Sorry no.

Please note that this harms your wish to get help.

But you can dig deeper yourself: there have been 5 intermediate snapshots on https://wingit.blob.core.windows.net/files/index.html, maybe you can pinpoint a bit finer where the regression happened.

Other than that, the report does not even mention which protocol was used for transport. That's good for me, because I do not have to set aside time to try to dig deeper, as crucial information is simply unavailable, so I can go take care of other tasks instead. That's bad for you, though, as you do not get the assistance you seek.

Assuming that you use the most prevalent protocol (HTTPS): by setting GIT_TRACE_CURL=1 before running git fetch, you will see a lot more fine-grained information. As you are unwilling to share the URL to your repository, I assume that there will be sensitive information in that trace, therefore you will have to try to analyze it yourself.

Good luck, and good hunting!

[AdrianIssott]

Many thanks for trying to help given I'm not able to provide a URL for you to test against. I can say the URL is of the form [email protected]:<org>/<repo>.git. As a result we aren't using HTTPS so when I tried setting GIT_TRACE_CURL=1 I didn't get any additional output from the git fetch.

However, I did find test those intermediate snapshots and found:

• Git-prerelease-2.12.2.windows.2.1.g7df5db8d31-64-bit.exe: Fast
• Git-prerelease-2.12.2.windows.2.406.gd14a8f8640-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.408.g91e25e0ef0-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.797.g7a2040a790-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.799.g9f22cae658-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.800.gede8f145e0-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.2276.geba7af3dbb-64-bit.exe Slow

Is there any other information I can dig out to help move this along?

[dscho]

• Git-prerelease-2.12.2.windows.2.800.gede8f145e0-64-bit.exe Fast
• Git-prerelease-2.12.2.windows.2.2276.geba7af3dbb-64-bit.exe Slow

Is there any other information I can dig out to help move this along?

The next steps to narrow things down would be to see what the changes are between the commits from which those two versions were built. You can find out by calling git version --build-options with both versions.

Other than that, there is always a chance that an updated dependency (in this case, most likely openssh) caused the regression. You could find out by comparing the files /etc/package-versions.txt from both versions.

[bbolli]

Might be related to https://public-inbox.org/git/[email protected]/T/#t

[jiez]

I saw the same issue when working with one of our internal repositories, which has about 2000 tags.

mkdir test
cd test
git init .
git -c core.askpass=true fetch --tags --progress ssh://GIT-REPOSITORY-URL "+refs/heads/:refs/remotes/origin/"

For 2.12.2.windows.2 64bit or earlier versions, it only takes about 30s. But for 2.13.3.windows.1, it takes more then 18min.

[jiez]

Same version of git for Linux does not have the same issue. So it seems it's Windows specific.

[dscho]

@jiez you probably want to do some digging on your own (as outlined in earlier comments), as it seems that you are also unable to provide a minimal, complete & verifiable example?

[jiez]

mkdir test
cd test
git init .
time git fetch --tags https://github.com/jiez/test-git-for-windows-1233.git

The above test uses about 5m50s for 2.13.x and 2.14.x, but only about 15s for 2.12.x.

[jiez]

It's caused by this commit:

https://github.com/git/git/commit/524a9fdb51b1c7de5bcb83f6ea44015dabef466c

commit 524a9fdb51b1c7de5bcb83f6ea44015dabef466c
Author: Michael Haggerty [e-mail]
Date: Sun Apr 16 08:41:27 2017 +0200

refs_verify_refname_available(): use function in more places

Change `lock_raw_ref()` and `lock_ref_sha1_basic()` to use
`refs_verify_refname_available()` instead of
`verify_refname_available_dir()`. This means that those callsites now
check for conflicts with all references rather than just packed refs,
but the performance cost shouldn't be significant (and will be
regained later).

These were the last callers of `verify_refname_available_dir()`, so
also delete that (very complicated) function.

Signed-off-by: Michael Haggerty <[email protected]>
Signed-off-by: Junio C Hamano <[email protected]>

[jiez]

I tested on another machine using the method I mentioned in one of my previous comments.
On Windows,
without this commit: 3s
with this commit: 1min
On Linux,
without this commit: 1s
with this commit: 4.5s

[dscho]

It seems that this issue is not at all Windows-specific, and that a patch to fix things was presented here: https://public-inbox.org/git/[email protected]/t/#m146140068886c31b94a3c5603158c522784f30b1

Could you verify that this fixes the problem?

[dscho]

To make it easier for you (cherry-picking the commit in pu succeeds, but the result does not compile), I provided a test merge that you can easily get into your Git clone via

git fetch https://github.com/dscho/git tentative-fix-for-1233
git checkout FETCH_HEAD

[jiez]

With that fix, I got 5.3s on Windows, not as fast as before, but is acceptable.

[jiez]

That patch has been merged into upstream

https://github.com/git/git/commit/8ec617c80cc37dcd66821f89678fc8797af96f10

[dscho]

Thanks for testing. I am weighing whether we should merge this branch early, i.e. before the v2.15.x track. Opinions?

[jiez]

I am not familiar with git development. Will the next 2.14.x release include this fix? I took a quick look at how releases are cut on upstream git project. It seems each release is just made from the master branch. So the next 2.14.x release will include it. If so, we can just wait for the next 2.14 release. I currently just stick to 2.12.2.

[dscho]

Upstream Git releases are cut from the master branch for *.0 versions, i.e. the next time master will serve as a base for a Git version is v2.15.0. Maintenance releases such as v2.14.1, v2.14.2, etc are cut from maint.

In that light: no, unless we specifically merge tentative-fix-for-1233, Git for Windows next version (which will almost certainly come before October 27th, the tentative date for Git v2.15.0) won't have the fix.

[dscho]

I made up my mind. Waiting for over two months to see this regression fixed is way too long a time.

From the upstream twitches, it looks as if v2.14.2 is relatively close (even if the upstream Git maintainer dropped no hint about v2.14.2), therefore I will hold off with a new version for now.

I want to ask everybody who experienced this problem to test the new snapshot, though: https://wingit.blob.core.windows.net/files/index.html

[AdrianIssott]

I just tried the 37d08f901a snapshot which contains the fix in 121eceb9767da4f76a24a5c5f8bf426ca561645e and it worked for me. Fetch performance was distinctly faster that it was with 2.13.0.windows.1.

Many thanks!

[marcinsmialek]

Here are my results with @jiez's test - https://github.com/git-for-windows/git/issues/1233#issuecomment-321001668

git version 2.14.1.windows.1
real    0m49.083s
user    0m0.000s
sys     0m0.000s

git version 2.14.1.windows.1.48.g37d08f901a
real    0m4.934s
user    0m0.000s
sys     0m0.015s

That's a huge difference if you ask me.

[dscho]

Thank you for the feedback! Glad that it made things faster again.