git stash very slow

[dacmot]

All variants of the git stash command ('nothing', list, show, pop, drop, etc.) are very slow (several seconds). All other commands are very fast (under a second at most).

This issue is very similar to https://github.com/msysgit/git/issues/259. Since msysgit seems to be deprecated I thought I'd open a new one here. I found a couple of questions on stackoverflow where others seem to be experiencing the same issue

• https://stackoverflow.com/questions/44159850/git-stash-on-windows-extremly-slow-compared-to-libgit2
• https://stackoverflow.com/questions/37529016/git-stash-is-slow-on-windows

I have tried all the tips in the following links (see git config -l below). Initially I noticed that using Git CMD instead of Git Bash solved most performance issues. Adding export PS1='$ ' helped Git Bash a lot, but startup is still slow. Nothing seemed to help git stash much though.

• https://github.com/git-for-windows/git/wiki/Diagnosing-performance-issues
• https://stackoverflow.com/questions/37529016/git-stash-is-slow-on-windows
• https://stackoverflow.com/questions/4485059/git-bash-is-extremely-slow-in-windows-7-x64
• https://stackoverflow.com/questions/2835775/msysgit-bash-is-horrendously-slow-in-windows-7

Setup

Just did a fresh install of Git 2.13.3 with LFS (uninstalled, reboot, install). Has Git 2.11.1 before with the same issue.

$ git --version --build-options
git version 2.13.3.windows.1
built from commit: faaf2320f32f6ac52a69502c6b157b6ea4781f50
sizeof-long: 4
machine: x86_64

Running Windows 7 Enterprise 64bit SP1. My .gitconfig is located on a local SSD, and so are all my repositories. I would expect all commands except push, fetch and pull to be lightning fast.

$ cmd.exe /c ver

Microsoft Windows [Version 6.1.7601]

Install options:

$ cat /etc/install-options.txt
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

Here is my git config:

$ git config -l
core.symlinks=false
core.autocrlf=true
core.fscache=true
color.diff=auto
color.status=auto
color.branch=auto
color.interactive=true
help.format=html
diff.astextplain.textconv=astextplain
rebase.autosquash=true
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.required=true
filter.lfs.process=git-lfs filter-process
credential.helper=manager
alias.lg=!git lg2
alias.lg1=!git lg1-specific --all
alias.lg2=!git lg2-specific --all
alias.lg3=!git lg3-specific --all
alias.lg1-specific=log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(auto)%d%C(reset)'
alias.lg2-specific=log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(auto)%d%C(reset)%n''          %C(white)%s%C(reset)%n''          %C(dim white)@ %an%C(reset)'
alias.lg3-specific=log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset) %C(bold cyan)(committed: %cD)%C(reset) %C(auto)%d%C(reset)%n''          %C(white)%s%C(reset)%n''          %C(dim white)- %an <%ae> %C(reset) %C(dim white)(committer: %cn <%ce>)%C(reset)'
alias.di=difftool -d -g -symlinks
alias.me=mergetool -t bc3
alias.st=status -s -b
alias.br=branch -a -vv
alias.pr=!f() {    git fetch -p;     git checkout master;    git pull;     git branch -vv | perl -nae 'system(qw(git branch -d), $F[0]) if $F[3] eq q{gone]}';     }; f
alias.pb=!f(){    git rev-parse --abbrev-ref HEAD | perl -nae 'system(qw(git push --set-upstream origin), $F[0])';     }; f
user.name=dacmot
user.email=...email...
core.preloadindex=true
core.fscache=true
core.ignorestat=true
core.symlinks=true
core.autocrlf=true
core.safecrlf=true
gc.auto=256
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.repositoryformatversion=0
core.filemode=false
core.bare=false
core.logallrefupdates=true
core.ignorecase=true

Details

• Both git CMD and git bash have the same issue. Using bash here to show total execution time.
  
  
  
  • It looks like a combination of a lot processes and git-stash list taking ~3 seconds. When looking at process explorer, there a lot of children sh.exe and git.exe processes being spawned 3-4 levels deep, and multiple times.
  
  
  • Even on an empty repository and no remotes. Stash is a local command anyway so I'm stumped why it would take so long, even just to list nothing.
  
  
  • I also tried on a different computer (4-core laptop instead of 12-core desktop, same OS, v2.8) with similar results.
  
  
  • As mentioned earlier, everything (.gitconfig, Program Files/Git, repositories) is on a local SSD drive.
  
  

$ mkdir testrepo
$ cd testrepo
$ git init
$ time `GIT_TRACE=true git stash list`
14:10:18.893173 git.c:547               trace: exec: 'git-stash' 'list'
14:10:18.893673 run-command.c:626       trace: run_command: 'git-stash' 'list'
14:10:21.582173 git.c:547               trace: exec: 'git-sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
14:10:21.582673 run-command.c:626       trace: run_command: 'git-sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
14:10:21.979173 git.c:547               trace: exec: 'git-sh-i18n--envsubst' 'usage: $dashless $USAGE'
14:10:21.979673 run-command.c:626       trace: run_command: 'git-sh-i18n--envsubst' 'usage: $dashless $USAGE'
14:10:22.907173 git.c:322               trace: built-in: git 'rev-parse' '--git-dir'
14:10:23.510173 git.c:322               trace: built-in: git 'rev-parse' '--git-path' 'objects'
14:10:24.277173 git.c:322               trace: built-in: git 'rev-parse' '--show-prefix'
14:10:24.542173 git.c:322               trace: built-in: git 'rev-parse' '--show-toplevel'
14:10:24.856173 git.c:322               trace: built-in: git 'rev-parse' '--git-path' 'index'
14:10:25.184173 git.c:322               trace: built-in: git 'config' '--get-colorbool' 'color.interactive'
14:10:25.480173 git.c:322               trace: built-in: git 'rev-parse' '--verify' '--quiet' 'refs/stash'

real    0m7.218s
user    0m0.046s
sys     0m0.078s
$ time `GIT_TRACE_PERFORMANCE=true git stash list`
14:09:20.479673 trace.c:435             performance: 0.023168150 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' '--exec-path'
14:09:22.034173 trace.c:435             performance: 0.165639637 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
14:09:22.429173 trace.c:435             performance: 0.141833128 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' 'usage: $dashless $USAGE'
14:09:23.257673 trace.c:435             performance: 0.023435518 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-dir'
14:09:23.857173 trace.c:435             performance: 0.023190146 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'objects'
14:09:24.625173 trace.c:435             performance: 0.024125201 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-prefix'
14:09:24.937673 trace.c:435             performance: 0.023932128 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-toplevel'
14:09:25.202173 trace.c:435             performance: 0.022960415 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'index'
14:09:25.458673 trace.c:435             performance: 0.021897785 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'config' '--get-colorbool' 'color.interactive'
14:09:25.768173 trace.c:435             performance: 0.026290051 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
14:09:26.063173 trace.c:435             performance: 7.028777991 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'stash' 'list'

real    0m7.403s
user    0m0.078s
sys     0m0.109s

as opposed to (there's some weird errors tossed in there, but commands seem to execute anyway)

$ touch file.txt
$ time `GIT_TRACE_PERFORMANCE=true git add file.txt`
14:16:21.266248 trace.c:435             performance: 0.103129281 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'add' 'file.txt'

real    0m0.406s
user    0m0.000s
sys     0m0.140s
$ time `GIT_TRACE_PERFORMANCE=true git commit -m "Add file"`
14:17:17.700033 trace.c:435             performance: 0.220335711 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'commit' '-m' 'Add file'
bash: [master: command not found

real    0m0.742s
user    0m0.046s
sys     0m0.170s
$ time `GIT_TRACE_PERFORMANCE=true git log`
14:18:38.590208 trace.c:435             performance: 0.057085767 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'log'
bash: commit: command not found

real    0m0.565s
user    0m0.015s
sys     0m0.234s
$ git log
commit 3d22327861884ce8059ef368c19456888deee55e (HEAD -> master)
Author: dacmot <...email...>
Date:   Thu Jul 13 14:17:17 2017 -0400

    Add file

$ time `GIT_TRACE_PERFORMANCE=true git branch -vv`
14:20:39.576900 trace.c:435             performance: 0.071054403 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' 'branch' '-vv'
bash: file.txt: command not found

real    0m0.581s
user    0m0.015s
sys     0m0.202s
$ git branch -vv
* master 3d22327 Add file

Let me know if I can provide more information, or if (how) I can dig up this issue further.

[shabeepk]

I am facing the same issue. And it seems to be slow regardless of the repository size, file count and history.

[dscho]

git stash is implemented as a shell script. That's what makes it slow on Windows, as it has to spawn processes left and right.

[shabeepk]

The performance has suddenly gone down, it wasn't like this a few releases before, I tried downgrading to the 2.12 release but still not effect.

I am pretty sure something has gone wrong recently, as stash has always been a shell script and such a sudden decrease in performance is obviously not because of the fact that it is a shell script.

[dacmot]

@dscho I see. I'm puzzled why https://github.com/msysgit/git/issues/259 was closed then. I will try older versions to see if it is a regression as @shabeepk suggests.

I found https://github.com/git-for-windows/git/pull/508 which seems to want to address it. As per your comments, there is also work being done upstream.

[dscho]

I'm puzzled why msysgit#259 was closed then.

Be puzzled no more: msysGit was retired (I was a bit sick and tired that people misused the name to refer to Git for Windows all the time, when it was really the SDK to build Git for Windows). So i closed all the tickets.

As to the performance suddenly going down, I am a bit puzzled about that, and would be very grateful if you could spend the time to bisect this, as I have no time to work on this ticket (or pretty much any other ticket on this tracker) for the coming week at least.

[shabeepk]

One other thing to note is that the performance has gone down across repositories (from what I can see), and it has gone down regardless of where the repository resides as well. All of them are on SSDs...

So either it is a problem related to the main installation, the release itself, or something else that I can't even think about 😄

[dscho]

As to the performance suddenly going down, I am a bit puzzled about that, and would be very grateful if you could spend the time to bisect this

I would still be very grateful about that. There is simply no substitute for bisecting this. And as I stated, my time is occupied by other things.

[dacmot]

I understand very well the time constraints. I have not had time to do this yet. The only windows machine I have access to is my work computer and it really has not been a good time to start messing up with my git installation. I'll get to it eventually...

[linquize]

Does busybox-based git help?
https://github.com/git-for-windows/git/wiki/MinGit#experimental-busybox-based-mingit

[dscho]

@linquize BusyBox-based MinGit is still very experimental. And it is MinGit-only.

[marcinsmialek]

Git version 2.14.0.windows.2 over here
Regular:

$ time `GIT_TRACE_PERFORMANCE=true git stash list`
08:17:30.108534 trace.c:435             performance: 0.000014348 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' '--exec-path'
08:17:30.881145 trace.c:435             performance: 0.065384250 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
08:17:31.080863 trace.c:435             performance: 0.064181276 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' 'usage: $dashless $USAGE'
08:17:31.514970 trace.c:435             performance: 0.002313441 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-dir'
08:17:31.795047 trace.c:435             performance: 0.002013263 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'objects'
08:17:32.168782 trace.c:435             performance: 0.002935317 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-prefix'
08:17:32.302843 trace.c:435             performance: 0.002353465 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-toplevel'
08:17:32.439476 trace.c:435             performance: 0.002247742 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'index'
08:17:32.572492 trace.c:435             performance: 0.002856402 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'config' '--get-colorbool' 'color.interactive'
08:17:32.709921 trace.c:435             performance: 0.002977229 s: git command: 'C:\Program Files\git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
08:17:32.858577 trace.c:435             performance: 3.408056301 s: git command: 'C:\Program Files\git\mingw64\bin\git.exe' 'stash' 'list'

real    0m3.546s
user    0m0.000s
sys     0m0.062s

MinGit:

$ time `GIT_TRACE_PERFORMANCE=true /c/tools/MinGit/cmd/git.exe stash list`
08:18:14.885097 trace.c:435             performance: 0.000021900 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' '--exec-path'
08:18:15.663268 trace.c:435             performance: 0.077651867 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
08:18:15.863623 trace.c:435             performance: 0.063006242 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'sh-i18n--envsubst' 'usage: $dashless $USAGE'
08:18:16.315430 trace.c:435             performance: 0.002394621 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--git-dir'
08:18:16.583204 trace.c:435             performance: 0.002962881 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--git-path' 'objects'
08:18:16.944649 trace.c:435             performance: 0.002846963 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--show-prefix'
08:18:17.071472 trace.c:435             performance: 0.002701216 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--show-toplevel'
08:18:17.213124 trace.c:435             performance: 0.002885099 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--git-path' 'index'
08:18:17.338163 trace.c:435             performance: 0.002989689 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'config' '--get-colorbool' 'color.interactive'
08:18:17.467285 trace.c:435             performance: 0.004548723 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
08:18:17.604570 trace.c:435             performance: 3.380032214 s: git command: 'C:\tools\MinGit\mingw64\bin\git.exe' 'stash' 'list'

real    0m3.583s
user    0m0.000s
sys     0m0.062s

They're pretty much the same in terms of speed.

[dscho]

@dacmot installing Git for Windows' SDK, compiling & bisecting Git leaves your existing Git installation alone. One less excuse 😃

[dacmot]

There is a small but noticeable difference starting at 2.10. I'm noticing more commands executed in the trace and slightly longer execution time for the stash command itself which takes the bulk of the whole time.

It is however still an order of magnitude slower than other commands, even in what should be the simplest case (list of an empty stash in an empty repository).

$ git --version --build-options
git version 2.9.3.windows.2
sizeof-long: 4
$ time `GIT_TRACE_PERFORMANCE=true git stash list`
11:59:33.323370 trace.c:420             performance: 0.026721893 s: git command: 'git.exe' 'rev-parse' '--git-dir'
11:59:34.012870 trace.c:420             performance: 0.025457405 s: git command: 'git.exe' 'rev-parse' '--git-path' 'objects'
11:59:34.930870 trace.c:420             performance: 0.025980405 s: git command: 'git.exe' 'rev-parse' '--show-toplevel'
11:59:35.312870 trace.c:420             performance: 0.027458003 s: git command: 'git.exe' 'rev-parse' '--git-path' 'index'
11:59:35.700870 trace.c:420             performance: 0.032429439 s: git command: 'git.exe' 'config' '--get-colorbool' 'color.interactive'
11:59:36.085870 trace.c:420             performance: 0.031969492 s: git command: 'git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
11:59:36.414870 trace.c:420             performance: 5.614246234 s: git command: 'git.exe' 'stash' 'list'

real    0m5.958s
user    0m0.000s
sys     0m0.171s

$ git --version --build-options
git version 2.10.0.windows.1
sizeof-long: 4
machine: x86_64
$ time `GIT_TRACE_PERFORMANCE=true git stash list`
14:33:08.627370 trace.c:435             performance: 0.170253210 s: git command: 'git.exe' 'sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
14:33:09.115870 trace.c:435             performance: 0.174124878 s: git command: 'git.exe' 'sh-i18n--envsubst' 'usage: $dashless $USAGE'
14:33:10.069370 trace.c:435             performance: 0.026764906 s: git command: 'git.exe' 'rev-parse' '--git-dir'
14:33:10.782870 trace.c:435             performance: 0.025988715 s: git command: 'git.exe' 'rev-parse' '--git-path' 'objects'
14:33:11.717370 trace.c:435             performance: 0.026028306 s: git command: 'git.exe' 'rev-parse' '--show-toplevel'
14:33:12.105870 trace.c:435             performance: 0.026355793 s: git command: 'git.exe' 'rev-parse' '--git-path' 'index'
14:33:12.491870 trace.c:435             performance: 0.031842408 s: git command: 'git.exe' 'config' '--get-colorbool' 'color.interactive'
14:33:12.876370 trace.c:435             performance: 0.030773435 s: git command: 'git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
14:33:13.209870 trace.c:435             performance: 7.351044142 s: git command: 'git.exe' 'stash' 'list'

real    0m7.699s
user    0m0.015s
sys     0m0.187s

[dacmot]

And it might be getting worse:

`` $ git --version --build-options git version 2.14.1.windows.1 built from commit: 82d9b3f3b2407b52251620597d4b14933685459d sizeof-long: 4 machine: x86_64 $ timeGIT_TRACE_PERFORMANCE=true git stash list`
15:25:56.481411 trace.c:435 performance: 0.021421505 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' '--exec-path'
15:25:58.178750 trace.c:435 performance: 0.156245112 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' '--variables' 'usage: $dashless $USAGE'
15:25:58.615337 trace.c:435 performance: 0.154385772 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'sh-i18n--envsubst' 'usage: $dashless $USAGE'
15:25:59.559026 trace.c:435 performance: 0.020936141 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-dir'
15:26:00.211657 trace.c:435 performance: 0.022307184 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'objects'
15:26:01.036822 trace.c:435 performance: 0.020815900 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-prefix'
15:26:01.366387 trace.c:435 performance: 0.022007070 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--show-toplevel'
15:26:01.702955 trace.c:435 performance: 0.021160983 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--git-path' 'index'
15:26:02.043523 trace.c:435 performance: 0.026478967 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'config' '--get-colorbool' 'color.interactive'
15:26:02.383591 trace.c:435 performance: 0.025933482 s: git command: 'C:\Program Files\Git\mingw64\libexec\git-core\git.exe' 'rev-parse' '--verify' '--quiet' 'refs/stash'
15:26:02.704655 trace.c:435 performance: 7.647866843 s: git command: 'C:\Program Files\Git\mingw64bin\git.exe' 'stash' 'list'

real 0m7.994s
user 0m0.062s
sys 0m0.156s

[whoisj]

@dscho when did the certificate that signed Git for Windows change? It is possible that there are anti-malware scanners at fault here. I'm asking about the cert because, I believe, many of them had recognized the need for allowing Git for Windows to operate smoothly and left binaries signed with it.

At some point the certificate changed. It could be starting with that change that some/all anti-malware software stopped recognizing Git for Windows.

I know that Microsoft and Symantec worked together to get signed Git for Windows binaries to go harassed in order to optimize performance.

[dscho]

when did the certificate that signed Git for Windows change?

That was on June 13th this year. Symantec had promised to update their white-listing, but it is possible that their process is even slower than I expected. I'll ping them.

[whoisj]

There a dozen other providers as well, all of whom may or may not have realized that the certificate has been changed. I can help track this down if we think it might be the root cause.

Which release was the last signed by the old cert?

[dscho]

I am fairly certain that v2.13.0 was the last release with the old signature, and v2.13.1 was the first release with the new one.

[whoisj]

@dacmot does the v2.13.0 vs v2.13.1 demarcation align with your experience? I'm specifically referring to the "and it may be getting worse" statement you made a few post up - thanks.

[dacmot]

I don't think so. I didn't check these two specifically though. I started bisecting from 2.5. The difference between 2.10 and 2.14 was only 0.3s, and I would justify it more by the extra commands in the trace than by a delisting with antivirus software.

Speaking of antivirus software, we use trend micro. It's possible it's playing a role overall, but our IT department keeps a whitelist internally so I would be surprised if they didn't add it on. What makes me doubt it's the problem is that stash is the only command that is significantly slower.

[Peter-Optiway]

git flow command are also slow, we also use trend micro as Antivirus.

[linquize]

git flow is also written in shell script, so it is slow

[dscho]

It is funny, my numbers are very different. Granted, my machine is a nice 16-core one and not exactly a sloth (neither in terms of speed nor in terms of cuteness). But still, the difference should not be that much:

me@work MINGW64 /
$ git version
git version 2.14.1.windows.1

me@work MINGW64 /
$ GIT_TRACE_PERFORMANCE=1 git --exec-path
12:05:36.341076 trace.c:435             performance: 0.000007698 s: git command: 'D:\test-git\Portable-2.14.1-64\mingw64\bin\git.exe' '--exec-path'
D:\test-git\Portable-2.14.1-64\mingw64/libexec/git-core

This is on my D: drive, which is a spinning disk. The performance is obscenely faster on my SSD.

What is really funny, however, is that my test with Git for Windows v2.10.0 shows that v2.14.1 is faster, i.e. the opposite of your finding:

me@work MINGW64 /
$ git version
git version 2.10.0.windows.1

me@work MINGW64 /
$ GIT_TRACE_PERFORMANCE=1 git --exec-path
D:\test-git\Portable-2.10.0\mingw64/libexec/git-core
12:05:21.073595 trace.c:435             performance: 0.000020894 s: git command: 'git.exe' '--exec-path'

This test allowed Windows Defender to do its thing, so it may be even faster if I would turn the antivirus off.

Of course, all of my experiments happen on Windows 10 (with Creators Update, and soon with Fall Creators Update), even when I test in VMs because Windows 10 is so much faster and more robust than Windows 7 in my hands.

Maybe there is a chance for you to experiment just with the performance of git --exec-path (because it is quicker to test than git stash, and it would appear from your traces that the performance issues you experience may very well stem from process startup being slow in general), and maybe you can experiment with and without antivirus doing its thing?

It is also entirely possible that your setup had WFP (Windows File Protection) turned on, and for some reason does not play well with my code signature.

Also, Process Monitor may be able to shed some light into your issue. Maybe, just maybe, there is something on your machine that injects itself into every process (some graphics drivers have been known to do that, for example).

[dacmot]

$ GIT_TRACE_PERFORMANCE=1 git --exec-path
15:05:10.417448 trace.c:435             performance: 0.006657777 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' '--exec-path'
C:\Program Files\Git\mingw64/libexec/git-core

I doubt my machine is orders of magnitude slower than yours... I have 12-cores at 2.4GHz, 32 GB of ram (no pagefile) and an SSD. Still that seemed reasonably fast. Like I mentioned before, git stash seems to be the only unreasonably slow command.

I tried procmon on git.exe --exec-path but I'm not sure what I'm looking for... There seems to be a lot (about 2/3 of the 2371 lines of log) of registry access to HKCR\MIME\Database\Charset. Some references to digital guardian which could be the issue (some sort of data protection garbage forced on us). We've had issues with it in the past although it mostly seems to be behaving lately. I don't have any other programs that seem unreasonably slow. And why just git stash?

[dacmot]

$ GIT_TRACE_PERFORMANCE=1 git --exec-path
15:05:10.417448 trace.c:435             performance: 0.006657777 s: git command: 'C:\Program Files\Git\mingw64\bin\git.exe' '--exec-path'
C:\Program Files\Git\mingw64/libexec/git-core

I doubt my machine is orders of magnitude slower than yours... I have 12-cores at 2.4GHz, 32 GB of ram (no pagefile) and an SSD. Still that seemed reasonably fast. Like I mentioned before, git stash seems to be the only unreasonably slow command.

I tried procmon on git.exe --exec-path but I'm not sure what I'm looking for... There seems to be a lot (about 2/3 of the 2371 lines of log) of registry access to HKCR\MIME\Database\Charset. Some references to digital guardian which could be the issue (some sort of data protection garbage forced on us). We've had issues with it in the past although it mostly seems to be behaving lately. I don't have any other programs that seem unreasonably slow. And why just git stash?

[dscho]

And why just git stash?

No idea ;-)

[whoisj]

Git stash is mostly shell script, and it very well could contain non-ASCII characters in it. If it does, the parser might start loading localization tables (hence scans of HKCR\MIME\Database\Charset). I've seen many customers being burned by repeated registry scans getting flagged by antimalware and thus everything slows to a crawl.

I'm not naming any antimalware vendors here, but I've seen this with big name mega corp vendors, and the smaller ones too. Seems like registry scanning is a thing "bad" software does and determining intent is a difficult thing to do.

[mercurykd]

Is it forever?

[dscho]

@mercurykd there have been patches sent to the Git mailing list to turn git stash into a builtin. They needed some help, and if I was not so busy trying to address issues on this here bug tracker, I would have helped those patches already.

Read: the more you help out here on the Git for Windows bug tracker, the more I can focus on e.g. helping git stash become a fast builtin, implemented in performant, portable C.

[zhouxq555]

Is this problem solved?
I also encountered this problem

[dscho]

@zhouxq555 No, this problem is not yet solved.

But hope springs eternally! The patch series to start making git stash a proper builtin (it is a Unix shells script right now, which requires a lot of emulation to even run on Windows) is still alive and kicking: https://public-inbox.org/git/[email protected]/. If you are a decent software developer with even rudimentary understanding of Git's inner working, you can help that effort by reviewing those patches. Otherwise, I am afraid that you are out of control here and simply have to wait it out.

[mercurykd]

Is there a solution that emulated git windows through git in linux subsystem on windows?

[whoisj]

Is there a solution that emulated git windows through git in linux subsystem on windows?

I do not see how this would help, can you elaborate?

[mercurykd]

This is a question not related to the main topic ... just saw the solution to run the git client via git in bash on windows (linux subsystem), by swapping paths through the bat script, is this real?

[fladnagg]

Hi, I also have performance issue with Git for windows (all git commands, not only stash), and after many tests I finally found a workaround.

with PowerShell (win 7 enterprise, 64b SP1) :

Measure-Command { Git-2.22\cmd\git --version }
...
TotalMilliseconds : 2970,0967

After analysis with ProcessMonitor, I saw a lot of call to IRP_MJ_DIRECTORY_CONTROL from F750E6C3-38EE-11D1-85E5-00C04FC295EE, which is WFP (Windows File Protection)

In release 2.9.3.2 (https://github.com/git-for-windows/git/releases/tag/v2.9.3.windows.2) git executable was signed, for performance issue with WFP.
and... it's a lot better with this version : 90ms instead of 2900ms !

But, as said before here, in 2.13.1, git has been signed with a new certificate... with again performance issue.

I have benchmark the git --version command with some versions :

2.9.3     ~2900 ms
2.9.3.2  ~90 ms
2.13.0   ~70 ms
2.13.1   ~2900 ms
2.22.0   ~2900 ms

Now, for stash, as you said, it's a shell script... with lots of calls to the git command. If each git command take ~3s, this may explain the performance problem.

In my case, I wanted to write client hooks, with also a lot of git calls.

So... please fix the certificate. In the meantime, the workaround is to use a version between 2.9.3.2 and 2.13.0

[PhilipOakley]

Please confirm which git version you are using, and which certificate is being referred to.

I know that the new v2.23.rc version has an updated code sign cert and that Defender hadn't picked up that cert as valid yet, so it may be related (e.g. old cert may have had a time out?)

Otherwise dunno.

[dscho]

please fix the certificate.

You're funny. The old code-signing certificate is no longer valid, I have to get a new one every year. The signatures of already-signed executables are timestamped, so they still validate okay, but if I tried to use the same certificate to sign binaries _today_, it would _not_ result in a valid signature.

Now, for stash, as you said, it's a shell script...

Did you notice that there is an experimental built-in version of git stash now? There was an option on the last installer page to opt into that. But nowadays it is the default, unless you specifically turn it off using the config setting stash.useBuiltin = false.

[marcinsmialek]

Please, avoid personal attacks.
Anyway, I can recommend the built-in version of stash. It worked great for me since the first experimental release, even on corporate machines with lots of av / monitoring software.

[dscho]

Okay, I'll close this with the recommendation that everybody who experiences a slow git stash should turn on the built-in stash either by reinstalling and checking that option on the last installer page, or by calling git config --global stash.usebuiltin true.