Edit by @tarsius: The cause of this has been found and it has been fixed on Emacs' master branch. See this for more information, including how to get a backport of that fix.
Edit by @tarsius: The fix is included in Emacs 26.1.
"2.9.0-34-gfee9c17c"
GIT: 2.11.0
magit-stage of a single file with a 5-line diff is perceivably slow, around 1 second.
magit-unstage also suffers from the same perceived slowdown.
Some profiling data:
- Stage
magit-stage 1 0.8872 0.8872
magit-stage-1 1 0.887063 0.887063
magit-run-git 1 0.885151 0.885151
magit-process-file 39 0.855452 0.0219346666
magit-refresh 1 0.669937 0.669937
magit-refresh-buffer 1 0.668759 0.668759
magit-status-refresh-buffer 1 0.666563 0.666563
magit-git-insert 14 0.491143 0.0350816428
magit-git-str 30 0.3144180000 0.0104806000
magit-rev-parse-safe 28 0.2852569999 0.0101877499
magit-git-wash 4 0.225156 0.056289
magit-call-git 1 0.215206 0.215206
magit-toplevel 16 0.2069670000 0.0129354375
magit-call-process 1 0.195608 0.195608
magit-git-items 2 0.1731359999 0.0865679999
- Unstage
magit-unstage 1 0.943516 0.943516
magit-unstage-1 1 0.943387 0.943387
magit-run-git 1 0.924936 0.924936
magit-process-file 44 0.912173 0.0207312045
magit-refresh 1 0.666068 0.666068
magit-refresh-buffer 1 0.665038 0.665038
magit-status-refresh-buffer 1 0.662804 0.662804
magit-git-insert 14 0.485324 0.0346659999
magit-git-str 35 0.3729509999 0.0106557428
magit-rev-parse-safe 33 0.3445889999 0.0104420909
magit-call-git 1 0.258859 0.258859
magit-toplevel 17 0.2507959999 0.0147527058
magit-call-process 1 0.243106 0.243106
magit-git-wash 4 0.207833 0.05195825
magit-git-items 2 0.165855 0.0829275
magit-insert-unstaged-changes 1 0.16204 0.16204
magit-insert-status-headers 1 0.157472 0.157472
magit-insert-untracked-files 1 0.152695 0.152695
magit-insert-headers 1 0.143047 0.143047
magit-insert-head-branch-header 1 0.143032 0.143032
magit-git-string 17 0.116647 0.0068615882
magit-insert-remaining-headers 1 0.114531 0.114531
magit-process-finish 1 0.112665 0.112665
magit-mode-get-buffers 2 0.112007 0.0560035
magit-process-buffer 2 0.111826 0.055913
- Command line
❯❯ time git add test
real 0m0.014s
user 0m0.002s
sys 0m0.004s
The amount of time spent for such a frequently-used operation is enough to lead to
annoyance and mental anguish on the part of the user :-]
Also note that I have 30 untracked files and 10 unstaged changes in this repo.
Maybe this plays into magit-refresh/magit-refresh-buffer?
atomontage
All 50 comments
More profiling results, this time from a new completely empty repo.
Results come from me staging and unstaging a single 5-line file.
- Stage
magit-stage 1 0.487667 0.487667
magit-stage-untracked 1 0.487638 0.487638
magit-run-git 1 0.485922 0.485922
magit-process-file 27 0.462281 0.0171215185
magit-rev-parse-safe 26 0.3255129999 0.0125197307
magit-git-str 26 0.3253689999 0.0125141923
magit-call-git 1 0.271569 0.271569
magit-toplevel 8 0.256617 0.032077125
magit-call-process 1 0.248202 0.248202
magit-refresh 1 0.214344 0.214344
magit-refresh-buffer 1 0.211877 0.211877
magit-status-refresh-buffer 1 0.211705 0.211705
magit-process-setup 1 0.154661 0.154661
magit-mode-get-buffers 2 0.15148 0.07574
magit-git-insert 5 0.07983 0.015966
- Unstage
magit-unstage 1 0.568939 0.568939
magit-unstage-1 1 0.568908 0.568908
magit-run-git 1 0.547663 0.547663
magit-process-file 32 0.546206 0.0170689375
magit-rev-parse-safe 31 0.4087369999 0.0131850645
magit-git-str 31 0.4085759999 0.0131798709
magit-call-git 1 0.33064 0.33064
magit-toplevel 9 0.3180239999 0.0353359999
magit-call-process 1 0.309545 0.309545
magit-refresh 1 0.217015 0.217015
magit-refresh-buffer 1 0.215841 0.215841
magit-status-refresh-buffer 1 0.215716 0.215716
magit-process-setup 1 0.157354 0.157354
magit-process-buffer 2 0.146106 0.073053
magit-mode-get-buffers 2 0.1431339999 0.0715669999
magit-process-finish 1 0.132236 0.132236
magit-git-insert 5 0.083577 0.0167154
Half a second+ seems excessive for this case, no?
atomontage
on 29 Dec 2016
The cause of this probably is System Integrity Protection a new of OS X El Capitan. You should disable that as described here (you might better instructions elsewhere).
tarsius
on 29 Dec 2016
Not sure how SIP plays into this, can you elaborate? Why would SIP affect git being executed through
magit/Emacs and not the same git being executed outside? The profiling results I posted point
to a number of things contributing to the slowdown.
I use my own git executable which is placed in /opt/local, nothing to do with Xcode.
I instrumented some of the frequently called functions to print their arguments.
For the last scenario I described, completely empty new repo, stage/unstage one 5-file file,
magit-process-file is called "27" times for the stage alone. This looks more than excessive
since it will create a new process running git each time.
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel) [2 times]
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-cdup) [2 times]
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel) [2 times]
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-cdup) [2 times]
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel)
process-file: (/opt/local/bin/git nil *magit-process: test nil --no-pager --literal-pathspecs -c core.preloadindex=true add -- test.c)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-cdup) [2 times]
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel)
process-file: (/opt/local/bin/git nil nil nil --no-pager --literal-pathspecs -c core.preloadindex=true update-index --refresh)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --verify HEAD)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --git-dir)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-cdup)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true config --list -z)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true status -z --porcelain)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true diff --no-ext-diff --no-prefix --)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true diff --cached --no-ext-diff --no-prefix --)
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --verify refs/stash)
process-file: (/opt/local/bin/git nil nil nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse @{upstream})
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true symbolic-ref --short HEAD)
process-file: (/opt/local/bin/git nil nil nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse @{upstream})
A lot/most of these calls look completely redundant to me.
atomontage
on 29 Dec 2016
Not sure how SIP plays into this, can you elaborate?
No, I don't use macOS and I don't know any more about it. It's just that other users had the same issue before and disabling SIP made it go away, so there is a good change that might work for you too.
Why would SIP affect git being executed through magit/Emacs and not the same git being executed outside? The profiling results I posted point to a number of things contributing to the slowdown.
If calling git is slow, that might never-the-less be unnoticeable but when calling it multiple times as is necessary to refresh magit's status buffer, the slowdown accumulates.
I use my own git executable which is placed in /opt/local, nothing to do with Xcode.
Make sure this is really the case. Emacs and your shell might not use the same $PATH.
I instrumented some of the frequently called functions to print their arguments.
How exactly did you do that? Show my the advice or patch.
For the last scenario I described, completely empty new repo, stage/unstage one 5-file file, magit-process-file is called "27" times for the stage alone.
It is called 27 times to do that and to update the status buffer.
This looks more than excessive since it will create a new process running git each time.
Actually it doesn't. Magit uses a cache now and most of these identical calls to git are actually reduced to just one. Git is still being called more than once, so the effect of whatever makes calling git slow still accumulates. (That's why I am asking for the patch/advice: to see whether it takes the cache into account.)
A lot/most of these calls look completely redundant to me.
In many places we need to know where the repository is, so we have to figure it out. But we use a cache because starting subprocesses is slow on Windows. If you get unlucky, it can also be slow on macOS, probably because of the issue I mentioned.
Please give disabling System Integrity Protection a change. Maybe there is another issue, but I would first like to rule this likely cause out.
tarsius
on 29 Dec 2016
If calling git is slow, that might never-the-less be unnoticeable but when calling it multiple times as is
necessary to refresh magit's status buffer, the slowdown accumulates.
Calling git once (to stage) through magit-process-file takes about the same amount of time to calling "git
add" outside Emacs. I've posted timings for both in my previous messages, it's 0.017s vs 0.014s which is
about the same. I don't think the root issue here is the time it takes to call git, it's (as you said) the accumulation of all these calls, that are a lot. Of course it could be that normally git-add should take much less than 0.0xx seconds and it doesn't in my case, but that seems improbable. I will nonetheless verify
without SIP and also on different OS versions.
I use my own git executable which is placed in /opt/local, nothing to do with Xcode.
Make sure this is really the case. Emacs and your shell might not use the same $PATH.
From my previous post here:
process-file: (/opt/local/bin/git nil (t nil) nil --no-pager --literal-pathspecs -c core.preloadindex=true rev-parse --show-toplevel) [2 times]
You can clearly see the path is correct.
I instrumented some of the frequently called functions to print their arguments.
How exactly did you do that? Show my the advice or patch.
I added (message "process-file: %s" args) in the beginning of magit-process-file.
You can see the output in my previous post.
This looks more than excessive since it will create a new process running git each time.
Actually it doesn't. Magit uses a cache now and most of these identical calls to git are actually reduced to just one. Git is still being called more than once, so the effect of whatever makes calling git slow still accumulates. (That's why I am asking for the patch/advice: to see whether it takes the cache into account.)
Through printing the arguments of magit-process-file we can see that process-file ends up getting called 27 times for 1 stage operation, each time creating a new git process. Now, AFAIK, process-file does not use a cache, so if you say that Magit is using a cache now that should be in-effect before you end up calling Emacs process-file. Since we can see the latter get called 27 times, that doesn't seem to be the case here. Maybe it's a configuration issue or maybe something else is causing the Magit cache to be bypassed.
Can you give me some pointers as to variables/functions I need to check to help you track this down?
I will disable SIP and get back to you on that too.
atomontage
on 29 Dec 2016
I am also experiencing this problem (OSX El Capitan, with SIP/rootless already disabled, homebrew [email protected], magit-git-executable explicitly set to the homebrew git and not system git). It's the most noticable when staging/unstaging (not sure when it started unfortunately).
This is the top of a profile from elp (only magit instrumented). I staged and unstaged a 1-line file from a magit status buffer, in a relatively small repo:
magit-run-git 2 1.725056 0.862528
magit-process-file 77 1.662533 0.0215913376
magit-refresh 2 1.191127 0.5955635
magit-refresh-buffer 2 1.172525 0.5862625
magit-status-refresh-buffer 2 1.172009 0.5860045
magit-unstage 1 0.882784 0.882784
magit-unstage-1 1 0.88274 0.88274
magit-stage 1 0.86318 0.86318
magit-stage-untracked 1 0.86315 0.86315
magit-git-str 59 0.8044480000 0.0136347118
magit-git-insert 26 0.6937960000 0.0266844615
magit-rev-parse-safe 55 0.6741869999 0.0122579454
magit-call-git 2 0.5339050000 0.2669525000
magit-call-process 2 0.48829 0.244145
magit-toplevel 38 0.4752409999 0.0125063421
magit-insert-status-headers 2 0.474912 0.237456
magit-insert-headers 2 0.43959 0.219795
magit-insert-head-branch-header 2 0.439552 0.219776
magit-insert-remaining-headers 2 0.366764 0.183382
magit-mode-get-buffers 4 0.278153 0.06953825
magit-process-setup 2 0.266046 0.133023
magit-insert-upstream-branch-header 2 0.235569 0.1177845
magit-git-string 26 0.2333819999 0.0089762307
magit-git-items 4 0.22027 0.0550675
magit-insert-untracked-files 2 0.189578 0.094789
magit-process-finish 2 0.152561 0.0762805
magit-process-unset-mode-line 2 0.152111 0.0760555
magit-process-buffer 2 0.13961 0.069805
magit-insert-unpushed-to-upstream 2 0.136475 0.0682375
magit-insert-tags-header 2 0.130556 0.065278
magit-rev-verify 7 0.126569 0.0180812857
magit-process-set-mode-line 2 0.1261010000 0.0630505000
magit-insert-unstaged-changes 2 0.125846 0.062923
magit-git-wash 8 0.125328 0.015666
magit-git-exit-code 6 0.1119690000 0.0186615
magit-rev-format 6 0.1063379999 0.017723
magit-insert-log 4 0.101463 0.02536575
magit-get 30 0.100881 0.0033627
magit-get-all 30 0.1007709999 0.0033590333
magit-config-get-from-cached-list 32 0.1006610000 0.0031456562
I do have some customizations to magit hooks, so if necessary I could rerun this from a clean config if it'd be helpful.
jguenther
on 29 Dec 2016
jguenther: In your case, magit-process-file ended up being called 77 (!) times.
That's 77 git invocations for a single stage operation.
Surely there's something wrong here.
EDIT: I tried again on my setup with SIP disabled, no difference at all.
Forgot to mention in my previous posts, all the data I've posted is from a stock Emacs 25.1 with
nothing extra loaded except magit itself that is using a stock configuration.
atomontage
on 29 Dec 2016
77 operations for a stage and then unstage, but yeah that does seem excessive. I will try to do this from an empty emacs config with just magit installed and see if it's any different.
jguenther
on 29 Dec 2016
(not sure when it started unfortunately)
But it's a somewhat recent development?
The cache I mentioned earlier only covers "refreshing the status buffer, after something made that necessary". It does not cover "save unsaved file-visiting buffers before refresh" (I think, will have to check. Also is might be fairly simple to change that.) And it also doesn't cover "revert file-visiting buffers after files have changed on disc" (using a cache here would be rather difficult).
tarsius
on 29 Dec 2016
Let's say there's room for improvement here... I'll look into this in more depth as soon as possible, probably not before the second half of January though. (Unless I find a quick solution during the next few hours.)
tarsius
on 29 Dec 2016
I could swear it wasn't this slow in the past, otherwise I would have noticed.
I'll try bisecting and see if things got considerably worse or if my memory is playing tricks on me.
atomontage
on 29 Dec 2016
magit-process-file is called "27" times for the stage alone. This looks more than excessive since it will create a new process running git each time.
Yes and no.
First, this is the result of very old design decisions (in short "just regenerate everything, when something might have changed") and I plan to address that soon. However that will be a major amount of work, as that will involve completely refactoring many central parts of magit.
Looking at the arguments a bit closer I see that rev-parse --show-cdup/--show-toplevel are being called a bit much. That is probably the result of ecc4114eeeafb4390b054bf1e04a39a8ef13adc1 and this would probably fix it:
diff --git a/lisp/magit-git.el b/lisp/magit-git.el
index db830b93..d5130f79 100644
--- a/lisp/magit-git.el
+++ b/lisp/magit-git.el
@@ -1496,7 +1496,7 @@ (defun magit-config-get-from-cached-list (key)
(--> key
(replace-regexp-in-string "\\`[^.]+" #'downcase it t t)
(replace-regexp-in-string "[^.]+\\'" #'downcase it t t))
- (magit--with-refresh-cache (list 'config (magit-toplevel))
+ (magit--with-refresh-cache (list 'config default-directory)
(let ((configs (make-hash-table :test 'equal)))
(dolist (conf (magit-git-items "config" "--list" "-z"))
(let* ((nl-pos (cl-position ?\n conf))
@npostavs Do you think this is safe (i.e. never slower than no caching at all)? Not doing it kinda defeats the purpose of using config --list (i.e. calling git more in order to call it less).
As for the other calls
# stage the change
add -- test.c
# update the status buffer
rev-parse --verify HEAD
config --list -z
status -z --porcelain
diff --no-ext-diff --no-prefix --
diff --cached --no-ext-diff --no-prefix --
rev-parse --verify refs/stash
symbolic-ref --short HEAD
update-index --refresh
rev-parse @{upstream} (okay this is called twice, why?)
there isn't much that can be done about those without major refactoring.
There are multiple strategies for improving performance that I intend to implement, all essentially boiling down to "don't do things that aren't actually necessary". Or in other words make Magit smarter, by investing the time necessary to make that happen (lot's of work). I actually already intended to write about that fairly soon.
I am afraid I don't see a way to improve this significantly any time soon, i.e. before the rewrite. Except for the thing mentioned above. Also if you can find causes for the slowness that are more concrete than "git is being called to often" (I've known that for a long time), then I might be able to squeeze out a bit of performance here and there.
But overall the time for squeezing is over, what's needed now are some major changes.
tarsius
on 29 Dec 2016
@tarsius Ok, I suspected as much, it's sorta obvious from the profiling results. I'll still do the bisect and see if there's a specific point where things got noticeably worse.
atomontage
on 29 Dec 2016
Looking at the arguments a bit closer I see that rev-parse --show-cdup/--show-toplevel are being called a bit much. That is probably the result of ecc4114
Those calls are not from magit-get-from-cached-list (except for the last), so I don't think that code is relevant.
Most of the calls seem to be from magit-maybe-save-repository-buffers, magit-process-setup (magit-process-set-mode-line), magit-process-finish (magit-process-unset-mode-line).
and this would probably fix it:
@@ -1496,7 +1496,7 @@ (defun magit-config-get-from-cached-list (key) - (magit--with-refresh-cache (list 'config (magit-toplevel)) + (magit--with-refresh-cache (list 'config default-directory)
I used (magit-toplevel) under the assumption it would be cached anyway.
npostavs
on 30 Dec 2016
Most of the calls seem to be from
magit-maybe-save-repository-buffers,magit-process-setup(magit-process-set-mode-line),magit-process-finish(magit-process-unset-mode-line).
Ah, I think we can optimize the process part. We already store the default-directory in the process object, now we just have to use it. But it's to late here for me to look into it right now.
tarsius
on 30 Dec 2016
I am closing this because there's not anything I can do right now, it's a known issue, and I am going to address it eventually anyway. But if you have concrete suggestions for tweaks, then keep them coming.
tarsius
on 31 Dec 2016
If it's a known issue can't you leave this open (or rename it to be more generic since it applies to other OSes too) so we can track it?
If you're dead-set on closing this, can you at least update the Performance section of the manual to make it obvious that "calling git too much" can be an issue. Right now the manual mentions refreshing buffers and recreating their content from scratch, which could be a different issue entirely, or it could be the root cause of calling git too much or both as implied by the profiling data (refreshing adds its own overhead on top of the time spent by calling out to git).
If you make it obvious that calling out to git a lot is a problem, others will have more information to get going with, assuming they want to dive in and attempt to fix.
atomontage
on 31 Dec 2016
Okay we can keep this open.
If you make it obvious that calling out to git a lot is a problem, others will have more information to get going with, assuming they want to dive in and attempt to fix.
As I said that requires major refactoring, so they won't succeed (unless they spend a few months on it, which I will do).
tarsius
on 31 Dec 2016
Also I closed this with the intention to very soon write about this in more detail. Once I have done that, I will close this issue again.
tarsius
on 31 Dec 2016
Will throw my hat in the ring and also say that Linux's Emacs / Magit seem to be way faster than on OSX.
Ubuntu inside a VM on my Windows 10 desktop has instantaneous everything whereas every action on OSX has probably a 500-700 ms delay to it.
It's just surprising staging a file doing OSX -> Remote Into Windows -> Ubuntu in Vmware is faster than native OSX staging a file.
Though my desktop is probably much faster than my macbook 6850K processor vs whatever is in the 2013 Macbook.
It's on my TODO list at some point to test a Linux VM from the Macbook and see if it really is just a hardware thing and not something else intrinsic to OSX causing this slowness.
jojojames
on 14 Mar 2017
I have tried disabling SIP and I confirm it does not fix the issue. If someone has a workaround for this it would be greatly appreciated.
I have also tried a blank emacs config with only magit installed and the issue is still there. (Other places mentioned that performance issues my be related to themes or packages.)
My initial awareness of this issues comes from working on the same repo from a 2013 MBP running linux (magit is blazing fast) and from a 2017 MBP running Sierra (annoying lag on just about every magit operation).
jsab
on 16 Mar 2017
I have installed VirtualBox and setup a Fedora 25 VM to test @jojojames's hypothesis. Although, using that VM (i.e. dragging windows around, typing in terminal) is slow and laggy, magit is snappy and much faster than on the macOS sierra host.
jsab
on 16 Mar 2017
Is it git itself that is slower? Can you time some git commands on the VM and compare to macOS?
npostavs
on 16 Mar 2017
I timed some git commands on https://github.com/http-kit/http-kit and it would seem that git on macOS is a few milliseconds slower on every command:
| Command | Fedora VM | macOS Host |
|--------------------------+-----------+------------|
| git checkout 2.1.0 | 0.016 | 0.032 |
| git reset master | 0.014 | 0.017 |
| git status | 0.005 | 0.008 |
| git reset HEAD --hard | 0.011 | 0.024 |
| git status | 0.005 | 0.008 |
| git reset HEAD --hard | 0.004 | 0.007 |
| git branch popo master | 0.003 | 0.006 |
| git checkout popo | 0.004 | 0.008 |
| git add wawa | 0.002 | 0.006 |
| git commit -am "wawa" | 0.003 | 0.010 |
| git rebase origin/master | 0.074 | 0.191 |
|--------------------------+-----------+------------|
Fedora git version: 2.9.3
macOS git version: 2.12.0
jsab
on 16 Mar 2017
git on macOS is a few milliseconds slower on every command:
Hmm, I would call that twice as slow on every command. I'm a bit surprised, I thought macOS had a reasonable fork/exec implementation.
Here is some elisp to record all the calls during magit-refresh. On my box GNU/Linux box, refreshing the magit repository takes 146ms out of which 125ms is used by calling git.
(defvar git-call-list t)
(defun around-magit-process-file (fun program &optional infile buffer display &rest args)
(when (listp git-call-list)
(push (cons program (mapcar #'substring-no-properties
(cl-set-difference args magit-git-global-arguments :test #'equal)))
git-call-list))
(let ((tbeg (current-time))
(tend nil))
(prog1 (apply fun program infile buffer display args)
(setq tend (current-time))
(when (listp git-call-list)
(push (float-time (time-subtract tend tbeg))
(car git-call-list))))))
(advice-add 'magit-process-file :around #'around-magit-process-file)
(defun around-magit-refresh (fun &rest args)
(let ((git-call-list ()))
(apply fun args)
(with-current-buffer (get-buffer-create "*magit calls*")
(let ((total-time 0.0))
(pcase-dolist (`(,time . ,call) git-call-list)
(insert (format "|%.3fs" time) "|" (mapconcat #'identity call " ") "\n")
(cl-incf total-time time))
(insert (format "\n\n%d calls in %.3fs\n"
(length git-call-list) total-time)))
(pop-to-buffer (current-buffer)))))
(advice-add 'magit-refresh :around #'around-magit-refresh)
(setq magit-refresh-verbose t)
Used the code from @npostavs and triggered a magit refresh in the magit buffer.
On macOS host:
````
|0.036s|git rev-parse @{upstream}
|0.037s|git name-rev --name-only --no-undefined @{upstream}
|0.039s|git name-rev --name-only --no-undefined upstream/popo
|0.036s|git rev-parse @{upstream}
|0.041s|git rev-parse --verify refs/stash
|0.038s|git diff --cached --no-ext-diff --no-prefix --
|0.037s|git diff --no-ext-diff --no-prefix --
|0.038s|git status -z --porcelain
|0.036s|git rev-parse --git-dir
|0.039s|git describe --contains HEAD
|0.040s|git describe --long --tags
|0.038s|git rev-parse --verify upstream/popo
|0.038s|git config --list -z
|0.037s|git rev-parse --show-cdup
|0.038s|git show --no-patch --format=%h %s popo^{commit} --
|0.039s|git symbolic-ref --short HEAD
|0.037s|git rev-parse --verify HEAD
|0.037s|git update-index --refresh
|0.038s|git rev-parse --show-toplevel
19 calls in 0.720s
````
On Fedora VM:
````
|0.001s|git rev-parse @{upstream}
|0.001s|git rev-parse @{upstream}
|0.002s|git rev-parse --verify refs/stash
|0.002s|git diff --cached --no-ext-diff --no-prefix --
|0.002s|git diff --no-ext-diff --no-prefix --
|0.003s|git status -z --porcelain
|0.002s|git rev-parse --git-dir
|0.002s|git describe --contains HEAD
|0.003s|git describe --long --tags
|0.001s|git config --list -z
|0.001s|git rev-parse --show-cdup
|0.002s|git show --no-patch --format=%h %s popo^{commit} --
|0.002s|git symbolic-ref --short HEAD
|0.001s|git rev-parse --verify HEAD
|0.002s|git update-index --refresh
|0.002s|git rev-parse --show-toplevel
16 calls in 0.029s
````
It seems to reveal a 35 ms overhead on every call. The same calls on the command line seem to take around 5 ms.
jsab
on 27 Mar 2017
It seems to reveal a 35 ms overhead on every call. The same calls on the command line seem to take around 5 ms.
That is interesting. Do you see the same overhead for git --version? And non-git commands (e.g., /bin/true)?
(benchmark 1 '(magit-process-file "git" nil nil nil "--version")) ;"Elapsed time: 0.003932s"
(benchmark 1 '(process-file "git" nil nil nil "--version")) ;"Elapsed time: 0.004032s"
(benchmark 1 '(call-process "git" nil nil nil "--version")) ;"Elapsed time: 0.005116s"
(benchmark 1 '(magit-process-file "/bin/true" nil nil nil)) ;"Elapsed time: 0.001756s"
(benchmark 1 '(process-file "/bin/true" nil nil nil)) ;"Elapsed time: 0.001690s"
(benchmark 1 '(call-process "/bin/true" nil nil nil)) ;"Elapsed time: 0.001583s"
~$ time git --version
git version 2.11.1
real 0m0.005s
user 0m0.000s
sys 0m0.003s
~$ time /bin/true
real 0m0.002s
user 0m0.000s
sys 0m0.000s
Well, looks like this is an emacs issue with call-process and not a magit issue.
(benchmark 1 '(magit-process-file "git" nil nil nil "--version")) "Elapsed time: 0.034923s"
(benchmark 1 '(process-file "git" nil nil nil "--version")) "Elapsed time: 0.035838s"
(benchmark 1 '(call-process "git" nil nil nil "--version")) "Elapsed time: 0.035308s"
(benchmark 1 '(magit-process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.029940s"
(benchmark 1 '(process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.029977s"
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.029757s"
````
→ time git --version
git version 2.12.0
git --version 0.00s user 0.00s system 51% cpu 0.006 total
→ time /usr/bin/true
/usr/bin/true 0.00s user 0.00s system 34% cpu 0.007 total
````
jsab
on 27 Mar 2017
I just tested with the macOS preinstalled emacs (GNU Emacs 22.1.1) and call-process is fine.
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.003534s"
jsab
on 27 Mar 2017
Since the preinstalled emacs is -nw, I tested with the 25.1.1 emacs but with the -nw option.
(benchmark 1 '(magit-process-file "git" nil nil nil "--version")) "Elapsed time: 0.007546s"
(benchmark 1 '(process-file "git" nil nil nil "--version")) "Elapsed time: 0.006457s"
(benchmark 1 '(call-process "git" nil nil nil "--version")) "Elapsed time: 0.006530s"
(benchmark 1 '(magit-process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.012839s"
(benchmark 1 '(process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.005551s"
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.005667s"
Thus, the problem is only when running from the GUI version. Magit is snappy when running from -nw.
jsab
on 27 Mar 2017
I thought this might be an issue with the emacs I'm using (https://github.com/d12frosted/homebrew-emacs-plus) so I downloaded the version from https://emacsformacosx.com/ but the situation is significantly worse:
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.043362s"
[EDIT] I have relaunched emacs-plus and now I'm getting 43 ms also.
jsab
on 27 Mar 2017
It seems that running emacs --daemon; emacsclient -c makes the situation a little bit better:
(benchmark 1 '(magit-process-file "git" nil nil nil "--version")) "Elapsed time: 0.022398s"
(benchmark 1 '(process-file "git" nil nil nil "--version")) "Elapsed time: 0.022581s"
(benchmark 1 '(call-process "git" nil nil nil "--version")) "Elapsed time: 0.022048s"
(benchmark 1 '(magit-process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.021268s"
(benchmark 1 '(process-file "/usr/bin/true" nil nil nil)) "Elapsed time: 0.021270s"
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.022116s"
jsab
on 27 Mar 2017
Thus, the problem is only when running from the GUI version. Magit is snappy when running from
-nw.
Hmm, there was also a OSX GUI-specific bug reported at https://github.com/magit/magit/issues/2295#issuecomment-143498612. The underlying problem has not been discovered, though it seems to be an OSX bug: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21573#17
npostavs
on 27 Mar 2017
I just tried https://github.com/railwaycat/homebrew-emacsmacport and the situation is even worse:
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.080534s"
However, it is also fine in -nw.
jsab
on 27 Mar 2017
For people that will come here for a workaround on spacemacs, here is what I did:
- Install https://github.com/abertsch/Menlo-for-Powerline or your favorite powerline font
- Enable xclip https://github.com/sdwolf/spacemacs-xclip
- Change iTerm Preferences, Profiles, Keys, Option as +esc
- Enjoy fast magit on macOS
jsab
on 27 Mar 2017
Benchmark of Aquamacs (http://aquamacs.org/):
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.016182s"
jsab
on 4 Apr 2017
For me on macOS 10.11 w/ emacs -Q and emacs 25.1 installed via homebrew I get:
Elapsed time: 0.003509s
Strangely, with my full spacemacs config, it doubles:
Elapsed time: 0.007726s
These are both results of running:
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil))
What could cause that doubling?
EDIT
It has to do with the frame size. The larger the frame, the longer call-process takes. When I did the above tests, my emacs -Q was a much smaller frame. `emacs -Q* is still slightly faster it seems for the same frame size, but only by about 10ms rather than 35+
aaronjensen
on 8 Apr 2017
FYI, reported upstream:
http://lists.gnu.org/archive/html/bug-gnu-emacs/2017-04/msg00201.html and YAMAMOTO Mitsuharu has an idea.
aaronjensen
on 8 Apr 2017
Thanks @aaronjensen!
tarsius
on 8 Apr 2017
I can confirm that doing what was suggested makes for a significant improvement. I now understand why people consider magit more than fast enough (who do not use macs). Very exciting 😄
I've submitted a pull to add an experimental patch to homebrew-emacs-plus.
If anyone else wants to try it before it is merged, know that it is experimental, it could cause emacs to hang or who knows what else. That said, if you want to:
$ brew tap d12frosted/emacs-plus
$ brew edit emacs-plus
And add (somewhere with the other patches):
patch do
url "https://gist.githubusercontent.com/aaronjensen/28e26a6c6c0dc767176bc35d5545d10e/raw/42b53ce8cca354f9a6249ffa63b5773abcef958e/GNU-Emacs-25.1-OS-X-enable-vfork.patch"
sha256 "37d699b42754994c8aa86f245f418b0e26cf7b4b48f29de58117086663e73ac2"
end
Then:
$ brew reinstall emacs-plus
This is great.
Thanks @jsab @npostavs @aaronjensen @tarsius
I wonder what benchmarks we are getting.
I'm only getting 2-3 ms difference.
Yamamoto Port
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil))
"Elapsed time: 0.011376s"
"Elapsed time: 0.011855s"
"Elapsed time: 0.012701s"
"Elapsed time: 0.010775s"
"Elapsed time: 0.011097s"
Built off Master with
-/* The following solves the problem that Emacs hangs when evaluating
- (make-comint "test0" "/nodir/nofile" nil "") when /nodir/nofile
- does not exist. Also, setsid is not allowed in the vfork child's
- context as of Darwin 9/Mac OS X 10.5. */
-#undef HAVE_WORKING_VFORK
-#define vfork fork
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil))
"Elapsed time: 0.008599s"
"Elapsed time: 0.009335s"
"Elapsed time: 0.009192s"
"Elapsed time: 0.008738s"
"Elapsed time: 0.008734s"
"Elapsed time: 0.007592s"
jojojames
on 8 Apr 2017
Hallelujah! The @aaronjensen patch works for me:
(benchmark 1 '(call-process "/usr/bin/true" nil nil nil)) "Elapsed time: 0.002732s"
(That is with my full spacemacs config, running emacs -Q yields a gain of about 0.4 ms.)
Note that for people who originally did not install emacs-plus from source you will need to build from source for the patch to take effect after you added the snippet with brew edit:
brew reinstall --build-from-source emacs-plus
In conclusion, this makes magit as fast in GUI emacs than it is in -nw emacs on macOS.
Big thanks to everyone!
jsab
on 9 Apr 2017
Hey all, I've updated the patch with some fixes from the mailing list.
Also, here are some updated installation instructions until it is merged in:
$ brew uninstall emacs-plus
$ brew tap aaronjensen/emacs-plus
$ brew reinstall aaronjensen/emacs-plus/emacs-plus --with-vfork
As for benchmarks @jojojames I saw a bigger improvement % wise, but my machine must be faster overall. I went from ~5-6ms to ~1-2ms.
Btw, on the bench it takes out some variability (and probably more closely replicates what magit is actually doing) if you run it 100 times (just divide the result by 100 afterwards):
(benchmark 100 '(call-process "/usr/bin/true" nil nil nil))
Elapsed time: 0.123614s
I don't have saved numbers, but before the patch I was above 0.500s for the same test.
aaronjensen
on 10 Apr 2017
You're right, it's a significant speedup (rebuilt Emacs off master).
Without patch
(benchmark 100 '(call-process "/usr/bin/true" nil nil nil))
"Elapsed time: 1.878366s"
With patch
(benchmark 100 '(call-process "/usr/bin/true" nil nil nil))
"Elapsed time: 0.540134s"
FYI, the patch is now merged to emacs master:
https://github.com/emacs-mirror/emacs/commit/a13eaddce2ddbe3ba0b7f4c81715bc0fcdba99f6
And it's available by default via homebrew w/ emacs-plus:
aaronjensen
on 20 Apr 2017
@tarsius Not what you think, but you may be able to consider this closed now that there shouldn't be a significant difference between mac and linux perf.
aaronjensen
on 20 Apr 2017
I have added a summary to the manual and updated the original post above to link to it.
Thanks again to everyone helping with this!
tarsius
on 21 Apr 2017
FYI, the patch is now merged to emacs master:
What other commits have to be cherry-picked to the emacs-25 branch for this to work?
That commit by itself applies, but apparently doesn't compile:
https://www.reddit.com/r/emacs/comments/66o45w/patch_emacs_to_speedup_magit_on_macos/dgk6989/.
tarsius
on 21 Apr 2017
I think it'd be a couple commits. There were changes to expose DEV_TTY which emacs-25 doesn't have yet, so I modified the patch:
We could probably figure out exactly which commits are needed, but there could be quite a long thread in that sweater.
aaronjensen
on 21 Apr 2017
Thanks, I am now linking to this backport.
tarsius
on 22 Apr 2017
Related issues
ipburbank
·
3Comments
razzmatazz
·
3Comments
aspiers
·
4Comments
mgalgs
·
4Comments
mmcclimon
·
4Comments
Most helpful comment
I can confirm that doing what was suggested makes for a significant improvement. I now understand why people consider magit more than fast enough (who do not use macs). Very exciting 😄
I've submitted a pull to add an experimental patch to homebrew-emacs-plus.
If anyone else wants to try it before it is merged, know that it is experimental, it could cause emacs to hang or who knows what else. That said, if you want to:
And add (somewhere with the other patches):
Then: