Magit: Improve performance

[tarsius]

Magit's performance isn't so good to say the least. I would go as far as to say that mediocre and in certain situations outright bad performance currently is Magit's biggest problem.

We have made some improvements in the past, like e.g. caching calls to git during a buffer refresh, but these improvements are to a large extend outweighed by the addition of new features. Already there exist useful features that are disabled by default because they can be to costly in some cases.

One major difference between Git and Magit is that the latter shows all kinds of useful information up-front. The major disadvantage of Magit's approach is that this information has to be always up-to-date (else it couldn't be relied on), and that doing so costs time.

To achieve good performance the cost of keeping (primarily, but not only) the status buffer up-to-date has to be reduced. And in order to do so, some fundamental changes have to be made. Doing that will take a while because it makes it necessary to replace some core abstractions. The good news is that improving or even replacing those abstractions will have other benefits beside improved performance.

---

A These are the primary causes of bad performance:

1. After the user has performed some action, the complete status buffer is refreshed. And that is done by recreating its content from scratch. Every time. All of it.

2. The required information is collected by calling git many times. Until recently, there was no alternative, but now Emacs has a foreign function interface and so we can start using libgit2 now.

3. Expensive parsing (primarily diffs and logs) is done synchronously, even when we don't need the complete result immediately (because most of it is not visible until the user scrolls).

4. The results of expensive parsing are not being cached. (The result of calls to git are cached, but that mitigates the effect of (2), not (1).)

5. Many sections are created by inserting the output directly into the user-visible buffer, where it is then manipulated to look and feel the way we want to. Without changing that, it would be very hard to address (3) and (4).

---

B There are three primary paths for improving performance:

1. 2959 Start using libgit2 instead of calling git over and over again.

2. 2985 Completely change how sections are created (using separate parsing-buffers, which likely will be preserved as a caching mechanism) and make it possible to update sections asynchronously and independently of the containing buffer.

3. Closely related to (2), only update those parts of a buffer that actually need updating.

4. Delay updating buffers that are not visible.

5. Insert diffs in two steps. Get a list of the modified files. For files that are expanded get the actual diff and insert it.

Luckily these two paths can be tackled in parallel, because in most cases a given section is either created by inserting and then "washing" git output (A5) or by processing a list of items (or a single value) using Elisp and then inserting that as the sections content (A2).

So aside from this overview, these changes will also be discussed independently, in dedicated issues.

[jkdufair]

I'd love to help contribute on number 2. I've developed some decent familiarity with the section management functions. I'll keep an eye out for the new issue.

[xmo-odoo]

Just want to comment that this would be really cool, especially number 2: I regularly do source-rewriting on a fairly large project with rewriters which are a bit hit-or-miss (they'll false-positive and rewrite stuff they should not and the serialisation will rewrite slightly more than it should) so there needs to be a post-op filtering to create the actual commits (remove stuff or split changes across multiple commits)

add -p/checkout -p kinda work but the lack of context and the keyboard answers means it's easy to "go automatic" and end up e.g. reverting a change which should not have or staging one which shouldn't be.

Magit's diff-committed and diff-uncommitted are absolute game-changers there, they're absolutely stellar in UI and UX. Their issue however is that staging changes are slow, and the slowness becomes huge as the diff size grows, with a ~15kloc diff (wasn't kidding about the bigness) it takes about 1mn30 between me hitting stage and Emacs being interactive again (though this is on 24.5, I should probably update).

[braham-snyder]

@xmo-odoo Have you tried magit-ediff-stage? Although it's not quite the same as [un-]staging directly from magit's status buffer, it avoids the regeneration of the status buffer per-stage (so will likely be far faster per-stage), while retaining better interactivity than stock git.

[xmo-odoo]

@braham-snyder I'm really not fond of ediff (though that may be because I've almost never used it so I find it odd), and the complete overview of the working copy is probably the thing I love most about magit's diff views.

[Alexander-Shukaev]

One more observation. magit-refresh can turn out to be very time consuming, especially for *-refine-*-related stuff. Oftentimes, I have magit-status (or any other magit buffer) buffer not displayed (no window displaying any of them), but when I do changes to the code in a corresponding repository, magit-refresh is still invoked. How about teaching magit to detect whether buffer(s) which it wants to refresh are not displayed, and thus queue actual refreshing on each of them (via e.g. buffer-local facilities) until they are actually displayed?

[tarsius]

I am planning to do that too. Added it to the description above.

[jypma]

There may be some good pointers over at gitstatus that can help also make magit faster. Although someone probably ought to add the multi-threading and caching improvements of gitstatusd to git itself.

[pooyataher]

It seems that everybody here is aware of these issues, but just for the record I must mention that:

• magit-status takes 30 seconds to run on my fast machine.

I installed magit yesterday and I was excited about it, but for now I switch back to git at command line because git status responds promptly for my small projects.

[jdtsmith]

There are _three primary paths_ for improving performance:
_1. ...
2.
3.
4.
5._
Luckily these _two paths_ can be tackled in parallel...

Looks like this one is on an expanding front :).

[plusoptix]

These performance issue is still really annoying. Please fix this!
At gentoo the support for git.el will be removed sooner or later. So I really wanted to use magit, but it's absolutely unusable: stashing, staging, updating the view takes 20-40 seconds - on a Core i7 machine with 8 cores!
I really want to use emacs, but so it is not possible. Please - fix it! I will sponsor chocolate.

[tarsius]

20-40 seconds

That's not the usual experience. Either you are using a humongous repository or something is broken for you that isn't broken for other people.

[pooyataher]

20-40 seconds

That's not the usual experience. Either you are using a humongous repository or something is broken for you that isn't broken for other people.

@tarsius Thanks for getting back to me. Whatever the issue was, luckily I cannot replicate that situation anymore! I am now using Magit everyday. Thank you!

[plusoptix]

OK, thank you for the feedback: I digged deeper: This happens, when you have gettext- / *.po files in your tree. If you dismiss all changes on these kind of files (we have about 20 in our repo), magit responses quite fast.

I tried also to start the emacs without init.el, so no special hooks are activated to open gettext files.

Is this a new bug or should I only add it here as hint for other users, which have the same issue.

[tarsius]

Actually lets look at this a bit. 20 untracked (or ignored?) files isn't much at all.

So if you have those ~20 files, then it takes 20-40 seconds to do a simple refresh (C-g)? Please time it. And without them it takes less than a second?

Are these files being ignore by Git (i.e. is there such a rule in .gitignore)? If there wasn't such a rule, does adding one make the issue go away? Even if it does, if you just added such a rule now, then remove it again for the subsequent tests.

Does the Magit status buffer show any information about these files? (Is there a Ignored files or Untracked files section?) Does removing all magit-insert-*-files sections from magit-status-section-hook fix the issue? If so, then figure out which of these functions is the cause.

[plusoptix]

More input to this issue:
There are 20 git controller po-files in my repo
(check with $> git status | grep \.po)
changed: cs.po
changed: da.po
changed: de.po
changed: el.po
changed: en.po
changed: es.po
changed: fr.po
changed: hu.po
changed: it.po
changed: ja.po
changed: lv.po
changed: pl.po
changed: poqt_gui.pot
changed: pt.po
changed: ro.po
changed: ru.po
changed: tr.po
changed: uk.po
changed: zh_CN.po
changed: zh_TW.po
-> entering "g" at magit buffer
-> utimer -s
Elapsed Time: 0 days 00:00:39.198 (39.198 seconds)

$> git checkout cs.po da.po de.po el.po en.po es.po fr.po hu.po it.po ja.po
-> revert the first half of the files
changed: lv.po
changed: pl.po
changed: poqt_gui.pot
changed: pt.po
changed: ro.po
changed: ru.po
changed: tr.po
changed: uk.po
changed: zh_CN.po
changed: zh_TW.po
-> entering "g" at magit buffer
-> utimer -s
Elapsed Time: 0 days 00:00:23.259 (23.259 seconds)

$> git checkout da.po...
-> checkout the first half, so the second half of the files stay changed:
changed: lv.po
changed: pl.po
changed: poqt_gui.pot
changed: pt.po
changed: ro.po
changed: ru.po
changed: tr.po
changed: uk.po
changed: zh_CN.po
changed: zh_TW.po

$> git checkout *.po

• no changed po-files left over
  -> enter "g" at magit buffer
  -> utimer -s
  Elapsed Time: 0 days 00:00:00.158 (0.158 seconds)

How large are our files:
$> wc *.po
5643 24040 189701 cs.po
4661 17088 144576 da.po
5428 23581 186507 de.po
4691 19496 182921 el.po
5623 24424 183887 en.po
5831 26770 198447 es.po
5872 26756 200695 fr.po
4341 15857 135002 hu.po
5867 26355 199595 it.po
5318 18889 188949 ja.po
5559 23807 190925 lv.po
4662 18840 155765 pl.po
5866 26783 199455 pt.po
5744 25187 192231 ro.po
5864 25840 232415 ru.po
5758 24131 192257 tr.po
4810 17666 168804 uk.po
5280 18832 177455 zh_CN.po
5273 18809 177316 zh_TW.po

About 300 translations per language. A lot of translations are sentences.
There are 7 further .po files, which are not git tracked and added into the .gitignore. These files do not interfere.

[plusoptix]

At the moment I have in my magit-status.el
(defcustom magit-status-sections-hook
'(magit-insert-status-headers
magit-insert-merge-log
magit-insert-rebase-sequence
magit-insert-am-sequence
magit-insert-sequencer-sequence
magit-insert-bisect-output
magit-insert-bisect-rest
magit-insert-bisect-log
magit-insert-untracked-files
magit-insert-unstaged-changes
magit-insert-staged-changes
magit-insert-stashes
magit-insert-unpushed-to-pushremote
magit-insert-unpushed-to-upstream-or-recent
magit-insert-unpulled-from-pushremote
magit-insert-unpulled-from-upstream)

When I remove
magit-insert-stashes
magit-insert-unpushed-to-pushremote
magit-insert-unpushed-to-upstream-or-recent
magit-insert-unpulled-from-pushremote
magit-insert-unpulled-from-upstream

Then I can reduce the time from 38 to 10 seconds

[tarsius]

I just remembered we recently added something to make this easier: set magit-debug-refresh to t and then post the output from *Messages*.

Here's what those numbers normally look like:

Refreshing magit...
Running magit-pre-refresh-hook...done (0.002s)
Refreshing buffer `magit: magit'...
  magit-insert-error-header                          5.28e-06
  magit-insert-diff-filter-header                    3.5779e-05
  magit-insert-head-branch-header                    0.005917256
  magit-insert-upstream-branch-header                0.009664677
  magit-insert-push-branch-header                    0.00295999
  magit-insert-tags-header                           0.029366055
  magit-insert-status-headers                        0.051382107
  magit-insert-merge-log                             0.002367351
  magit-insert-rebase-sequence                       0.000558741
  magit-insert-am-sequence                           0.000267262
  magit-insert-sequencer-sequence                    0.000472803
  magit-insert-bisect-output                         0.000263354
  magit-insert-bisect-rest                           5.3672e-05
  magit-insert-bisect-log                            5.2426e-05
  magit-insert-untracked-files                       0.002597223
  magit-insert-unstaged-changes                      0.003391459
  magit-insert-staged-changes                        0.005237672
  magit-insert-stashes                               0.020132359
  magit-insert-modules                               0.002975827
  magit-insert-worktrees                             0.002223353
  magit-insert-unpushed-to-pushremote                0.041945618
  magit-insert-unpushed-to-upstream-or-recent        0.032749889
  magit-insert-unpulled-from-pushremote              0.000270675
  magit-insert-unpulled-from-upstream                0.021647526
  forge-insert-pullreqs                              0.021850482
  forge-insert-issues                                0.015691664
Refreshing buffer `magit: magit'...done (0.236s)
Running magit-post-refresh-hook...done (0.008s)
Refreshing magit...done (0.253s, cached 84/116)

[aaronjensen]

FYI I don't see magit-debug-refresh in the MELPA version.

FWIW, refresh takes about 400ms in my repo and I don't have forge enabled. Though I just disabled magit-todos and it brought it down to 200ms... yuck.

Edit: Is there a way to limit what gets refreshed when staging and unstaging? Ideally it'd only refresh the staged and unstaged sections and it'd be great to disable things like magit-todos during that refresh.

[kyleam]

FYI I don't see magit-debug-refresh in the MELPA version.

I think that was meant to be magit-refresh-verbose.

[aaronjensen]

Ah, thanks. Looks like forge still takes a little time even if the repo isn't forge enabled yet, and magit-todos (specifically magit-todos-branch-list) is very expensive.

Refreshing buffer ‘magit: iheartjane’...
  magit-insert-error-header                          2e-06
  magit-insert-diff-filter-header                    6e-06
  magit-insert-head-branch-header                    0.015213
  magit-insert-upstream-branch-header                0.020369
  magit-insert-push-branch-header                    0.004845
  magit-insert-tags-header                           0.011144
  magit-insert-status-headers                        0.057213
  magit-insert-merge-log                             0.004333
  magit-insert-rebase-sequence                       0.000206
  magit-insert-am-sequence                           9.9e-05
  magit-insert-sequencer-sequence                    0.000195
  magit-insert-bisect-output                         0.000101
  magit-insert-bisect-rest                           2.6e-05
  magit-insert-bisect-log                            2.6e-05
  magit-insert-untracked-files                       0.02154
  magit-insert-unstaged-changes                      0.018993
  magit-insert-staged-changes                        0.010953
  magit-todos--insert-todos                          0.158363
  magit-insert-stashes                               0.006123
  magit-insert-unpushed-to-pushremote                0.026288
  magit-insert-unpushed-to-upstream                  0.011508000000000001
  magit-insert-unpulled-from-pushremote              8.9e-05
  magit-insert-unpulled-from-upstream                0.006936
  forge-insert-pullreqs                              0.042196
  forge-insert-issues                                5e-06
Refreshing buffer ‘magit: iheartjane’...done (0.387s)

[Alexander-Shukaev]

Repositories with large amount of both files and changes per commit, e.g. produced by generated code (e.g. when the source code is regenerated after an update to the code generator), make magit-status totally unusable. I know me and others already complained about it many times and suggested a number of improvements. However, I have a feeling that implementing the comprehensive list of improvements from the top might take a lot of time. It's also not clear what's the current status of this sub-project.

Long story short, not sure if the following idea is implied by the comprehensive list of improvements at the top but I'd still mention it. I believe a drastic performance improvement for magit-status could already be achieved if all the sections that potentially insert diffs are eliminated and only lists of the corresponding files affected per section are displayed instead. This solves three main performance problems:

1. Don't have to wait for diffs to be calculated and inserted when they might not even be needed at this point at all.
2. Don't grow the buffer extra large as Emacs is (to say softly) notorious of not working well with extra large buffers.
3. Don't have to rebuild/refresh that extra large buffer over and over again upon any update (even a ridiculously small one).

However, doing only that would spoil usability too indeed. The idea is to remap the key that is currently expanding diffs (which are already inserted in the buffer) to a function that would instead create a separate buffer where sections of diff hunks are inserted for that particular file only and immediately focus it; upon quit, focus back the parent magit-status buffer. So magit-status buffer acts as a "master" buffer towards any hunk buffers opened, e.g. when magit-status buffer is killed, all hunk buffers belonging to it are killed as well (at least by default). Whatever actions/features/bindings (e.g. (un)staging, reverting, etc.) are available in the magit-status buffer for the sections of diff hunks can be replicated in such separate file diff dedicated buffer. This essentially sounds like a feature request for an additional (buffer-wise) mode of operation for magit-section, which is planned to be released separately anyway as per #4003.

This does not sound like a lot of work and can be introduced as an optional mode to use for performance reasons. What do you guys think?

NOTE:

2985 looks interesting indeed but it's far too complex and ambitious to implement in the nearest future. One has to be reasonable with the ratio of outcome to delivery time/effort. Solving the outstanding performance problem can be done in stages, where the above approach seems appealing as the very first stage since it offers high ratio.

[ymherklotz]

I seem to be having a performance issue with Magit on MacOS Catalina, it takes 0.6s to refresh a repository with one commit vs 0.033s on linux with the same repository. This increases to 1s on the main repository I work on and it therefore takes a while to perform any actions such as adding, removing files or committing.

I have read Magit's guide on performance for MacOS, and have tried various versions of emacs from 26.3 to master (28.0). I have also tried to find posts about the same issue, and have tried their solutions but they have not worked either (such as turning off themes). Most recommended upgrading the emacs version from 26.1 which I already had done though. This issue persists if I launch emacs only with magit enabled and in the config file.

I believe that it has something to do with git taking a long time to launch from emacs, which sounds similar to the fork vs vfork issue from emacs 26.1. I am not sure though how to check if my emacs version is actually calling vfork correctly.

Is there anything else I should try to debug? For now I only notice this problem in magit which is why I am posting it here and it seems to happen when only magit is enabled in the config file.

From the commandline, git executes as quickly as on linux. Two magit versions below:

Magit 20200307.319, Git 2.25.1, Emacs 26.3, gnu/linux
Magit 20200307.319, Git 2.25.1, Emacs 28.0.50, darwin

For comparison, here is the output for linux and mac after running magit-refresh on both linux and mac (on a repository with one file and one commit and similar hardware):

Linux Performance

Refreshing magit...
Running magit-pre-refresh-hook...done (0.009s)
Refreshing buffer ‘magit: newproject’...
  magit-insert-error-header                          1.653e-06
  magit-insert-diff-filter-header                    0.003189448
  magit-insert-head-branch-header                    0.002403502
  magit-insert-upstream-branch-header                6.1721e-05
  magit-insert-push-branch-header                    3.5632e-05
  magit-insert-tags-header                           0.002833871
  magit-insert-status-headers                        0.011557551
  magit-insert-merge-log                             0.001288038
  magit-insert-rebase-sequence                       0.00017291
  magit-insert-am-sequence                           0.000111263
  magit-insert-sequencer-sequence                    0.000126705
  magit-insert-bisect-output                         9.7657e-05
  magit-insert-bisect-rest                           2.0356e-05
  magit-insert-bisect-log                            1.3118e-05
  magit-insert-untracked-files                       0.002335742
  magit-insert-unstaged-changes                      0.002110061
  magit-insert-staged-changes                        0.004152358
  magit-insert-stashes                               0.001360473
  magit-insert-unpushed-to-pushremote                6.6282e-05
  magit-insert-unpushed-to-upstream-or-recent        0.008802725
  magit-insert-unpulled-from-pushremote              4.1933e-05
  magit-insert-unpulled-from-upstream                1.104e-05
Refreshing buffer ‘magit: newproject’...done (0.037s)
Running magit-post-refresh-hook...done (0.000s)
Refreshing magit...done (0.049s, cached 50/73)

MacOS Performance

Refreshing magit...
Running magit-pre-refresh-hook...done (0.039s)
Refreshing buffer ‘magit: random’...
  magit-insert-error-header                          2e-06
  magit-insert-diff-filter-header                    0.055977
  magit-insert-head-branch-header                    0.057386
  magit-insert-upstream-branch-header                5.3e-05
  magit-insert-push-branch-header                    3.4e-05
  magit-insert-tags-header                           0.057151
  magit-insert-status-headers                        0.199779
  magit-insert-merge-log                             0.02891
  magit-insert-rebase-sequence                       0.000172
  magit-insert-am-sequence                           8.1e-05
  magit-insert-sequencer-sequence                    0.000156
  magit-insert-bisect-output                         8.3e-05
  magit-insert-bisect-rest                           2.2e-05
  magit-insert-bisect-log                            2.2e-05
  magit-insert-untracked-files                       0.033737
  magit-insert-unstaged-changes                      0.02892
  magit-insert-staged-changes                        0.056712
  magit-insert-stashes                               0.029054
  magit-insert-unpushed-to-pushremote                5e-05
  magit-insert-unpushed-to-upstream-or-recent        0.143981
  magit-insert-unpulled-from-pushremote              3.9e-05
  magit-insert-unpulled-from-upstream                1.2e-05
Refreshing buffer ‘magit: random’...done (0.554s)
Running magit-post-refresh-hook...done (0.000s)
Refreshing magit...done (0.600s, cached 50/73)

[theckman]

@ymherklotz I'm not personally experiencing this, but have experienced similar things before when my shell was slow to start because of junk I had in my .zshrc file. That isn't happening to you is it?

[saghaulor]

@ymherklotz In your comment you mentioned that setting the absolute path to git helped significantly.

I have actually messed around with it a bit more and think I have found a solution! For some reason setting magit-git-executable to the absolute path of git (/usr/local/bin/git) solved the problem, even if it was pointing to the exact same git executable (Checked using magit-debug-git-executable).

I know some other people were reporting the same problem, so I hope that solves it for them too.

Now I get the following new performance, which is 2x slower than Linux, but inline with the speed of git on MacOS vs Linux.

MacOS Performance

Refreshing buffer ‘magit: random’...
  magit-insert-error-header                          1e-06
  magit-insert-diff-filter-header                    2.9e-05
  magit-insert-head-branch-header                    0.008377
  magit-insert-upstream-branch-header                4.7e-05
  magit-insert-push-branch-header                    3.8e-05
  magit-insert-tags-header                           0.007386
  magit-insert-status-headers                        0.021236
  magit-insert-merge-log                             0.003121
  magit-insert-rebase-sequence                       0.000118
  magit-insert-am-sequence                           6.8e-05
  magit-insert-sequencer-sequence                    0.000126
  magit-insert-bisect-output                         7.4e-05
  magit-insert-bisect-rest                           2.1e-05
  magit-insert-bisect-log                            2e-05
  magit-insert-untracked-files                       0.00416
  magit-insert-unstaged-changes                      0.003453
  magit-insert-staged-changes                        0.006959
  magit-insert-stashes                               0.003041
  magit-insert-unpushed-to-pushremote                3.6e-05
  magit-insert-unpushed-to-upstream-or-recent        0.01527
  magit-insert-unpulled-from-pushremote              7e-05
  magit-insert-unpulled-from-upstream                8e-06
Refreshing buffer ‘magit: random’...done (0.066s)

How exactly did you set the magit-git-executable path? Do I just add

(setq magit-git-exectuable "/usr/local/bin/git")

to my emacs config?

I think my path was already set?

magit-git-executable: "/usr/local/bin/git" (2.26.2)
exec-path: ("/Users/stephenaghaulor/.nvm/versions/node/v12.7.0/bin/" "/Users/stephenaghaulor/.rbenv/shims/" "/usr/local/bin/" "/usr/bin/" "/bin/" "/usr/sbin/" "/sbin/" "/Library/Apple/usr/bin/" "/Users/stephenaghaulor/.local/bin/" "/Users/stephenaghaulor/.rbenv/bin/" "/Users/stephenaghaulor/src/gocode/bin/" "/Applications/Emacs.app/Contents/MacOS/bin-x86_64-10_14/" "/Applications/Emacs.app/Contents/MacOS/libexec-x86_64-10_14/" "/Applications/Emacs.app/Contents/MacOS/libexec/")
  /Users/stephenaghaulor/.nvm/versions/node/v12.7.0/bin/ (t)
  /Users/stephenaghaulor/.rbenv/shims/ (t)
  /usr/local/bin/ (t)
    /usr/local/bin/git (2.26.2)
  /usr/bin/ (t)
    /usr/bin/git (2.24.2 (Apple Git-127))
  /bin/ (t)
  /usr/sbin/ (t)
  /sbin/ (t)
  /Library/Apple/usr/bin/ (t)
  /Users/stephenaghaulor/.local/bin/ (nil)
  /Users/stephenaghaulor/.rbenv/bin/ (nil)
  /Users/stephenaghaulor/src/gocode/bin/ (nil)
  /Applications/Emacs.app/Contents/MacOS/bin-x86_64-10_14/ (t)
  /Applications/Emacs.app/Contents/MacOS/libexec-x86_64-10_14/ (t)
  /Applications/Emacs.app/Contents/MacOS/libexec/ (t)

I could really use some help, I'm seeing abysmal performance.

Refreshing buffer ‘magit: random’...
  magit-insert-error-header                          1e-06
  magit-insert-diff-filter-header                    0.005116
  magit-insert-head-branch-header                    0.033715
  magit-insert-upstream-branch-header                0.302397
  magit-insert-push-branch-header                    7.9e-05
  magit-insert-tags-header                           0.414043
  magit-insert-status-headers                        0.763833
  magit-insert-merge-log                             0.004443
  magit-insert-rebase-sequence                       0.000231
  magit-insert-am-sequence                           0.00011
  magit-insert-sequencer-sequence                    0.000217
  magit-insert-bisect-output                         0.000115
  magit-insert-bisect-rest                           3.3e-05
  magit-insert-bisect-log                            3.3e-05
  magit-insert-untracked-files                       1.42289
  magit-insert-unstaged-changes                      0.971694
  magit-insert-staged-changes                        0.030547
  magit-insert-stashes                               0.062513
  magit-insert-unpushed-to-pushremote                7.3e-05
  magit-insert-unpushed-to-upstream-or-recent        0.098215
  magit-insert-unpulled-from-pushremote              6.3e-05
  magit-insert-unpulled-from-upstream                0.07922
Refreshing buffer ‘magit: random’...done (4.544s)

Please help.

EDIT:
The above stats are for a very large project. I profiled an empty git repo and saw comparable stats.

Refreshing buffer ‘magit: magit-test’...
  magit-insert-status-headers                        0.004225
  magit-insert-merge-log                             0.004057
  magit-insert-rebase-sequence                       0.000237
  magit-insert-am-sequence                           0.000109
  magit-insert-sequencer-sequence                    0.000224
  magit-insert-bisect-output                         0.000112
  magit-insert-bisect-rest                           3.2e-05
  magit-insert-bisect-log                            3.2e-05
  magit-insert-untracked-files                       0.014533
  magit-insert-unstaged-changes                      0.008059
  magit-insert-staged-changes                        0.00895
  magit-insert-stashes                               0.004064
  magit-insert-unpushed-to-pushremote                0.004015
  magit-insert-unpushed-to-upstream-or-recent        0.011885
  magit-insert-unpulled-from-pushremote              2.9e-05
  magit-insert-unpulled-from-upstream                6e-06
Refreshing buffer ‘magit: magit-test’...done (0.067s)

[ptn]

@saghaulor that's how I set that, but it didn't make a difference for me. What did help was disabling sections for the status buffer. In my case, the worst offenders were:

  (remove-hook 'magit-status-sections-hook 'magit-insert-unpushed-to-pushremote)
  (remove-hook 'magit-status-sections-hook 'magit-insert-unpushed-to-upstream-or-recent)
  (remove-hook 'magit-status-sections-hook 'magit-insert-unpulled-from-pushremote)
  (remove-hook 'magit-status-sections-hook 'magit-insert-unpulled-from-upstream)

Disabling sections is documented in the official manual (it's a single line at the bottom).

[cartesian-theatrics]

Any chance this will be addressed in the near term? I concur this is by far the biggest problem with magit. I absolutely love this tool, but for large repos it is approaching not worth using unfortunately.

[aspiers]

magit-refresh-verbose was already mentioned in a previous comment over a year ago, but I guess I'm not the only one who missed that super helpful tip. I just saw it mentioned in this blog post via Sacha Chua's weekly newsletter, so hopefully that's of use to people reading here.

[seokbeomKim]

magit-refresh-verbose was already mentioned in a previous comment over a year ago, but I guess I'm not the only one who missed that super helpful tip. I just saw it mentioned in this blog post via Sacha Chua's weekly newsletter, so hopefully that's of use to people reading here.

Thanks for the link, it was really helpful in my case.
Using magit was really slow on huge codes like Linux kernel. But after removing some hooks including above tips, I could get rid of time-consuming features which are unnecessary for my usages.

;; revision
(remove-hook 'magit-revision-sections-hook 'magit-insert-revision-headers)
(remove-hook 'magit-revision-sections-hook 'magit-insert-revision-notes)
(remove-hook 'magit-revision-sections-hook 'magit-insert-revision-diff)
(remove-hook 'magit-revision-sections-hook 'magit-insert-revision-tag)
(remove-hook 'magit-revision-sections-hook 'magit-insert-xref-buttons)

;; status
(remove-hook 'magit-status-sections-hook 'magit-insert-tags-header)
(remove-hook 'magit-status-sections-hook 'magit-insert-status-headers)

[Profpatsch]

  magit-insert-error-header                          2.145e-06 
  magit-insert-diff-filter-header                    0.007394417 
  magit-insert-head-branch-header                    0.006269013 
  magit-insert-upstream-branch-header                1.054e-05 
  magit-insert-push-branch-header                    9.621e-06 
  magit-insert-tags-header                           0.014738266 !
  magit-insert-status-headers                        0.032404343 !!
  magit-insert-merge-log                             0.002848184 
  magit-insert-rebase-sequence                       2.420365681 !!
  magit-insert-am-sequence                           0.000166788 
  magit-insert-sequencer-sequence                    0.000218156 
  magit-insert-bisect-output                         0.000103943 
  magit-insert-bisect-rest                           2.768e-05 
  magit-insert-bisect-log                            4.8204e-05 
  magit-insert-untracked-files                       0.014268144 !
  magit-insert-unstaged-changes                      0.00530372 
  magit-insert-staged-changes                        0.014727778 !
  magit-insert-stashes                               0.073646006 !!
  magit-insert-unpushed-to-pushremote                5.2532e-05 
  magit-insert-unpushed-to-upstream-or-recent        0.019587845 !
  magit-insert-unpulled-from-pushremote              3.435e-06 
  magit-insert-unpulled-from-upstream                3.128e-06 

magit-insert-rebase-sequence is taking 2.5 seconds on a small-ish repo here with 2500 commits and 100 stashes

[jdelStrother]

@Profpatsch do you have a rebase in progress? My magit-insert-rebase-sequence is 0.0002 seconds, on a repo with 41,307 commits and 100 stashes.

Refreshing buffer ‘magit: web’...
  magit-insert-error-header                          3e-06 
  magit-insert-diff-filter-header                    0.018225 !
  magit-insert-head-branch-header                    0.022502 !
  magit-insert-upstream-branch-header                0.023856 !
  magit-insert-push-branch-header                    0.009783 
  magit-insert-status-headers                        0.086622 !!
  magit-insert-merge-log                             0.009021 
  magit-insert-rebase-sequence                       0.000208 
  magit-insert-am-sequence                           9.6e-05 
  magit-insert-sequencer-sequence                    0.000189 
  magit-insert-bisect-output                         9.9e-05 
  magit-insert-bisect-rest                           3.1e-05 
  magit-insert-bisect-log                            2.9e-05 
  magit-insert-untracked-files                       0.047831 !!
  magit-insert-unstaged-changes                      0.013027 !
  magit-insert-staged-changes                        0.021695 !
  magit-insert-stashes                               0.091677 !!
  magit-insert-unpushed-to-pushremote                7.2e-05 
  magit-insert-unpushed-to-upstream-or-recent        0.131429 !!
  magit-insert-unpulled-from-pushremote              5e-06 
  magit-insert-unpulled-from-upstream                0.092876 !!
  forge-insert-pullreqs                              0.028664 !
  forge-insert-issues                                0.001083 
Refreshing buffer ‘magit: web’...done (0.553s)
Running magit-post-refresh-hook...done (0.014s)
Refreshing magit...done (0.570s, cached 56/86 (65%))

[Profpatsch]

@jdelStrother yes, of course, the function is a no-op without a rebase.

[Profpatsch]

it’s not a big rebase either, I’m rebasing maybe 10 commits.

[Profpatsch]

My initial profiling has pointed to https://github.com/magit/magit/blob/21454777281247d97814ce5fb64f4afe39fab5da/lisp/magit-sequence.el#L978-L1038 as the function taking most of the time.

[aspiers]

If you want to do more profiling, you may find https://github.com/aspiers/etrace useful.