projectile-mode makes editior extremally laggy and slow
I'm using projectile for a while, and I'm extremaly happy with that.
But recently, after package update something went wrong. When projectile-mode is enabled, all my actions in editor (like typing, and navigating in file) started to be very slow, practically making normal work impossible.
When I disable projectile-mode, then all issues magically disappear and I can continue my work. I compared two projects (both .git based).
- My config files (several lisp files) - edition of all files is slow as hell
- Drupal 8 projects (massive amount of php files) - the same.
I believe, that this wasn't the case in a past, but when I recently upgraded to newest (20171009.84) version, it started to crawl.
jsobiecki
All 43 comments
Try cleaning up the cache (C-p i)
Silex
on 10 Oct 2017
I have got the exact same problem.
M-x projectile-invalidate-cache
didn't help. Sorry
I try to find a workaround. If you have one, plz post.
rreckel
on 10 Oct 2017
Here is the profile log:
- command-execute 35500 65%
- call-interactively 35500 65%
- byte-code 30924 56%
- helm-M-x-read-extended-command 30924 56%
- helm-comp-read 30924 56%
- helm 30923 56%
- apply 30923 56%
- helm 30923 56%
- apply 30923 56%
- helm-internal 30923 56%
- helm-read-pattern-maybe 28503 52%
- read-from-minibuffer 28490 52%
- redisplay_internal (C function) 28422 52%
- eval 28422 52%
- format 28422 52%
- projectile-project-type 28422 52%
- projectile-detect-project-type 28422 52%
- cl-find-if 28422 52%
- apply 28422 52%
- cl-find 28422 52%
- apply 28422 52%
- cl-position 28422 52%
- cl--position 28422 52%
- #<compiled 0x40411bc1> 28422 52%
- projectile-go-project-p 28366 52%
- projectile-current-pr... 28332 52%
- cl-mapcan 28332 52%
- apply 28332 52%
- cl-mapcar 28332 52%
- mapcar 28332 52%
- projectile-dir-f... 28332 52%
- projectile-dir-... 20937 38%
- mapcar 18370 33%
- #<compiled 0x... 18370 33%
- file-relativ... 11729 21%
file-remote-p 56 0%
#<compiled ... 23 0%
string-pref... 3 0%
- projectile-get... 2275 4%
- projectile-fi... 1768 3%
- shell-comman... 878 1%
- process-file 873 1%
- apply 873 1%
- call-proc... 15 0%
beacon--... 6 0%
- #<compiled ... 3 0%
kill-buffer 3 0%
split-string 747 1%
- projectile-ge... 499 0%
- projectile-g... 499 0%
- projectile-... 499 0%
- projectile... 453 0%
- shell-com... 434 0%
- process-... 274 0%
- apply 269 0%
- call-p... 7 0%
- ucs-n... 7 0%
gene... 7 0%
- #<compil... 154 0%
- kill-bu... 150 0%
- replac... 6 0%
unrec... 4 0%
helm-k... 2 0%
split-str... 1 0%
- projectile... 40 0%
- projectil... 40 0%
- projecti... 34 0%
- file-ex... 5 0%
- ucs-no... 4 0%
- inser... 2 0%
beac... 2 0%
ucs-n... 1 0%
gener... 1 0%
- cl-remove-... 1 0%
apply 1 0%
- projectile-ge... 2 0%
- projectile-p... 2 0%
projectile-... 2 0%
- projectile-pr... 1 0%
projectile-f... 1 0%
- projectile-adju... 7388 13%
- projectile-rem... 7080 13%
- cl-remove-if 6832 12%
- apply 6832 12%
- cl-remove 6824 12%
- #<compiled... 6813 12%
- cl-some 828 1%
#<compil... 700 1%
- #<compil... 94 0%
string-... 15 0%
You should try this workaround: https://github.com/brocode/emacs_subconfigs/commit/831d6549694e04e460b973a550e08177bba6c7c4
As I understand it a new feature is that projectile tries to determine the project type by going through heuristics. For golang projects the heuristic is "go through all files in the project and see if it ends with .go". Honestly I think that is a pretty terrible idea because going through all files in the project is always a bad idea (esp. if it's just to find out if the project is golang or not!).
The workaround I linked removes the project type from the modeline so it's not computed anymore.
There was https://github.com/bbatsov/projectile/issues/1182 open but it was fixed in the sense that projectile does not try to determine the project type when there is no project (it would go through all files in your $HOME then). But if your project is huge you will feel the pain.
(note: the format string in that workaround commit is broken, you want Projectile[%s] instead)
mriehl
on 10 Oct 2017
Hmm, I guess I underestimated this - I'll try to take a look at it soon, but in the meantime everyone can just set the modeline to nil or something like old value:
(setq projectile-mode-line
'(:eval (format " Projectile[%s]"
(projectile-project-name))))
bbatsov
on 10 Oct 2017
Looking at the profiling data it seems something is wrong with the project type caching, otherwise I can't imagine what would be causing this slowdown.
(defun projectile-detect-project-type ()
"Detect the type of the current project."
(let ((project-type (cl-find-if
(lambda (project-type)
(let ((marker (plist-get (gethash project-type projectile-project-types) 'marker-files)))
(if (listp marker)
(and (projectile-verify-files marker) project-type)
(and (funcall marker) project-type))))
(projectile-hash-keys projectile-project-types))))
(when project-type
(puthash (projectile-project-root) project-type projectile-project-type-cache))
project-type))
(defun projectile-project-type ()
"Determine the project's type based on its structure."
(if projectile-project-type
projectile-project-type
(let ((project-root (ignore-errors (projectile-project-root))))
(if project-root
(or (gethash project-root projectile-project-type-cache)
(projectile-detect-project-type)
'generic)
'generic))))
That's all the relevant code and it's pretty simple. At least right away nothing seems wrong with respect to caching here.
bbatsov
on 10 Oct 2017
For what is worth in my tests I don't see any lag.
bbatsov
on 10 Oct 2017
@bbatsov @mriehl - Workaround works for me, thanks!
jsobiecki
on 10 Oct 2017
Thx a lot.
The workaround worked for me too!
So, if I understand correctly: Every time the cursor moves, the mode line is updated, and so every time projectile tries to determine the project type, which is very slow (to determine go projects).
If there is a problem with my projectile cache, how can I remove/invalidate it?
As I said M-x projectile-invalidate-cache just tells me that it removed 0 items from the recentf list, and does not help in any way.
Here is a second part of the profile, that could confirm the slowness in determining go projects:
- ... 9193 36%
Automatic GC 8877 35%
- line-move-visual 316 1%
- posn-at-point 316 1%
- eval 316 1%
- format 316 1%
- projectile-project-type 316 1%
- projectile-detect-project-type 316 1%
- cl-find-if 316 1%
- apply 316 1%
- cl-find 316 1%
- apply 316 1%
- cl-position 316 1%
- cl--position 316 1%
- #<compiled 0x416538b3> 316 1%
- projectile-go-project-p 316 1%
- cl-some 316 1%
- #<compiled 0x4165386d> 316 1%
- file-name-extension 315 1%
file-name-sans-versions 30 0%
#<compiled 0x4152e21f> 2 0%
Same issue here, and the workaround helps.
vkazanov
on 11 Oct 2017
@bbatsov I think it's just the golang detection that is super inefficient.
(cl-some
(lambda (file)
(string= (file-name-extension file) "go"))
(projectile-current-project-files)))
On linux (using strace -yy -f for syscall tracing) this translates in a crapton of fstat, fcntl, getdents, openat and close syscalls for each folder in the project. If you have a .npm or lots of directories this is super super slow.
In the elisp profiles projectile-go-project-p 28366 52% seems to be a dead giveaway.
I'm not sure why file-name-extension translates into so many syscalls. But a workaround could be to do something that isn't O(n) with the number of files/directories in the project for golang detection.
Like the rustlang detection that is more or less O(1) since it looks for Cargo.toml.
For instance we could be using $GOPATH and just see if the project cwd is in that path. This would result in false negatives ($GOPATH not set but go files in the project) but at the same time since golang is super opinionated you can't go build or go test if you're not in the $GOPATH sooooo...
The caching seems to be working for me (if I let the initial project type detection complete, which takes upwards of 5 minutes (!) then it's using the cached type). Also the -nw emacs seems to be doing this in the background (?) so it can stack a lot of projectile-go-project calls before it actually stores a cached type.
mriehl
on 11 Oct 2017
Same issue and I reached same workaround.
I'm using projectile on C# project, so I think it's not just golang detection problem.
- command-execute 2119 74%
- call-interactively 2119 74%
- funcall-interactively 1973 69%
- next-line 1969 69%
- line-move 1969 69%
- line-move-partial 1969 69%
- pos-visible-in-window-p 1243 43%
- eval 1243 43%
- format 1241 43%
- projectile-project-type 1241 43%
- projectile-detect-project-type 1241 43%
- cl-find-if 1241 43%
- apply 1241 43%
- cl-find 1241 43%
- apply 1241 43%
- cl-position 1241 43%
- cl--position 1241 43%
- #<compiled 0x41474245> 1241 43%
- projectile-verify-files 1027 36%
- cl-every 1027 36%
- projectile-verify-file 1027 36%
file-expand-wildcards 1027 36%
+ projectile-go-project-p 180 6%
+ projectile-cabal-project-p 34 1%
maginemu
on 11 Oct 2017
@mriehl Yeah, that's very inefficient indeed, but on the other hand once the type is detected it is cached. However, on a bigger project the initial delay might be quite significant. If someone has a better way to detect go projects - I'll all ears. :-)
bbatsov
on 12 Oct 2017
I've tested this again locally - for I see that project types are properly cached and retried. Some of you should instrument projectile-project-type with edebug to see why the caching is not working for you.
bbatsov
on 12 Oct 2017
While edebugging, I noticed that my directory .emac.d gets detected as 'generic.
As it seems, this means that projectile-detect-project-type doesn't put the project-root into projectile-project-type-cache. (I checked and projectile-project-type-cache was empty.
This can perhaps be the cause why editing my init.el is so extremely slow.....
Hope this helps a bit.
Let me know if you need more info.
rreckel
on 12 Oct 2017
Also had this issue and disabled the modeline component. Emacs was nearly unusable.
Does this callback need to happen every time the cursor moves? I would imagine that if it fired on switching buffers that would be sufficient.
pcmantz
on 12 Oct 2017
@rreckel Ah, that's probably it! I'm assuming a lot of people had issues outside projects.
@pcmantz Yeah, that's how it works by default - I wish there were ways to tune this behaviour. It caused so much pain, that I'm seriously considering making this static just on find-file, but that has downsides as well (e.g. when you're doing things like switch-project the modeline won't update).
bbatsov
on 12 Oct 2017
affected too :( , i'll try the workaround, regards
blallau
on 13 Oct 2017
I'm also affected on a small-ish php project (<5000 loc). This was causing 7-10s of lag between cursor moves, and persisted between Emacs restarts. The workaround works for me though!
idfellows
on 13 Oct 2017
Workaround works for me, thanks
blallau
on 13 Oct 2017
I'm experiencing the same high latency, but only noticed it when editing an elisp file. My C++ projects weren't affected, or were affected much less (that I didn't notice).
The workaround @bbatsov provided seems to have fixed the issue.
Thanks
nickenchev
on 16 Oct 2017
I'm affected as well. Mostly the problem appears when editing elisp files.
pronobis
on 18 Oct 2017
Let me make a wild arsed guess here....
Those of us who are experiencing this have something mounted that imposes a delay when queried.
Usual culprits are...
network drives like cifs/samba
or some sort of hardrive that is powered down if unused for awhile and takes awhile to spin up.
My guess is @bbatsov is not experiencing this because none of the directories between his current work directory and / have one of these mounts.
A simple fix would be stop looking for vc directories if you hit /home/* or /
Alas, I suspect this won't be bullet proof, so I suspect it would be better to turn off this feature by default.
john-carter
on 18 Oct 2017
That could be the case indeed. However, the lagging never stops and makes working with emacs pretty much impossible. As for making the feature optional, I totally agree. I don't even see the information displayed for projectile in the modeline until I expand the list (which I never do). I value performance and responsiveness much more.
pronobis
on 18 Oct 2017
Same as @nickenchev, C++ projects are OK. Python, elisp, etc are not.
carlosgeos
on 18 Oct 2017
I've met this issue while trying to edit elisp files for Prelude (~/.emacs.d/personal/). All of my python projects are fine. Workaround works for me too.
linnik
on 18 Oct 2017
The workaround also works for me. Before it local (on a SSD) and remote Python projects were completely unusable. However I should add that the machine I tested it on is very old (2008-ish).
D4N
on 18 Oct 2017
I'm having this problem in Python Projects on a 2016 MBP (SSD). Without the workaround it was completely unusable!
pbgc
on 18 Oct 2017
I thought I would give Projectile at test drive since it seems to be very popular. I ran directly into this problem and eventually worked out the modeline hack myself. The problem it seemed to me, as a naive, first time user, is that when there is no match for the project type, i.e. when it is “generic”, then the variable projectile-project-type remains set to nil. This means every movement of the cursor causes type detection to start again. And reading the comments above that means repeatedly walking over the entire project subtree!
I will keep with Projectile for a bit, but this has shaken my faith a little.
alastairdb
on 18 Oct 2017
I am experiencing this issue as well. I realized it was super slow at the time I was editing my personal emacs configs. The workaround suggested by @bbatsov also worked for me.
thiagowfx
on 18 Oct 2017
Just experienced this after updating. Couldn't even scroll properly. In any buffer (yaml, list-packages, ...) where projectile was enabled. Removing the type from the mode-line worked.
codesuki
on 19 Oct 2017
I got this issue with org-mode files. The workaround has fixed the issue for me.
MiloDavis
on 19 Oct 2017
Is a fix being worked on for this? Perhaps the new feature should be customizable so it can be more easily disabled?
wkmanire
on 20 Oct 2017
It's rather a problem, though workaround has worked. I wish that got a permanent fix. Thanks!
ysmolsky
on 20 Oct 2017
Just adding another voice in here--I stopped using emacs for a few days because it became basically unusable. bbatsov's mode line workaround has resolved the issue for me.
zingbretsenucb
on 20 Oct 2017
I'll drop my word here also:
It works well on easily distinguishable project types like python. But it lags heavily on my home folder which is under git too. I guess it's quite typical case if you check the comments mentioning this issue :)
Modeline workaround works great though, thanks a lot @bbatsov
valignatev
on 23 Oct 2017
Yeah, the problem is the caching of the generic project type. I'm sorry for
the pain this small change caused to so many of you. I'll try to find some
time to look if I can fix this or I'll simply revert it.
On Mon, Oct 23, 2017 at 06:16 Valentin Ignatyev notifications@github.com
wrote:
I'll drop my word here also:
It works well on easily distinguishable project types like python. But it
lags heavily on my home folder which is under git too. I guess it's quite
typical case if you check the comments mentioning this issue :)Modeline workaround works great though, thanks a lot @bbatsov
https://github.com/bbatsov—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/bbatsov/projectile/issues/1183#issuecomment-338543734,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGVykm0xQcfipjdqKNk_Znb1zPdtR5Rks5svBMegaJpZM4Pzlkr
.>
Best Regards,
Bozhidar Batsov
bbatsov
on 23 Oct 2017
Just to note I'm also affected (editing anything it seems with projectile-mode enabled).
mgrbyte
on 24 Oct 2017
Here’s an analysis that you might find useful. (And thank you for providing a workaround in the meantime.)
There are three elements that, combined, I think are enough to explain everybody’s observed behavior. I list them below. First, for this watching along, this is how project detection works:
To determine a project type the first time,
projectile-detect-project-typejust cycles through all known project types, until one reports success.Verifying _one_ type is typically very fast: just checking if a file, e.g. _requirements.txt_ exists in the root directory (or sometimes a small number of them, e.g. _Gemfile_ plus _spec_).
Once the correct project type is found, the result is cached in a local variable.
And here are the three elements that affect each of the above, making it slow in different scenarios:
By cycling through all known types, how many “attempts” are done varies depending on the _order_ of the list of known types. Consider:
Known types: Python, C++, Go, Java, Ruby.
Verifying whether a given project is Go is slow: it potentially visits the whole directory tree. It is _very_ slow, in particular, for non-Go projects!
Compare, using the list above:
Huge C++ project: fast to detect
Tiny Ruby project: fast to detect! Since it's tiny, the overhead of checking for Go is negligible.
Huge Java project: slow to detect _the first time_, because it recurses the tree in search for a single Go file. Faster afterwards, though: once the correct type was detected, it gets cached.
Negative results (i.e. when no type was assigned) are not cached. This explains why “generic” projects are permanently slow: it never hits the cache.
Some possible ameliorations to consider:
- pushing function-based project detection to the end of the list
- cache negative results (with care)
- use "*.go" as a wildcard, and assume there'll always be at least one Go file at top level
Long-term advice to make it faster.
do not use
file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.instead, modify
projectile-detect-project-typeto perform _only one_ call to obtain the list of top-level file names and then, for each attempted project type:- use set intersection if the marker files contain no wildcards
- use fnmatch if it does
- pass on that list of top-level file names to types that use a function to detect themselves
I hope this helps.
Thank you for your work on Projectile, I use it everyday and it improved my use of Emacs _a lot_.
dato
on 29 Oct 2017
I use this hacky workaround to cache the lighter for each filename:
(defvar projectile-lighter-cache (make-hash-table :test 'equal))
(setq projectile-mode-line
'(:eval
(or (gethash (buffer-file-name) projectile-lighter-cache)
(puthash (buffer-file-name)
(format " Projectile[%s(%s)]"
(projectile-project-name)
(projectile-project-type))
projectile-lighter-cache))))
juergenhoetzel
on 31 Oct 2017
do not use file-expand-wildcards, since (it seems to me) that will hit the disk for each of the (on average) 15 attempted project types, performing a full readdir() of the project root.
Indeed file-exists-p is much faster:
ELISP> (benchmark 100 '(file-exists-p "test.el"))
"Elapsed time: 0.002275s"
ELISP> (benchmark 100 '(file-expand-wildcards "test.el"))
"Elapsed time: 0.327794s (0.101474s in 1 GCs)"
I wonder why file-expand-wildcards is used in the first place. The use of wildcards isn't documented in
projectile-register-project-type and wildcards are not used in the implemented project types. So I just redefined this function:
(defun projectile-verify-file (file)
"Check whether FILE exists in the current project."
(file-exists-p (projectile-expand-root file)))
which resulted in an remarkable speedup.
juergenhoetzel
on 31 Oct 2017
I've fixed the fixing for 'generic project types, so this should work ok within projects, although to fix it better it needs to use buffer-local caching for the project type. (similar to what we have for projectile-cached-project-name) As I don't have time for this I'll just remove the project type from the the default modeline. If someone liked it - they can just add it back themselves.
bbatsov
on 31 Oct 2017
@dato Good points and very valid. I did a couple of tweaks to the project detection now. There are so many easy wins in Projectile here and there, unfortunately I rarely find time for it these days.
bbatsov
on 18 Jan 2018
Related issues
novakboskov
·
4Comments
DougBeney
·
3Comments
photex
·
6Comments
rschuetzler
·
3Comments
yangchenyun
·
8Comments
Most helpful comment
Hmm, I guess I underestimated this - I'll try to take a look at it soon, but in the meantime everyone can just set the modeline to
nilor something like old value: