Spyder: Indent guides makes the Editor extremely slow on large files

[CAM-Gerlach]

Problem Description

I've isolated the primary source of the slowness I was reporting earlier: it appears to be due to the Indent Guides, as it vanishes with them disabled and reappears with them enabled (particularly after closing/reopening the file).

The lags are directly proportional to file length in my testing; essentially unnoticeable on a 60 line file; tolerable but annoying on a 600 line file, and positively glacial on a 3000 line file (mainwindow.py). Turning off all the "Auto", showing blank spaces, wrap lines, CPU/memory usage and all introspection/analysis features made no difference.

What steps reproduce the problem?

1. Enable indent guides
2. Open mainwindow.py from the Spyder repo
3. Observe the majestic spectacle of a glacier moving

Versions

• Spyder version: 4.0.0.dev0 386403ed52
• Python version: 3.7.2
• Qt version: 5.9.6
• PyQt5 version: 5.9.2
• Operating System: Windows 8.1

Dependencies

pygments >=2.0    :  2.3.1 (OK)
qdarkstyle >=2.6.4:  2.6.5 (OK)
sphinx >=0.6.6    :  1.8.4 (OK)
pyls >=0.19.0     :  0.21.5 (OK)
nbconvert >=4.0   :  5.4.0 (OK)
pandas >=0.13.1   :  0.24.1 (OK)
numpy >=1.7       :  1.15.4 (OK)
sympy >=0.7.3     :  1.3 (OK)
cython >=0.21     :  0.29.4 (OK)
qtconsole >=4.2.0 :  4.4.3 (OK)
IPython >=4.0     :  7.2.0 (OK)
matplotlib >=2.0.0:  3.0.2 (OK)
pylint >=0.25     :  2.2.2 (OK)

[ccordoba12]

Thanks @CAM-Gerlach for finding the cause of this. It's is very serious indeed and we really need to fix for beta2.

@dalthviz, please work on this one.

[dalthviz]

@CAM-Gerlach @ccordoba12 I tried to reproduce this but I'm unable to experience the slow behaviour. Maybe there are other factors besides the Show Indent Guides being enable?

[bcolsen]

I can't reproduce this on Linux either. The file seem to be a little slow but usable with or without indent guides on my laptop.

I have had a large file slow to a crawl before at work, even typing was slow, but sure why.

[CAM-Gerlach]

Strange. I can repro it consistently with on the latest master in --safe-mode with clean prefs and just turning IG on and off, as tested just now. Its especially noticeable with two panes open; sometimes it makes the entire MainWindow UI lag. Sometimes scrolling is particularly slow and sometimes not so bad, but its particularly noticeable with typing (its hard to tell for sure in the GIF due to the low frame rate and being all scrolling, no apparent typing). I've also tested on my Fedora 28 Linux VM, and I see the same effect after enabling Indent Guides and closing and reopening the file; although its a little less dramatic past the innate slowness of the VM, typing and scrolling are still considerably less responsive with them enabled (though I don't notice the same difference in key repeat rate, possibly since a redraw isn't triggered every key, only every second or so).

I'm not on a blazing fast machine, but its an Ivy Bridge i7-3630QM with 4 cores/8 threads and nominally clocked at 2.4/3.4 (with a modest ~200 MHz manual downclock applied for power and thermal savings), and 24 GB DDR3-1600 RAM, a 850 Evo SSD and a GTX 670M.

[dalthviz]

@CAM-Gerlach besides the slow behaviour that you notice, do you see some Qt output like warnings or some sort of messages? I was checking and indeed the CPU usage increases when the Show Indent Guides option is enable, could you check your level of CPU usage and tell us the behaviour you see of that metric? Also, @bcolsen thank you for taking the time to check this :+1:

[CAM-Gerlach]

do you see some Qt output like warnings or some sort of messages?

Nope, not on either platform.

could you check your level of CPU usage and tell us the behaviour you see of that metric?

Sure. All on clean prefs, averaged over all 8 threads:

No IG, Idle: 10-15%
No IG, Scroll: 15-20%
No IG, Typing: 20-25%

IG on, Idle: 15-20%
IG on, Scroll: 20-25%
IG on, Typing: 25-40%

The increase isn't huge or consistent, but it does seem to be present.

[CAM-Gerlach]

@ccordoba12 @andfoy I installed PyLS 0.31 and pulled the latest master. Spyder launches fine and the code folding arrows now display and work as before. However, when I enable Indent Guides from the Source menu, the entire Spyder UI stops functioning correctly and starts flickering in and out as if in a movie "hacking" scene, and I get continuous error dialogs and a continual series of messages in the console. This happens enabling them from both preferences, from the Source menu, and having them enabled on startup, and on a project with two panes and many files down to a single 50-line temp.py file. Enabling and disabling other features from the source menu works, and so did indent guides from before this PR.

  File "C:\Users\C. A. M. Gerlach\Documents\dev\SpyderDev\spyder\spyder\plugins\editor\panels\indentationguides.py", line 63, in paintEvent

Traceback (most recent call last):
  File "C:\Users\C. A. M. Gerlach\Documents\dev\SpyderDev\spyder\spyder\plugins\editor\panels\indentationguides.py", line 63, in paintEvent
    total_whitespace = leading_whitespaces[max(line_number - 1, 0)]
KeyError: 1
Traceback (most recent call last):
  File "C:\Users\C. A. M. Gerlach\Documents\dev\SpyderDev\spyder\spyder\plugins\editor\panels\indentationguides.py", line 63, in paintEvent
    total_whitespace = leading_whitespaces[max(line_number - 1, 0)]
KeyError: 1
Traceback (most recent call last):
  File "C:\Users\C. A. M. Gerlach\Documents\dev\SpyderDev\spyder\spyder\plugins\editor\panels\indentationguides.py", line 63, in paintEvent
    total_whitespace = leading_whitespaces[max(line_number - 1, 0)]
KeyError: 1

In the attached system console, I get an endless stream of

QBackingStore::endPaint() called with active painter on backingstore paint device

with a few intermittent

QWidget::repaint: Recursive repaint detected

If I then disable indent guides, the flickering stops and Spyder seems to be usable as normal, although I still get the former message printed in the attached system console constantly.

Python 3.7.5, Win 8.1 x64 and Qt related stuff:

# packages in environment at C:\Miniconda3\envs\spyder-4:
#
# Name                    Version                   Build  Channel
pyqt                      5.9.2            py37h6538335_2
qt                        5.9.7            vc14h73c81de_0
qtawesome                 0.6.0                      py_0
qtconsole                 4.6.0                      py_0    spyder-ide
qtpy                      1.9.0                      py_0

[ccordoba12]

@CAM-Gerlach, please install python-language-server 0.31.1, which is available in our channel and try again.

@dalthviz, please test how indent guides are working on Windows to see if you can reproduce @CAM-Gerlach's error there.

[ok97465]

I installed python-language-server 0.31.1. I had the KeyError problem.
I can reproduce this problem in manajro and windows 10.

cloudpickle >=0.5.0 : 1.2.2 (OK)
pygments >=2.0 : 2.4.2 (OK)
qtconsole >=4.6.0 : 4.6.0 (OK)
nbconvert >=4.0 : 5.6.0 (OK)
sphinx >=0.6.6 : 2.2.1 (OK)
pylint >=0.25 : 2.4.3 (OK)
psutil >=0.3 : 5.6.3 (OK)
qtawesome >=0.5.7 : 0.6.0 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
zmq >=17 : 18.1.0 (OK)
chardet >=2.0.0 : 3.0.4 (OK)
numpydoc >=0.6.0 : 0.9.1 (OK)
spyder_kernels >=1.8.0;<2.0.0: 1.9.0.dev0 (OK)
qdarkstyle >=2.7 : 2.7 (OK)
atomicwrites >=1.2.0 : 1.3.0 (OK)
diff_match_patch >=20181111 : 20181111 (OK)
intervaltree : None (OK)
watchdog : None (OK)
keyring : None (OK)
pexpect >=4.4.0 : 4.7.0 (OK)
pympler : None (OK)
sympy >=0.7.3 : 1.4 (OK)
cython >=0.21 : 0.29.13 (OK)
IPython >=4.0 : 7.9.0 (OK)
matplotlib >=2.0.0 : 3.1.2 (OK)
pandas >=0.13.1 : 0.25.2 (OK)
numpy >=1.7 : 1.17.3 (OK)
scipy >=0.17.0 : 1.3.2 (OK)
pyls >=0.31.0;<0.32.0 : 0.31.1 (OK)
xdg >=0.26 : 0.26 (OK)
rtree >=0.8.3 : 0.8.3 (OK)

[CAM-Gerlach]

@ccordoba12 I can also confirm 0.31.1 and the latest Spyder master makes no difference, sorry; I still get the error.

[ccordoba12]

Yeah, it'll be fixed by PR #10782.

[CAM-Gerlach]

@ccordoba12 Pulling the latest master with #10782 applied and running PyLS 0.31.1, the latest available, the lag is still very bad with indent guides on, being only able to type 3.2 characters per second sustained even with one key on repeat over a 80 character, 0-indented line (300 ms per character), and there's around a 1000 ms interaction lag on the first character or any other changes. When I turn indent guides off, I get almost exactly double the characters per second, 6.3, and less than half the first keystroke delay. If I then also disable linting, code style warnings, and real-time code completion (which isn't triggering here, since there's no idle delay), things only improve a modest amount, to 6.9 max characters typed per second, with about the same small but noticeable interaction delay. For reference, in other editors (e.g. Notepad++), the rate at which I can type is 40+ chars per second with no interaction delay, limited only by keyboard repeat rate.

I can't find the original thread (not sure why I didn't link it here) where I did a ton of benchmarks, but from what I recall, this is closely in line with my previous findings, and has major usability impacts.

[eafpres]

I wanted to add that I see this on large files. Large is, say > 3000 lines. The lag is noticeable with guides on, goes away with guides off.

[ccordoba12]

Guides and folding are going to be turned off automatically and completely for >2000 files in Spyder 4.1.0. We'll try to come up with a better solution in a future release.

[eafpres]

Guides and folding are going to be turned off automatically and completely for >2000 files in Spyder 4.1.0. We'll try to come up with a better solution in a future release.

That sounds cool. Does it mean you can repro my symptoms?

We all really appreciate the effoft!

[ccordoba12]

That sounds cool. Does it mean you can repro my symptoms?

Indent guides are computed using the code folding information, and several users have reported that folding introduces serious lags in large files.

We all really appreciate the effoft!

Thanks! We are doing our best to improve the current situation.

[CAM-Gerlach]

That sounds cool. Does it mean you can repro my symptoms?

@eafpres I don't meant this as a negative criticism, sorry, but have you read the OP, lol? Or any of the other linked issues on which I and other Spyder developers (as well as users like yourself) reproduced and tested this issue at (exhaustive) length? They all describe, in far more specific and useful detail, the symptoms which you briefly summarize above.

[eafpres]

reproduced and tested this issue at (exhaustive) length? They all describe, in far more specific and useful detail, the symptoms which you briefly summarize above.

Yes, I read all of it, and the reason I was reading was because I had noticed the issue in certain cases. After reading the entire thread I seemed inconclusive. What you were reporting was not reproduced. Therefore, I thought adding a specific observation that it correlated to larger code files was useful. The near-term fix that is being put in seems to address this case, for which I am happy and grateful.