Spyder: Linter stays half-dead on Windows

[OverLordGoldDragon]

demo.mp4 No clue how to reproduce, it's a first; other spyder instance is running a TensorFlow session. Linter does detect #TODO and unused imports, but not syntax errors, and code folding fails. Kernel restart, Spyder restart, Anaconda Powershell Prompt restart didn't help - didn't try restarting computer, but think it'll work.

---

Problems: (reproduced in 4.1.2)

1. Syntax error red flag persists after fixing the error¹
2. Syntax errors no longer detected
3. Unused variables no longer detected
4. Code folding markers misplaced⁴
5. Code folding folds only partially⁵

1: This is usually the first 'symptom'; everything goes downhill from there. The red flags can be dismissed by cut-repasting the entire code in the file.
4:

5:

[OverLordGoldDragon]

So, restarting computer did fix everything - though it seems to be a drastic measure where a powershell prompt restart should suffice.

[OverLordGoldDragon]

Actually, nevermind; restarting after this bug (thinks vars are undefined when they are defined), linter's back to even more dead than it was (now it doesn't detect unused imports). This, though, is only on the --new-instance - but I doubt restarting both will fix it, as I tried that before I restarted computer.

[OverLordGoldDragon]

Reinstalled Spyder in both environments; the problems persist. The cause seems to be many long files open at once; the lines sum to over 20,000. Restarting with only 2000, all problems vanish - but simply closing the files and leaving 2000 lines' worth did not suffice.

[ccordoba12]

@andfoy, please take a look at this one. The problem seems to be on the PyLS side.

[OverLordGoldDragon]

Code folding also immediately gets wrecked upon opening a large file - from folding only parts of a function, to fold icons disappearing entirely.

[ccordoba12]

Folding depends on the same machinery that provides linting and code completion, to that's not surprising. But thanks for letting us know about it.

[OverLordGoldDragon]

The sidebar next to scrollbar that adds markers to highlighted variables and e.g. #TODO also gets confused with code folding, highlighting where it would with _unfolded_ code: img. I don't know if this stuff if 'obvious' as I'm unfamiliar with linter mechanisms, so I'll just report the bugs I see.

[OverLordGoldDragon]

The orange triangle shows nothing when hovered over - a first. This is on a recently-started Spyder instance with several mid-sized files open.

[OverLordGoldDragon]

Another example:

[ccordoba12]

But you said that this happens after some time working with Spyder and in large files, right?

[OverLordGoldDragon]

@ccordoba12 This time, I don't know about 'large'; the lines sum-total is 1865, spread across 6 files, with no single file exceeding 810 lines. But yes, this happens only after ~3+ hrs - don't know if simply idling counts, but continued work does.

[ccordoba12]

I found this problem too when the Python language server dies in the background. With the fixes implemented in PR #12020, this shouldn't be a problem anymore.

@OverLordGoldDragon, please comment here after 4.1.2 if problems are not fixed after restarting the server.

[OverLordGoldDragon]

@ccordoba12 I don't know what "the server" is, but the problems persist; the linter's been dying sooner lately for some reason, and the "death rate" seems to have carried over precisely to 4.1.2.

I'll summarize these problems in the main post.

[OverLordGoldDragon]

Actually - it just came back to life, and everything seems to be working properly now. So it just didn't realize its death until ~2 hours later - and the revival process involved a brief freeze "LSP Python" saying "restarting" (or alike). And I just right-clicked it, and it has an option to restart manually.

Alright, I'll give this a go next time it happens - but this should be included in 4.1.2's release notes.

[ccordoba12]

And I just right-clicked it, and it has an option to restart manually.

That's what I was talking about when I said that you can restart the server manually.

Alright, I'll give this a go next time it happens - but this should be included in 4.1.2's release notes.

It is. It's the first entry in the changelog for 4.1.2.

[OverLordGoldDragon]

@ccordoba12 To notify - the restart always works, but LSP dies rather frequently, and sometimes many times in a row just minutes after restarting. This makes me miss problems for the time being otherwise highlighted by the linter; would help to stabilize its uptime.

[OverLordGoldDragon]

@ccordoba12 I've noticed that Invalid Syntax, followed by "dead linter", is triggered almost exclusively by . (__getattr__) - e.g. .get, .append, .values. This is consistent with 4.1.2, unsure of earlier.

[ccordoba12]

Thanks for the pointer @OverLordGoldDragon! I confirmed this generates an internal error in Jedi, which probably kills the linter afterwards on Windows (that's not the case on Linux, which is my main OS).

I'll test on Windows later on to check if that's the case.

[OverLordGoldDragon]

@ccordoba12 Came across another trigger I recall seeing previously: repeatedly undoing & redoing (Ctrl + Z / Y) a (copy-) paste overwriting lines of code. Haven't done . within 10+ mins of it. This one didn't yield Syntax Error - instead, between successive undos & redos, linter failed to detect a missing import (first it did, then no longer).

[ccordoba12]

@OverLordGoldDragon, thanks a lot for giving us a reproducible example!! That's all we needed!

After hours of debugging I finally found the cause (it was Windows specific). Right now I'm preparing a pull request and the fix will be available in our 4.1.4 version (to be released in mid June).

[OverLordGoldDragon]

@ccordoba12 Wonderful, glad to be of help.

[ccordoba12]

@OverLordGoldDragon, I opened #12771 to fix this and it's a very simple patch. If could apply it to your current installation and test it while coding, it'd be really helpful for us (unfortunately, I don't use Spyder extensively for coding).

[OverLordGoldDragon]

@ccordoba12 Sure, I'm down; it's just this one line of code I need to edit?

-       server_log = subprocess.PIPE
+       server_log = None if os.name == 'nt' else subprocess.PIPE

[ccordoba12]

Yep, that's the one.

[OverLordGoldDragon]

@ccordoba12 So: Spyder open for 24+ hrs, been editing for 12+ hrs, with multiple tabs totaling 2-3k lines, plenty of pastes & .'s, profiling, unit testing, etc. Typically, LSP would be well past its 20th restart; it had _zero_ - working good as new throughout. ... problem solved?

[ccordoba12]

@ccordoba12 So: Spyder open for 24+ hrs, been editing for 12+ hrs, with multiple tabs totaling 2-3k lines, plenty of pastes & .'s, profiling, unit testing, etc. Typically, LSP would be well past its 20th restart; it had zero - working good as new throughout.

Fantastic!!!

... problem solved?

The answer seems to be a resounding yes! Will merge my PR later today.