Python-language-server: Stuck at analyzing in background and causing high cpu load

[kaushall]

Environment data

• VS Code version: 1.36.0 (Stable Channel)
• Extension version (ms-python.python): 2019.6.22090
• OS and version: Ubuntu 18.04 LTS
• Python version (& distribution if applicable, e.g. Anaconda): 2.7.15
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv

============================================================

Logs

User belongs to experiment group 'AlwaysDisplayTestExplorer - control'
Starting Microsoft Python language server.
[Info - 10:33:32 AM] Analysis cache path: /home/kaushal/.cache/Microsoft/Python Language Server
[Info - 10:33:32 AM] GetCurrentSearchPaths /home/kaushal/microfinance/venv/bin/python
[Info - 10:33:33 AM] Interpreter search paths:
[Info - 10:33:33 AM] /home/kaushal/microfinance/venv/lib/python2.7
[Info - 10:33:33 AM] /home/kaushal/microfinance/venv/lib/python2.7/lib-dynload
[Info - 10:33:33 AM] /usr/lib/python2.7
[Info - 10:33:33 AM] /usr/lib/python2.7/plat-x86_64-linux-gnu
[Info - 10:33:33 AM] /usr/lib/python2.7/lib-tk
[Info - 10:33:33 AM] /home/kaushal/microfinance/venv/local/lib/python2.7/site-packages
[Info - 10:33:33 AM] /home/kaushal/microfinance/venv/lib/python2.7/site-packages
[Info - 10:33:33 AM] User search paths:
[Info - 10:33:35 AM] Microsoft Python Language Server version 0.3.21.0
[Info - 10:33:35 AM] Initializing for /home/kaushal/microfinance/venv/bin/python
.
.
.
Analysis of pandas.core.dtypes.dtypes(Library) completed in 5.9831 ms.
Analysis of pandas.core.dtypes.api(Library) completed in 0.7383 ms.
Analysis of pandas.io.json.table_schema(Library) completed in 1.6775 ms.
Analysis of pandas.core.indexing(Library) completed in 32.862 ms.
Analysis of pandas.util.testing(Library) completed in 49.5013 ms.
Analysis of pandas.core.sorting(Library) completed in 12.6855 ms.
Analysis of pandas.plotting._misc(Library) completed in 12.8782 ms.
Analysis of pandas.testing(Library) completed in 0.1046 ms.
Analysis of pandas.core.algorithms(Library) completed in 34.2249 ms.
Analysis of pandas.core.base(Library) completed in 21.2535 ms.
Analysis of pandas.core.missing(Library) completed in 21.3684 ms.
Analysis of pandas.core.util.hashing(Library) completed in 6.393 ms.
Analysis of pandas.core.indexes.frozen(Library) completed in 5.9272 ms.
Analysis of pandas.io.formats.format(Library) completed in 63.51 ms.
Analysis of pandas.core.internals(Library) completed in 101.1854 ms.

Screenshot

[raaaaaymond]

Yep. I was having the same problem before. It was resolved for a while by some updates, and now I'm having the same problem again immediately after updating to 1.36. My environment is:

VSCode 1.36.0
Server version 0.3.20.0
Python extension version 2019.6.22090
MacOS 10.14.5
Python 3.7.2 (via Pyenv, no virtual environment)

[jakebailey]

I'm not sure this could be related to any VS Code update; the same version of the language server would be downloaded by any version of VS Code (currently 0.3.20). This is likely something else.

[jakebailey]

We just built 0.3.22, which is available in the beta/daily channels. It contains some fixes which we believe will help solve some of the leaks. You can set this and reload to update:

"python.analysis.downloadChannel": "beta"

[kaushall]

@jakebailey

I'm still facing this issue.

Current extension version (ms-python.python): 2019.6.24221

[kaushall]

Here is the version of language server,

[Info - 10:39:46 AM] Microsoft Python Language Server version 0.3.22.0
[Info - 10:39:46 AM] Initializing for /home/kaushal/microfinance/venv/bin/python

[AlexanderSher]

Do you have this repro on single unsaved document?

[kaushall]

No, I've closed the Python file just for screenshot purpose.

@AlexanderSher

[jakebailey]

To be clear, we are asking for the code and installed package list that triggers this problem; it's not easy to fix something we can't reproduce and debug.

[AlexanderSher]

Ok, so looks like we have a bug with six.moves: https://github.com/microsoft/python-language-server/issues/746
But it doesn't hang anything by itself. @kaushall, do you have a file with a minimal repro or it requires a specific project?

[kaushall]

I understand your concern and I'll try to provide maximum details I can. @jakebailey

@AlexanderSher It was working fine before the last update of the language server.

do you have a file with a minimal repro or it requires a specific project?
No, ,maybe it's specific to project, because whichever file I open it will stuck randomly at analysis stage. Sometime it will stuck on Pandas library and sometime on Six library.

How it is reproduced?

Open vs code >> open project >> select python file >> After this it will activate extension and on analysis stage it will stuck.

[AlexanderSher]

Can you add a new python file to your project with only import pandas, and attach full log as a file to the bug?

[kaushall]

Alright, Provide me few minutes. @AlexanderSher

[kaushall]

@AlexanderSher I tried with your instruction and analysis completed successfully this time.

[AlexanderSher]

Ok, so pandas works by itself. Can you add to this new file all imports from the file that hangs, close all files, restart VSC and open this new file?

[kaushall]

@AlexanderSher It stuck again, Here is the result:

[AlexanderSher]

Ok, can you copy everything from the output window and attach it as a file to this bug?

[kaushall]

@AlexanderSher Here are your requested files.

requirement.txt => Requirements of virtual env
logs.txt => requested logs file
test.py.txt => requested python file (remove .txt entension)

requirement.txt
logs.txt
test.py.txt

[AlexanderSher]

Ok, I have a repro. Looking into it now.

[AlexanderSher]

Ok, so module in question is MySQLdb.connections.

[AlexanderSher]

This is a minimal repro I was able to extract:

from weakref import proxy

class C0(): pass
class C1(C0): pass
class C2(C1): pass
class C3(C2): pass
class C4(C3): pass
class C5(C4): pass
class C6(C5): pass
class C7(C6): pass
class C8(C7): pass

class Test():
    def __init__(self):
        p = proxy(self)

    F1 = C1
    F2 = C2
    F3 = C3
    F4 = C4
    F5 = C5
    F6 = C6
    F7 = C7
    F8 = C8

It doesn't hang, but actually takes a lot of time to analyze. On my machine, this code sample takes 23 seconds on Python 2 and 10 minutes for Python 3.

@jakebailey, do you remember if we have seen anything similar in the past?

[jakebailey]

I haven't seen anything like this since the old language server, no. That was to do with very long inheritance chains where all bases would be re-enqueued for analysis every time a child was added, but that's not how the new LS works.

[jakebailey]

This looks to be somewhat similar, actually:

As for 2 vs 3, note the difference in adding class methods versus not.

[jakebailey]

I cleaned the above up (constructed the enumerable with Concat only then Distinct), and the time moved to SetClassMembers instead:

[jakebailey]

I'm actually not sure why CreateSpecificType is getting called at all; the above example contains no generics.

[AlexanderSher]

Weakref.proxy is generic, I think

[AlexanderSher]

It also creates 400K instances of PythonClassType, so I'll be digging in that direction.

If I make a balanced tree out of the type hierarchy to have smaller inheritance chain:

class C0(): pass
class C1(C0): pass
class C2(C0): pass
class C3(C1): pass
class C4(C1): pass
class C5(C2): pass
class C6(C2): pass
class C7(C3): pass
class C8(C5): pass

it still takes about 90 seconds.

[jakebailey]

Test should not be a base of C7.

[jakebailey]

All classes are IPythonTemplateType, so we get a specific type of C1:

Then inside of CreateSpecificType, newBases contains Test, because it's in args and is an IGenericType.

I'm not sure how this can be avoided, other than not making every class IPythonTemplateType...

[jakebailey]

proxy returns ReferenceType[Test], which has all of the members of Test, so when a specific type of ReferenceType is constructed, SetClassMembers iterates over the members of Test and creates specific types of them, adding Test as a base, presumably because Test is filling in some T in a generic. That's where this is going wrong.

[kaushall]

@jakebailey @AlexanderSher @MikhailArkhipov Thanks for fixing the issue and your hard work.

Is this fix available in beta/daily channel?

[jakebailey]

Yes, in daily, 0.3.27 and above.

[kaushall]

Issue seems to be fixed now. Thank you @jakebailey @AlexanderSher @MikhailArkhipov

[xobs]

I'm experiencing this issue still.

Downloading https://pvsc.azureedge.net/python-language-server-stable/Python-Language-Server-win-x64.0.3.76.nupkg...
Language server dowload complete
Unpacking archive... done
[Info  - 3:26:06 PM] Microsoft Python Language Server version 0.3.76.0
[Info  - 3:26:07 PM] Analysis cache path: C:\Users\smcro\AppData\Local\Microsoft\Python Language Server
[Info  - 3:26:07 PM] GetCurrentSearchPaths C:\Python37\python.exe 

It's stuck at "Analyzing in background, 100 items left". It's using 1GB of RAM. It frequently gets into this state. Reloading the window spins a roulette wheel and has a chance of finishing, but it's not guaranteed.

[xobs]

Additionally, it's saying I have undefined variables such as isinstance, hasattr, and range. These aren't variables they are Python keywords.

[marcelorosensaft]

Hello,

I'm new here and I have the same problem. I read this and several other threads that were closed after the problem was solved but mine is still here.

What should I provide to get my problem solved and be able to continue using VSCode?

Thank you in advance,
Marcelo

[MikhailArkhipov]

@marcelorosensaft - Try Pylance language server instead. It is on the VS Code marketplace.

[marcelorosensaft]

Thank you, Mikhail!

I installed PyLance and it seems to work very well