Python-language-server: Recursive or looping symlinks can cause language server to repeatedly analyze files

[DonJayamanne]

Original issue can be found here along with the details https://github.com/Microsoft/vscode-python/issues/2613

I commented previously on #2297 but I found that symlinks were still causing problems after I grabbed a recent build today.

---

Environment data

• VS Code version: 1.27.2
• Extension version (available under the Extensions sidebar): 2018.9.0-alpha, retrieved at Sep 18, 10:30AM PDT.
• OS and version: Mac OS 10.13.6
• Python version (& distribution if applicable, e.g. Anaconda): 2.7.14
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: N/A

Actual behavior

The Python language server gets caught in a super deep nest of recursive symlinks. The repro case given completes quickly (there's nothing to analyze) but still demonstrates the problem, I think.

If there is actual code to parse/analyze it pretty much goes on forever. "Analyzing workspace, #### items remaining" goes down and then quickly shoots up, and this repeats over and over.

I can workaround this by adding the offending symlinks/folder structures to a VSCode exclusion list.

Expected behavior

The Python language server should be able to handle these structures in some reasonable way.

Steps to reproduce:

1. Create a folder structure like this

/code
  /test
    /some
      test.py
      /folder
        /symlink -> ../../some
        __init__.py (empty)
        lib.py (empty)

1. test.py only needs to contain import folder.lib.
2. Enable trace output for the language server and watch the logs, it gets progressively deeper and deeper into a repeated folder/symlink/folder/symlink/ folder structure.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Initializing for /usr/local/opt/python/bin/python2.7
Loading files from /Users/username/code/test/some
Parsing document file:///Users/username/code/test/some/test.py
Parsing document file:///Users/username/code/test/some/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/test.py at version -1
Parse complete for file:///Users/username/code/test/some/folder/__init__.py at version -1
Analysis queued for file:///Users/username/code/test/some/test.py
Analysis queued for file:///Users/username/code/test/some/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/lib.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/test.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/test.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/lib.py
Parse complete for file:///Users/username/code/test/some/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/__init__.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py
<more>
Waiting for parsing to complete
Parsing complete. Waiting for analysis entries to enqueue
Enqueue complete. Waiting for analysis to complete
Received new analysis for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

---

Symlinks to a common folder (e.g. two projects which have a symlink to the same folder) also get re-analyzed but that's usually okay as analysis is pretty fast in general.

[MikhailArkhipov]

Yeah, I might have solution

[MikhailArkhipov]

Most probably no longer applies, so for verification with https://github.com/Microsoft/python-language-server/issues/432

[AlexanderSher]

If we create correct PathResolverSnapshot, there should be no problem.

[jakebailey]

Tested using this specific example, the new LS does not loop infinitely. There are some other symlink oddities (long import hints, loss of import completion once a symlink is imported), but I'll file separate issues for those.

[jakebailey]

Reopening, there may be some other looping issues.

[astrozot]

I actually still experiment this issue with the latest VS Code version (1.32.3).

[Battleman]

From my side, I seem to have problems in looping too (characteristic of the "Analyzing in background, X items left"). But in my side, it's not the file/directory/workspace structure, but more the virtual environment. With a simple file (no import, one function), and using a conda env from another project I got absolutely no problem.
But when I import certain packages (I noticed sklearn poses problems), LS tries to analyses this and get in trouble. But for example, numpy (also "complex" package) poses no problem at all.

STRANGER YET, when going with a blank conda env, with only numpy and sklearn, the same import leads to no problem at all. So it's linked to packages in environment, but there also seems to be a third party factor.

Problematic case:

Environment

• VS code 1.32.3,
• Python 3.7.3 (with anaconda 3)
• Using conda env
• On Debian 10
  [edit] and extension version : 2019.3.6352 [/edit]
  

  Description of the problem

As many others, with certain workspaces, the "Analyzing in background, X items left" appears, grows to a certain point, sometimes decreases and grows again, until being stuck indefinitely. Sucks up CPU and memory usage.

"Simple file" in question:

```import numpy as np
from sklearn.model_selection import KFold

def main():
print("This python is not buggy")
return True

if __name__ == "__main__":
main()

### Content of the problematic conda env (sorry it's not minimal, I didn't find yet a minimal case)

_tflow_select 2.3.0 mkl
absl-py 0.7.0 py36_0
alembic 1.0.6 py36_0
apscheduler 3.5.3 py36_1000 conda-forge
asn1crypto 0.24.0 py36_0
astor 0.7.1 py36_0
astroid 2.1.0 py36_0
autopep8 1.4.3 py36_0
backcall 0.1.0 py36_0
beautifulsoup4 4.7.1 py36_1
blas 1.0 mkl
bleach 3.1.0 py36_0
c-ares 1.15.0 h7b6447c_1
ca-certificates 2018.12.5 0
certifi 2018.11.29 py36_0
cffi 1.11.5 py36he75722e_1
chardet 3.0.4 py36_1
click 7.0 py36_0
cloudpickle 0.6.1 py36_0
cryptography 2.4.2 py36h1ba5d50_0
cycler 0.10.0 py36_0
cytoolz 0.9.0.1 py36h14c3975_1
dask-core 1.0.0 py36_0
dbus 1.13.6 h746ee38_0
decorator 4.3.0 py36_0
entrypoints 0.3 py36_0
expat 2.2.6 he6710b0_0
flask 1.0.2 py36_1
flask-migrate 2.2.1 py36_1000 conda-forge
flask-sqlalchemy 2.3.2 py36_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
gast 0.2.2 py36_0
glib 2.56.2 hd408876_0
gmp 6.1.2 h6c8ec71_1
grpcio 1.16.1 py36hf8bcb03_1
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.9.0 py36h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
icu 58.2 h9c2bf20_1
idna 2.8 py36_0
imageio 2.4.1 py36_0
intel-openmp 2019.1 144
ipykernel 5.1.0 py36h39e3cac_0
ipython 7.2.0 py36h39e3cac_0
ipython_genutils 0.2.0 py36_0
ipywidgets 7.4.2 py36_0
isort 4.3.4 py36_0
itsdangerous 1.1.0 py36_0
jedi 0.13.2 py36_0
jinja2 2.10 py36_0
jpeg 9b h024ee3a_2
jsonschema 2.6.0 py36_0
jupyter 1.0.0 py36_7
jupyter_client 5.2.4 py36_0
jupyter_console 6.0.0 py36_0
jupyter_core 4.4.0 py36_0
keras-applications 1.0.6 py36_0
keras-preprocessing 1.0.5 py36_0
kiwisolver 1.0.1 py36hf484d3e_0
lazy-object-proxy 1.3.1 py36h14c3975_2
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc 7.2.0 h69d50b8_2
libgcc-ng 8.2.0 hdf63c60_1
libgfortran-ng 7.3.0 hdf63c60_0
libpng 1.6.36 hbc83047_0
libprotobuf 3.6.1 hd408876_0
libsodium 1.0.16 h1bed415_0
libstdcxx-ng 8.2.0 hdf63c60_1
libtiff 4.0.10 h2733197_1001
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.9 he19cac6_0
mako 1.0.7 pypi_0 pypi
markdown 3.0.1 py36_0
markupsafe 1.1.0 py36h7b6447c_0
matplotlib 3.0.2 py36h5429711_0
mccabe 0.6.1 py36_1
mistune 0.8.4 py36h7b6447c_0
mkl 2019.1 144
mkl_fft 1.0.10 py36ha843d7b_0
mkl_random 1.0.2 py36hd81dba3_0
nbconvert 5.3.1 py36_0
nbformat 4.4.0 py36_0
ncurses 6.1 he6710b0_1
networkx 2.2 py36_1
notebook 5.7.4 py36_0
numpy 1.15.4 py36h7e9f1db_0
numpy-base 1.15.4 py36hde5b4d6_0
olefile 0.46 py36_0
openssl 1.1.1a h7b6447c_0
pandas 0.23.4 py36h04863e7_0
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py36_1
parso 0.3.1 py36_0
pcre 8.42 h439df22_0
pexpect 4.6.0 py36_0
pickleshare 0.7.5 py36_0
pillow 5.4.1 py36h34e0f95_0
pip 18.0 pypi_0 pypi
prometheus_client 0.5.0 py36_0
prompt_toolkit 2.0.7 py36_0
protobuf 3.6.1 py36he6710b0_0
ptyprocess 0.6.0 py36_0
pycodestyle 2.4.0 py36_0
pycparser 2.19 py36_0
pygments 2.3.1 py36_0
pylint 2.2.2 py36_0
pyopenssl 18.0.0 py36_0
pyparsing 2.3.1 py36_0
pyqt 5.9.2 py36h05f1152_2
pysocks 1.6.8 py36_0
python 3.6.8 h0371630_0
python-dateutil 2.7.5 py36_0
python-editor 1.0.3 py36_0
pytz 2018.9 py36_0
pywavelets 1.0.1 py36hdd07704_0
pyyaml 3.13 py36h14c3975_0
pyzmq 17.1.2 py36h14c3975_0
qt 5.9.7 h5867ecd_1
qtconsole 4.4.3 py36_0
readline 7.0 h7b6447c_5
requests 2.21.0 py36_0
scikit-image 0.14.1 py36he6710b0_0
scikit-learn 0.20.2 py36hd81dba3_0
scipy 1.1.0 py36h7c811a0_2
send2trash 1.5.0 py36_0
setuptools 40.6.3 py36_0
sip 4.19.8 py36hf484d3e_0
six 1.12.0 py36_0
soupsieve 1.7.1 py36_0
sqlalchemy 1.2.16 py36h7b6447c_0
sqlite 3.26.0 h7b6447c_0
tensorboard 1.12.2 py36he6710b0_0
tensorflow 1.12.0 mkl_py36h69b6ba0_0
tensorflow-base 1.12.0 mkl_py36h3c3e929_0
termcolor 1.1.0 py36_1
terminado 0.8.1 py36_1
testpath 0.4.2 py36_0
tk 8.6.8 hbc83047_0
toolz 0.9.0 py36_0
tornado 5.1.1 py36h7b6447c_0
traitlets 4.3.2 py36_0
typed-ast 1.1.0 py36h14c3975_0
tzlocal 1.5.1 py36_0
urllib3 1.24.1 py36_0
wcwidth 0.1.7 py36_0
webencodings 0.5.1 py36_1
werkzeug 0.14.1 py36_0
wheel 0.32.3 py36_0
widgetsnbextension 3.4.2 py36_0
wrapt 1.11.0 py36h7b6447c_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zeromq 4.2.5 hf484d3e_1
zlib 1.2.11 h7b6447c_3

### Extract of the python output

Analysis of IPython.core.magics.history(Library) queued
Analysis of scipy.optimize._slsqp(Compiled) completed in 47.8488 ms.
Analysis of sklearn.externals.joblib.pool(Library) queued
Analysis of _locale(CompiledBuiltin) queued
Analysis of sre_compile(Library) queued
Analysis of distutils.command.install(Stub) completed in 48.1344 ms.
Analysis of nbformat.v2.nbbase(Library) queued
Analysis of IPython.core.magics.extension(Library) queued
Analysis of IPython.utils.terminal(Library) queued
Analysis of scipy.optimize.moduleTNC(Compiled) completed in 39.3443 ms.
Analysis of nbformat.v1.convert(Library) queued
Analysis of IPython.utils.dir2(Library) queued
Analysis of numpy.testing.noseclasses(Library) queued
Analysis of email.parser(Library) queued
Analysis of IPython.utils.process(Library) queued
Analysis of nbformat.v1.nbbase(Library) queued
Analysis of nbformat.v2.nbpy(Library) queued
Analysis of prompt_toolkit.filters.app(Library) queued
Analysis of notebook.services.contents.fileio(Library) queued
Analysis of sre_parse(Stub) queued
Analysis of prompt_toolkit.win32_types(Library) queued
Analysis of pygments.formatters.latex(Library) queued
Analysis of prompt_toolkit.renderer(Library) queued
Analysis of tornado.process(Library) queued
Analysis of asyncio.transports(Stub) queued
Analysis of asyncio.streams(Library) queued
Analysis of nbformat.v1.nbjson(Library) queued
Analysis of sklearn.externals.joblib.externals.loky.process_executor(Library) queued
Analysis of dummy_threading(Library) queued
Analysis of sre_parse(Library) queued
Analysis of IPython.lib.display(Library) queued
Analysis of prompt_toolkit.eventloop.context(Library) queued
Import: ipython_genutils.ipstruct /storage/anaconda3/envs/facesearch/lib/python3.6/site-packages/ipython_genutils/ipstruct.py
Analysis of distutils.log(Stub) completed in 74.345 ms.
Analysis of ipython_genutils.ipstruct(Library) queued
Analysis of prompt_toolkit.enums(Library) queued
Analysis of IPython.core.magics.execution(Library) queued
Analysis of IPython.core.page(Library) queued
Analysis of nbformat.v3(Library) queued
....
...
Analysis of _heapq(Compiled) completed in 16.2774 ms.
Analysis of scipy.interpolate._bsplines(Library) completed in 17.8444 ms.
Analysis of colorsys(Library) completed in 0.6466 ms.
Analysis of IPython.core.inputtransformer2(Library) queued
Analysis of unittest(Library) completed in 31.8242 ms.
Analysis of numpy.version(Library) completed in 0.1255 ms.
Analysis of sklearn.decomposition._online_lda(Compiled) completed in 0.3446 ms.
Analysis of scipy.interpolate.interpnd(Compiled) completed in 1.2661 ms.
Analysis of scipy.interpolate._ppoly(Compiled) completed in 9.8247 ms.
Analysis of numpy.core._multiarray_tests(Compiled) completed in 0.9455 ms.
Analysis of scipy.interpolate.dfitpack(Compiled) completed in 0.3231 ms.
Analysis of multiprocessing(Library) completed in 0.1187 ms.
Analysis of sklearn.externals.joblib.externals.loky.backend._win_wait(Library) queued
Analysis of ipykernel.kernelapp(Library) queued
Analysis of multiprocessing.pool(Stub) completed in 20.9803 ms.
Analysis of multiprocessing.process(Stub) completed in 0.305 ms.
Analysis of numpy.testing._private(Library) completed in 17.188 ms.
Analysis of _multiprocessing(Compiled) completed in 10.1877 ms.
Analysis of _hashlib(Compiled) completed in 0.4461 ms.
Analysis of _sha3(Compiled) completed in 15.738 ms.
Analysis of pycparser(Library) completed in 0.6777 ms.
Analysis of sklearn.externals.joblib.externals.loky.reusable_executor(Library) queued
Analysis of IPython.core.completer(Library) queued
Analysis of IPython.core.extensions(Library) queued
Analysis of _blake2(Compiled) completed in 17.729 ms.
Analysis of _sha512(Compiled) completed in 27.4286 ms.
Analysis of _sha256(Compiled) completed in 1.6649 ms.
Analysis of _md5(Compiled) completed in 3.1989 ms.
```
SO, some files are queued, while some are completed in a matter of ms.
I guess that is the problem, some recursivity in specific packages. Maybe two (or more) of the packages in the env are referencing each other, creating a loop. Point is, with a conda env having only numpy and sklearn, this issue does not appear.

Sorry if I mis-followed the guidelines...

[jakebailey]

Please file a new issue; this one is closed and is probably not related.

Regarding "only numpy and sklearn", there's a lot behind those libraries which may be affected by things relating to my explanation in https://github.com/Microsoft/python-language-server/issues/832#issuecomment-477688703.