Pylance-release: Editable installs (pip install -e) aren't fully supported

[bobatsar]

Environment data

• Language Server version: Pylance language server 2020.6.1
• OS and version: Ubuntu 20.04
• Python version: Python 3.7.7

Expected behaviour

If python packages are installed in "develop" mode with

pip install -e package_name

code completion should work as normal.

Actual behaviour

Code completion is not working when packages are installed as "develop"

pip install -e package_name

When installing them normally with

pip install package_name

everything seems to work.

[erinxocon]

I wonder if this is related to my issue https://github.com/microsoft/pylance-release/issues/70

My package isn't installed by pip but it is symlinked into the site packages folder like pip would do in editable mode.

[jakebailey]

I do not believe we currently handle editable installs.

[wojtek-viirtue]

I can confirm the same as @bobatsar describes when using pipenv install -e. Installing a package normally everything works.

[zuozhiwen]

so do i
still not work

[NiklasRosenstein]

Hey, I'm encountering the same issue with editable installs. Are there any plans to tackle this issue? I'm also happy to look into contributing a fix, but I may need a pointer in the right direction. 🙂

[jakebailey]

Any updates we have will be posted here; I know we want to look into this but it will take some care to get right. There are other things that have taken priority over working on editable installs / pth files.

You could look at the pyright import resolver and path system to see what might be needed, but having personally added editable install support to the old language server previously, it may be tricky. (Or, it may be a simple bug, given we are supposed to be using the paths given by the interpreter and edtiable installs install pth files which modify what Python tell us.)

[erictraut]

Yeah, this is a tricky area of the code. I'd prefer that you don't attempt any changes. If you want to debug and report back, that'd be fine.

[jakebailey]

After looking at this code for another bug, there are a couple places that stand out as problematic in the current code. The main thing is that with editable installs, the interpreter provides the paths, so that's what we want to work. findPythonSearchPaths explicitly ignores paths in the workspace that are not also inside of the venv (so pth files pointing to workspace dirs doesn't work), and getPythonPathFromPythonInterpreter caches the result globally and never clears it on file update. By cutting out that first check, and then moving this cache to the import resolver (which is cleared on reconfig, or similar, or remove the cache entirely), it's likely we can make this work.

[jakebailey]

Nothing this as another example to use for testing: https://github.com/microsoft/pylance-release/issues/892#issuecomment-773508066

[mjmaurer]

If anyone is looking for a VSCode solution in the meantime, you can add something like:

    "python.analysis.extraPaths": [
        "vendor/django-allauth",
        "vendor/dj-rest-auth",
    ],

to your settings.

[erictraut]

@jakebailey, I think this should fully work now.

[jakebailey]

I believe it only works so long as the code you added to parse pth files by hand is triggered (and the pth file does not contain anything but strings; not true for libraries like matplotlib or pywin32)

In the case of a regular interpreter where that code is not used and we are relying on it to read sys.path (which should be almost always in Pylance), there was still the question as to what was going wrong that needs some debugging. My analysis in https://github.com/microsoft/pylance-release/issues/78#issuecomment-771067305 was wrong (which I figured out when you made your change), so I'd like to recheck.

[jakebailey]

I've tested it out, and things do work as expected, but only the first time things load. There's a global cache in front of our calls to python for path info which means that when the file watcher detects that the new package has been installed (the egg-link is created), we still return the old results. This cache is never cleared.

As far as I can tell, there are only a few uses of the call to python. Two of them only run on settings change, and the rest come from the import resolver which has another cache for this function specifically (that is cleared on import resolver invalidation, which happens on library changes). So, I think that it's safe for us to just remove this cache, and then editable installs will work both on first load and pip install -e . at runtime.

[jakebailey]

At this point (besides needing to force a restart when changing things while Pylance is running), I believe editable installs to be working.

I'd appreciate if those watching this thread who use editable installs could confirm that things work as expected. Pinging a few who have commented (sorry 🙂): @bobatsar @erinxocon @NiklasRosenstein @wojtek-viirtue @zuozhiwen @mjmaurer

[bobatsar]

@jakebailey on my side it seems good now.
I don't know if this fix has already landed (I am now on VSCode 1.54.3, python extension 2021.3.0 and pylance v2021.3.1) but my original issues seems to be gone.
Thanks. From my side this can be closed.

[jakebailey]

This issue has been fixed in version 2021.3.2, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202132-17-march-2021