Python-language-server: Unresolved import with packages under "src" when installed as editable

[eirikrye]

A project I am working on is structured with all its packages under a src folder in the project/workspace directory.

I am using a local virtualenv for development (.venv/bin/python), and the package is installed in editable mode, meaning it should be available in sys.path for that Python interpreter.

However, Visual Studio Code fails to resolve any of the imports from these packages, throwing unresolved-import errors and missing autocompletes.

Autocomplete instead suggests importing src.mypackage, instead of simply mypackage.

I have tried various variations of python.autoComplete.extraPaths: ["${workspaceFolder}/src"], ["src"], ["./src"], restarting the language server after each change, but none appear to work.

I have attempted with the beta `downloadChannel as well, to no avail.

The Output tab prints the following:

Starting Microsoft Python language server.
[...]
[Info  - 1:45:13 PM] Python search paths:
[...]
[Info  - 1:45:13 PM]     /Users/rye/Documents/my-project/.venv/lib/python3.7/site-packages
[Info  - 1:45:13 PM]     /Users/rye/Documents/my-project/src
[Info  - 1:45:13 PM] Configuration search paths:
[Info  - 1:45:13 PM]     /Users/rye/Documents/my-project/src
[Info  - 1:45:13 PM] Microsoft Python Language Server version 0.2.62.0

This seems to suggest that the language server has detected the folder both in the Python installation's sys.path, as well as the configured python.autoComplete.extraPaths

How do I get the language server to resolve my imports properly?

[jakebailey]

I don't believe editable installed packages work at the moment (#989), but I haven't yet tested to see if it had been working before and now is not.

If the code is in src/, then placing ./src into extraPaths (or PYTHONPATH via a .env file) should have worked, however. Are your imports all rooted at src? As in, if you do import foo.bar, is that located in src/foo/bar or similar? Or is the structure different because it's a package (which may be the case if you're able to do an editable install)?

[eirikrye]

Ah yes, you are correct. This seems to be related to editable installs (#989). Uninstalling the editable package in the virtualenv and reloading the language server has my autocompletes working. I didn't make the connection as I am pretty sure (anecdotal, yes) editable installs have worked fine in earlier versions.

This is not a proper fix though, as this now causes the VS Code embedded linters to throw similar errors (e.g. pylint's import-error) as the package is no longer in the interpreter's sys.path.

To answer your last question about the structure: I am building a package using the packaging structure described in this blog post, which has been adopted by some popular packages such as cryptography and twisted. My main reason is because editable installs without using src as the package root will cause Python to place the whole project directory in sys.path.

[jakebailey]

Interesting, I wouldn't have thought that doing an editable install would cause the imports to disappear if ./src is added to the search paths in the LS. My impression was that it'd work in both cases, as we'd find the import on disk and work normally (and just not find it via the pip installed method). Weird.

[eirikrye]

Yeah. It may be worth keeping this as a separate issue. I tried enabling the trace output for the language server, but nothing of interest appears in the log.

I created a minimal repository which can be used to locally reproduce the issue here: https://github.com/eirikrye/vscode-1013-repro

[j9ac9k]

I can confirm also having this issue (on macOS 10.14). I'll happily do any data gathering/testing to help address this.

[egabrum]

I should say, the problem is there not only when you try to import from editable installs (pip install -e ...), but also when you try to import from local ("non-packaged") files.

[jakebailey]

If you have a good reproduction we can test with, then we'd appreciate it. Many (if not all) of the currently open issues on unresolved imports are some variation of editable installs or egg imports (known issues), so something new would be great in a new issue for us to handle.

[eirikrye]

@egabrum Are you sure that is the case? My reproduction repo above shows that local imports (with extraPaths set properly) work fine as soon as you pip uninstall the editable install.

[jakebailey]

Good snapshot:

Good search paths:

Starting Microsoft Python language server.
[Info  - 3:32:43 PM] GetCurrentSearchPaths c:\Users\jabaile\test_projects\vscode-1013-repro\.venv\Scripts\python.exe 
[Info  - 3:32:43 PM] Python search paths:
[Info  - 3:32:43 PM]     c:\python37\dlls
[Info  - 3:32:43 PM]     c:\python37\lib
[Info  - 3:32:43 PM]     c:\python37
[Info  - 3:32:43 PM]     c:\users\jabaile\test_projects\vscode-1013-repro\.venv
[Info  - 3:32:43 PM]     c:\users\jabaile\test_projects\vscode-1013-repro\.venv\lib\site-packages
[Info  - 3:32:43 PM] Configuration search paths:
[Info  - 3:32:43 PM]     c:\Users\jabaile\test_projects\vscode-1013-repro\src
[Info  - 3:32:43 PM] Microsoft Python Language Server version 1.0.0.0
[Info  - 3:32:43 PM] Initializing for c:\Users\jabaile\test_projects\vscode-1013-repro\.venv\Scripts\python.exe

Bad snapshot:

Bad search paths:

Starting Microsoft Python language server.
[Info  - 3:29:45 PM] GetCurrentSearchPaths c:\Users\jabaile\test_projects\vscode-1013-repro\.venv\Scripts\python.exe 
[Info  - 3:29:45 PM] Python search paths:
[Info  - 3:29:45 PM]     c:\python37\dlls
[Info  - 3:29:45 PM]     c:\python37\lib
[Info  - 3:29:45 PM]     c:\python37
[Info  - 3:29:45 PM]     c:\users\jabaile\test_projects\vscode-1013-repro\.venv
[Info  - 3:29:45 PM]     c:\users\jabaile\test_projects\vscode-1013-repro\.venv\lib\site-packages
[Info  - 3:29:45 PM]     c:\users\jabaile\test_projects\vscode-1013-repro\src
[Info  - 3:29:45 PM] Configuration search paths:
[Info  - 3:29:45 PM]     c:\Users\jabaile\test_projects\vscode-1013-repro\src
[Info  - 3:29:45 PM] Microsoft Python Language Server version 1.0.0.0
[Info  - 3:29:45 PM] Initializing for c:\Users\jabaile\test_projects\vscode-1013-repro\.venv\Scripts\python.exe

Note how the "bad" output has ./src in the "Python search paths". This is likely due to the pth file entry added by the editable install, as that first list is coming from the Python interpreter (while the second is purely based on PYTHONPATH/extraPaths). See also: #537

I think the user paths should be searched before we get to the interpreter paths, though, so maybe that would be a fix. @AlexanderSher would know better what's going on.

--

An aside, but if we support pth files, that'd be nice to have just for the sake of people not needing to set up extraPaths or PYTHONPATH, but it sort of breaks when we are attempting to distinguish what is user code and what is not, which we achieve partially now by looking at the configured search paths (searchPaths in the init options) and seeing what's inside the root. If we have to do the same to the python interpreter's view, then we'll have trouble as virtualenvs are commonly placed inside the same root as the user code.

A quick search shows that there doesn't seem to be an easy way to disable pth files, which we could potentially use to distinguish them and treat them differently (i.e., apply the same rules as we do for searchPaths to figure out what is user code and what is not), not that it would 100% work either, since it's possible that a pth file points to something that isn't user code but exists in the workspace. This would require more thought.

EDIT: I believe that these get printed before the processing I do to classify the paths, so the difference being printed in the log is not actually where the path list ends up (but is probably related).

[egabrum]

@egabrum Are you sure that is the case? My reproduction repo above shows that local imports (with extraPaths set properly) work fine as soon as you pip uninstall the editable install.

@eirikrye , I haven't tried your repo but if I have 2 modules (one.py and two.py) in the same folder and I try to from two import blah, I get unresolved import 'variantIds'Python(unresolved-import)

If I re-enable Jedi, it's all good.

[jakebailey]

@egabrum Like this?

I'm not sure exactly where the error message you're talking about ("variantIds") would be coming from, at least in your example.

[egabrum]

@egabrum Like this?
I'm not sure exactly where the error message you're talking about ("variantIds") would be coming from, at least in your example.

Yes, exactly.
Sorry, ignore "variantIds", that's my actual code.

"python.jediEnabled": true, :

"python.jediEnabled": false, :

[jakebailey]

I think this issue is distinct from the editable install issues, then. I can't seem to make it happen, but if you have a project that does it, I'd appreciate it if you could post it somewhere and make a new issue.

(The message at that hover is #955, but the import not being resolved is another issue.)

[egabrum]

Interestingly, I get "unresolved import 'two'" with Python 3.7.1 and 3.6.6 64-bit, but not with 2.7.13 32-bit

[AlexanderSher]

Python 2.7 has different module resolution logic. Can you make a screenshot of your folder structure in VSCode?

[carmenbianca]

I have the same problem. I have created the tiniest possible repository that demonstrates the problem:

https://github.com/carmenbianca/vscodesrc

Just run:

python3 -mvenv venv
source venv/bin/activate
python setup.py develop

And then explore the project in VSCode. Module mypackage cannot be imported.

(mypackage) carmenbianca@thinkpad-a285 ~/Projektoj/vscodesrc $ tree                                                                                                                           
.
├── build
│   ├── bdist.linux-x86_64
│   └── lib
│       └── mypackage
│           └── __init__.py
├── dist
│   └── mypackage-0.0.1-py3.7.egg
├── setup.py
├── src
│   ├── mypackage
│   │   └── __init__.py
│   └── mypackage.egg-info
│       ├── dependency_links.txt
│       ├── PKG-INFO
│       ├── SOURCES.txt
│       └── top_level.txt
└── tests
    └── test_mypackage.py

9 directories, 9 files

[carmenbianca]

If you have a good reproduction we can test with, then we'd appreciate it. Many (if not all) of the currently open issues on unresolved imports are some variation of editable installs or egg imports (known issues), so something new would be great in a new issue for us to handle.

In addition to my previous post, I can reproduce this issue in the project I'm currently working on at https://github.com/fsfe/reuse-tool. It's basically the same thing: The language server cannot import reuse when it is installed via python setup.py develop.

[jakebailey]

pip's editable installs are effectively the same as setup.py develop ones; so it's not surprising that the behavior is the same, and would likely be under the same fix.

[jakebailey]

I'm going through all of these issues; both of those test repos work when you set extraPaths to src:

However, I'm aware that there are issues when this gets mixed with editable installs, but I'm trying to fully understand the scope of things before tackling that.

[j9ac9k]

Hi @jakebailey if I can help with testing something out, I'll gladly do so. I can see how the mixing up of these issues can be problematic from your end.

[jakebailey]

At the moment, our current recommendation is to use the extraPaths configuration to point the LS to your import roots. See the new documentation here: https://github.com/microsoft/python-language-server/blob/master/TROUBLESHOOTING.md#unresolved-import-warnings

A fix to editable installs was merged in #1183 (v0.3.1+), and will be available in the daily download channel shortly (and stable likely after). To switch to this for testing, you can set:

"python.analysis.downloadChannel": "daily"

Inferring this information automatically is more difficult than it seems, but there is likely work that can be done on that front in the future.