Pylance-release: v2021.1.3 broke pkgutil-style namespace packages - Missing Imports

[mvanderlee]

Environment data

• Language Server version: v2021.1.3
• OS and version: Windows 10 1909 18363.1316
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.7 - pyenv
• VirtualEnv - 20.1.0

Expected behaviour

I expect VSCode to be able to import from pkgutil-style namespaced packages.

Actual behaviour

It's not able to import from pkgutil-style namespaced packages

Logs

User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'LocalZMQKernel - experiment'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
User belongs to experiment group 'EnableIPyWidgets - experiment'
User belongs to experiment group 'RunByLine - experiment'
User belongs to experiment group 'CustomEditorSupport - control'
User belongs to experiment group 'pythonaa'
User belongs to experiment group 'pythonJoinMailingListVar2'
User belongs to experiment group 'pythonSendEntireLineToREPL'
User belongs to experiment group 'pythonInstallPylintButtonFirst'
> conda --version
> pyenv root
> python3.7 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3.6 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python3 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python2 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> python c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3.7 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3.6 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -3 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> py -2 c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> d:\git\sample-namespace-packages\pkgutil\pkg_b\.venv\Scripts\python.exe c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py -c "import sys;print(sys.executable)"
> conda info --json
Python interpreter path: .\.venv\Scripts\python.exe
Starting Pylance language server.
> conda env list
Error 2021-01-28 10:06:25: Failed to get conda environment list from conda [Error: spawn conda ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:81:21)] {
  errno: 'ENOENT',
  code: 'ENOENT',
  syscall: 'spawn conda',
  path: 'conda',
  spawnargs: [ 'env', 'list' ]
}
Error 2021-01-28 10:06:25: Detection of Python Interpreter for Command py and args -3.6 failed as file Python 3.6 not found!

 -3.7-64
 -2.7-64 does not exist
Error 2021-01-28 10:06:26: Detection of Python Interpreter for Command python3 and args  failed as file  does not exist
Error 2021-01-28 10:06:26: Failed to get interpreter information for 'C:\Users\maste\AppData\Local\Microsoft\WindowsApps\python.exe' [Error: Command failed: "C:\\Users\maste\AppData\Local\Microsoft\WindowsApps\python.exe" "c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py" "c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

    at ChildProcess.exithandler (child_process.js:304:12)
    at ChildProcess.emit (events.js:223:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\maste\\AppData\\Local\\Microsoft\\WindowsApps\\python.exe" "c:\\Users\\maste\\.vscode\\extensions\\ms-python.python-2021.1.502429796\\pythonFiles\\pyvsc-run-isolated.py" "c:\\Users\\maste\\.vscode\\extensions\\ms-python.python-2021.1.502429796\\pythonFiles\\interpreterInfo.py"'
}
Error 2021-01-28 10:06:26: Failed to get interpreter information for 'C:\Users\maste\AppData\Local\Microsoft\WindowsApps\python3.exe' [Error: Command failed: "C:\\Users\maste\AppData\Local\Microsoft\WindowsApps\python3.exe" "c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\pyvsc-run-isolated.py" "c:\Users\maste\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\interpreterInfo.py"
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

    at ChildProcess.exithandler (child_process.js:304:12)
    at ChildProcess.emit (events.js:223:5)
    at maybeClose (internal/child_process.js:1021:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)] {
  killed: false,
  code: 9009,
  signal: null,
  cmd: '"C:\\\\Users\\maste\\AppData\\Local\\Microsoft\\WindowsApps\\python3.exe" "c:\\Users\\maste\\.vscode\\extensions\\ms-python.python-2021.1.502429796\\pythonFiles\\pyvsc-run-isolated.py" "c:\\Users\\maste\\.vscode\\extensions\\ms-python.python-2021.1.502429796\\pythonFiles\\interpreterInfo.py"'
}
> conda --version

Code Snippet / Additional information

We use pkgutil-style namespace packages, as documented here:
https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages

I've forked their example project to reproduce the issue:
https://github.com/mvanderlee/sample-namespace-packages/tree/pylance_v2021.1.3

Steps:

1. Clone https://github.com/mvanderlee/sample-namespace-packages/tree/pylance_v2021.1.3 and ensure you're on the pylance_v2021.1.3 branch
2. Open sample-namespace-packages/pkgutil/pkg_b in VSCodee

3. Create virtual environment

   virtualenv .venv
   

4. Activate the virtualenv

   .\.venv\Scripts\activate
   

5. VSCode should've prompted you to use the virtualenv, or you can select it yourself by adding this to your settings.json

   {
       "python.pythonPath": ".venv\\Scripts\\python.exe"
   }
   

6. Install pkg_a

   pip install ..\pkg_a
   

7. Open example_pkg/b/__init__.py and observe that line 16 has an import error.
   

8. Revert Pylance to v2021.1.2 and observer that the import works fine.

Project layout:

Likely broken by this change: https://github.com/microsoft/pylance-release/issues/885#issuecomment-768812746

[erictraut]

I'll need to investigate more deeply, but from the screen shot above, it looks like both the local and installed packages are namespace modules. Therefore, "example_pkg.b.bb" should resolve to the local one. In other words, on first inspection, it looks like this is a regression that I introduced with my recent change.

[tibortakacs]

I can verify the behavior. v2021.1.2 works well, v2021.1.3 provides the same issue for me.

[erictraut]

Thanks for the excellent repro steps. I found and fixed the problem. The fix will be in the next release. Apologies for introducing the regression.

[mvanderlee]

Fantastic news @erictraut. Thank you for the quick fix.

[jakebailey]

This issue has been fixed in version 2021.2.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/main/CHANGELOG.md#202120-3-february-2021

[jdwestbrook]

I am running 2021.2.0 and I am still seeing the problem for packages which share
the same directory structure using -
__init__.py -> __path__ = __import__("pkgutil").extend_path(__path__, __name__)

This observation is for macos 11.2 on MacBook pro 15in 2018.

[mvanderlee]

2021.2.0 fixed it for me.

[jakebailey]

@jdwestbrook Do you have an example project somewhere we can test?

[jdwestbrook]

Here is an example -

https://github.com/rcsb/py-rcsb_utils_config.git

pyenv shell 3.9.1
git clone https://github.com/rcsb/py-rcsb_utils_config.git
cd py-rcsb_utils_config
pip install -e .

open vscode and then open file rcsb/utils/config/ConfigUtil.py

I am seeing that the import from rcsb.utils.io.FileUtil import FileUtil
cannot be resolved.

I am using the latest insiders build (daily) and the latest Pylance (daily) release.

[jakebailey]

If you don't do the editable installs, does it work as expected? Editable installs are not quite supported at the moment (#78), so it could be the case that your code was working due to a bug.

[jdwestbrook]

You nailed it. When explicitly installed the rcsb.utils.io module from PyPi, vscode finds the module.
I will check back on the progress of (#78) for support finding modules installed in the edit mode.
Thanks very much for you help!

[jakebailey]

Yeah, sorry about that. Thanks for that repo, though, as it'll be very useful when we try and get editable installs working.