Python-language-server: Add support for Implicit Namespace Packages

[AlexanderSher]

• Specification: https://www.python.org/dev/peps/pep-0420/
• General information about packages: https://docs.python.org/3/tutorial/modules.html#packages
• Import on Case-Insensitive Platforms: https://www.python.org/dev/peps/pep-0235/

[AlexanderSher]

Namespace packages can be presented in several directories. Consider the following folder structure:

├─ package1
│  └─ parent
│     └─ child
│        └─ one.py
├─ package2
│  └─ parent
│     └─ child
│        └─ two.py
└─ root
   └─ app.py

If app.py contains the following code:

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']
import parent.child.one
import parent.child.two

print(parent.child.__path__)
try: 
    print(parent.child.__file__)
except AttributeError:
    print("parent.child have no __file__ attribute")
print(parent.child.one.__file__)
print(parent.child.two.__file__)

executing python root\app.py will show

_NamespacePath(['path_to_project/package1/parent/child', 'path_to_project/package2/parent/child'])
parent.child have no __file__ attribute
path_to_project/package1/parent/child/one.py
path_to_project/package2/parent/child/two.py

[AlexanderSher]

from package import * doesn't import anything unless other imports have provided information for the namespace. If app.py contains the following code:

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']
from parent import *

print(child.__path__)

executing python root/app.py will produce an error:

  File "root/app.py", line 8, in <module>
    print(child.__path__)
NameError: name 'child' is not defined

Adding import to app.py that can provide information about the module

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']
from parent.child.one import *
from parent import *

print(child.__path__)

makes python root/app.py work normally:

_NamespacePath(['path_to_project/package1/parent/child', 'path_to_project/package2/parent/child'])

However, this won't add other modules inside package. For app.py like this:

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']

from parent.child.one import *
from parent.child import *
print(one.__file__)
print(two.__file__)

executing python root/app.py will produce an error:

Traceback (most recent call last):
  File "root/app.py", line 14, in <module>
    print(two.__file__)
NameError: name 'two' is not defined

[AlexanderSher]

Conflict resolution: if both regular and namespace packages are presented in search paths, namespace packages will be ignored:

├─ package1
│  └─ parent
│     └─ child
│        └─ one.py
├─ package2
│  └─ parent
│     └─ child
│        └─ two.py
├─ package3
│  ├─ parent
│  │  └─ child
│  │    └─ three.py
│  └─ __init__.py
└─ root
   └─ app.py

For app.py

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package3/', 'path_to_project/package2/']
import parent.child

print(parent.child.__path__)

executing python root/app.py will produce

_NamespacePath(['path_to_project/package3/parent/child'])

[AlexanderSher]

Functions can affect the import. A bit of edge case. Consider the following code in app.py:

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']
from parent.child import one
from parent.child import two
print(one.__file__)
print(two.__file__)

Execution of python root/app.py will produce:

path_to_project/package1/parent/child/one.py
path_to_project/package2/parent/child/two.py

If imports are moved into scope (i.e. function), there will be an error:

def add_imports():
    from parent.child import one
    from parent.child import two
    pass

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']
add_imports()
print(one.__file__)
print(two.__file__)

Traceback (most recent call last):
  File "root/app.py", line 12, in <module>
    print(one.__file__)
NameError: name 'one' is not defined



md5-55bed0500e007339c913e9c12ed7ae6c

path_to_project/package1/parent/child/one.py
path_to_project/package2/parent/child/two.py
```

[AlexanderSher]

Namespace packages can resolve dependencies through relative imports. For example, in a project structure:

├─ package1
│  └─ parent
│     └─ child1
│        └─ one.py
├─ package2
│  └─ parent
│     └─ child2
│        └─ two.py
└─ root
   └─ app.py

and file contents:

• one.py:

from ..child2.two import f2
def f1(): f2(); pass

• two.py:

def f1(): print("F1"); pass
def f2(): print("F2"); pass

• app.py

import sys
sys.path += ['path_to_project/package1/', 'path_to_project/package2/']

from parent.child1.one import *
f1()

executing python root/app.py prints F2

[jakebailey]

Do you know if anything you're updating will handle __all__? I've been going through the Django stuff, and found that they make use of * imports and __all__ modification quite a bit. See:

• https://github.com/django/django/blob/master/django/db/models/__init__.py
• https://docs.python.org/3/tutorial/modules.html#importing-from-a-package

EDIT: Testing on Django packages does seem to find stuff alright, but I can come up with an example like:

• whatever/__init__.py

from sys import version

__all__ = []

• app.py

from whatever import *

print(version)

Where we (currently) show version as being in scope, but running it fails saying version is not defined, which seems to hint that we might want to see how __all__ is going to play into this.

Forgive me if this isn't directly namespace package related, but I know you're updating the import resolution code, so I figured I'd ask. I'll make a new issue for this if needed.

[AlexanderSher]

This is actually the opposite task - support of the __init__.py, which can be only in ordinal packages. __all__ isn't the only option, you can simply define imports there. So yes, we need another epic for this.

[addisonElliott]

👍 Interested in this PR

In many Python projects, it's typical to alter the sys.path in unit tests to include the Python package.

This is one of the few things keeping me from fully switching to VS Code for Python dev...

[wtigotw]

👍 Interested in this PR

In many Python projects, it's typical to alter the sys.path in unit tests to include the Python package.

This is one of the few things keeping me from fully switching to VS Code for Python dev...

Same for me. This is the last major hurdle between me and VS Code as well.

[MikhailArkhipov]

Issues applicable to GA in this Epic are closed. So I am going to disconnect to remaining issues since they are not prioritized to GA and close the Epic.