Pylance-release: Local imports not resolving correctly

[aaronsmith1234]

Environment data

• Language Server version: 2020.6.1
• OS and version: Windows 10 Version 10.0.18363 Build 18363
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.8.2 32bit

Expected behaviour

When trying to import another python file from the same directory as the current file (e.g. main.py and emailRelay.py are in the same directory, and I am importing emailRelay.py into main.py), the import should not be linted unless file I am importing does not exist.

Actual behaviour

When trying to import another python file from the same directory as the current file (e.g. main.py and emailRelay.py are in the same directory, and I am importing emailRelay.py into main.py), Pylance lints it as an unresolved import

[erictraut]

This is intended behavior. I presume you are using absolute imports. Python 2 allowed absolute imports to be resolved from the same directory as the importing file, but Python 3 always uses import search paths. By default, Pylance assumes that the root directory of your workspace is included in the search path. It also adds src if it's present, since it's common to place all sources within a local directory of that name. I'm guessing that your "main.py" and "emailRelay.py" are not in either of these default directories. If you would like to add additional directories to your import search path, you can do this by specifying these local directories in the python.analysis.extraPaths setting. All directories specified in this setting are assumed to be relative to the workspace root.

Alternatively, you can use relative paths (e.g. from .emailRelay import X).

[judej]

@aaronsmith1234, were you able to resolve the problem by using python.analysis.extraPaths? Please let us know if this is still an issue.

[aaronsmith1234]

Hey sorry, haven't had a chance to trial yet; will give it a shot in the next few days. The relative imports didn't work as expected; I'll provide some code samples when I give it a shot.

[RaghavaDhanya]

Is this behaviour intended to be different from old language server? This is making moving from old language server harder. I think the old language server behaviour is more useful
My use case: Developing a package xyz and I use absolute imports such as from xyz import abc inside the package, but pylance tries to resolve this in the installed packages rather than in current workspace where as the old language server did the opposite. Pylance forces me to either use relative imports(Which is not recommeded by PEP-8) or manually uninstall the package everytime from the development venv. The package is always in my dev venv for testing.

[erictraut]

Pylance will resolve imports in the workspace, but it will resolve from the top-level directory of your workspace. It will not (by default) resolve absolute imports from the same directory as your source file if that source file isn't at the top level of your workspace. That's intended because it matches the import resolution behaviors of the Python interpreter itself.

For more details about how Pylance/Pyright resolve imports, refer to this documentation.

Pylance/Pyright resolves imports in a way that is consistent with the Python interpreter and PEP 561, which dictates type checkers should resolve imports.

[erictraut]

I'll also say that in my experience, relative imports are preferable to absolute imports for local files. PEP 8 says that both are acceptable.

[mrucker]

I feel like this discussion is missing an important nuance. That is, Python code can be executed by one of two means: (1) as part of an import and (2) as a script file. Depending on how code is executed import resolution can behave very differently.

When Python code is executed because of an import statement the resolution behavior of Python is not to resolve absolute imports from the same directory as your source file. In this case it is possible to use both absolute and relative imports and pylance seems to work great with import resolution (or has so far for me).

When Python code is executed as a script file, however, then the Python interpreter adds the script's location to the search path
(see the first bullet here). The search path addition ends up allowing resolution of absolute imports from the script file's directory at execution time. Furthermore, Python code executed as a script is not allowed to even have relative imports. If it does there is a runtime error.

The behavior of script files noted above only seems to be a problem for pylance/pyright when a project contains scripts files that aren't in the root of the project. When this is the case I have noticed three problems with import resolution checks (I want to say clearly though that none of this has changed how my code runs, just the error messages I see):

1. When the imports are absolute from the script's directory it will say things are wrong even though they are fine.
2. When the imports are relative from the scripts directory it will say things are good even though they are not fine.
3. When the imports are absolute from the workspace's root it will say things are good even though they are not fine.

To solve (1) and (2) pylance/pyright would have to magically know how a user intends to run a code file. I'm not sure there is any way for pylance to know that... To solve number (3) I think pylance should maybe not make the assumption that the root is in the search path. While I understand it is intended as a convenience for users, I think it can be confusing. Maybe new projects should pop up a message asking users if they want workspace roots added to PYTHONPATH rather than just assuming?

It is also worth noting that issue (3) also applies to code that has been written assuming it will be executed through import. For example, if somebody is making their own package that they mean to distribute on PyPI. In this case the developer is likely savvy enough to not use absolute imports from root... or it will just works so long as the root package folder is in the workspace root or src which is pretty clearly the intention and I assume will almost always be the case. Still... I guess something to consider.

[jakebailey]

To solve (1) and (2) pylance/pyright would have to magically know how a user intends to run a code file. I'm not sure there is any way for pylance to know that...

This is a time old problem for Python LSs (microsoft/python-language-server#1602). I think jedi is able to handle these cases differently by using a more loose import mechanism, where it looks around each file's directory to find imports that might resolve. We've not done this because we'd rather show that an error could occur than to potentially mislead (even though looser is certainly friendlier).

(3) is an interesting case, though, as it represents a false negative. I'm not sure how we'd detect this case either, but I can say that the solution where we don't add the root as an import path would be a pretty significant change. Most people are going to fit into the category of projects that we currently support out of the box.

It'd be interesting to see if there were any tell-tale signs that a given file is a script, and if we could use that as a heuristic for determining imports. Or, add some configuration in mspythonconfig.json that says which files are scripts that need to have the root directory replaced with the script directory.

One thing you may also consider if you have a specific scripts directory is to add that directory as another workspace, so that we resolve imports for that "project" distinctly.

[erictraut]

Pyright (which forms the core of Pylance) solves this by allowing you to specify multiple "execution environments". Each script or program entry point represents its own execution environment. My team owns a large, complex python code base, and we have several dozen execution environments defined, some with different Python version requirements.

For more details about how to configure execution environments in Pylance or Pyright, check out this documentation. If you scroll to the bottom of that page, you'll find an example config file.

[jakebailey]

The only edge case I can see with execution environments would be the interpreter selection. Based on that, is it true that if you don't pick an interpreter in the config, you get the "default" one, which isn't what the editor is sending per-workspace but the global default?

If so, we may want to tweak things to use, say, the interpreter for the workspace VS Code declares that happens to contain the execution environment, to be more aligned with what the UI is saying.

[mrucker]

@jakebailey, yeah I couldn't think of any obvious better solutions myself while mulling it over today. I think the real problem with (1) and (2) is that they are likely going to cause the most trouble for more casual users since most expert users I imagine will likely use a pattern leveraging pip install -e . to separate re-usable code from scripts. Unfortunately, those are the users who I think are also going to be least likely to be able to find a solution on their own. Out of all the languages I've worked in up to this point in my career I've found learning Python imports to be one of the most frustrating language features ever. I haven't quite been able to put my finger on why that is though...

Anyway, I should end this by saying on the whole though I'm incredibly happy with Pylance so far. I think you've made a great tool for the community that has the potential to really take off. I've been using it for about a week now and have been super impressed.

[erictraut]

Out of all the languages I've worked in up to this point in my career I've found learning Python imports to be one of the most frustrating language features ever.

I wholeheartedly agree with you there!

I agree that there's a tradeoff here. Do we make things easy for casual users at the expense of leaving holes in error reporting? In some cases, we've decided to switch behaviors based on the "python.analysis.typeCheckingMode" setting, under the assumption that casual users will leave it "off", whereas more advanced users who are interested in typing will switch it to "basic" or "strict". However, in this particular case, I wouldn't feel comfortable changing the import logic based on the typeCheckingMode setting.

[jakebailey]

However, in this particular case, I wouldn't feel comfortable changing the import logic based on the typeCheckingMode setting.

Agree, I would rather introduce some other import resolution knobs, if possible, or do our best to document and improve execution environments.

[aaronsmith1234]

I'm still pretty new to Python, so a lot of this is new to me. I added some code to the repo below to show my main concern; if you look in "subfolder" and look at the "main2.py" file, you'll see that Pylance highlights the import as unresolved, but then the code runs successfully. I think that was my big concern; Pylance doesn't seem to be aligning with what Python is actually executing since Python can find the code and run it just fine. Again, I'm new to Python, so maybe there's a nuance I'm missing here.

Also take a look at "main3.py" in the subfolder as well. Here, I am using a relative import, as suggested. Pylance now can correctly resolve the import. However, when I try to actually run the code, it errors out with an invalid syntax error. If I get rid of the relative import, then Pylance again can't resolve the import, but the code runs successfully (since after removing the import main3=main2).

I think that the mismatch between the behavior of Pylance and the behavior of my Python install (which in this case was 3.8.3 32 bit, slightly different than what I opened the issue with, running on home machine!) is what should be aligned. Let me know if I'm missing something here!

https://github.com/aaronsmith1234/pythonSamples

[erictraut]

The Python import system is very confusing, even for experienced Python users.

The syntax you're using for relative imports is illegal. Pyright should have flagged it as such. I've fixed that hole in the parser. You can't use the form "import .import2". You need to use the form "from . import import2".

When you execute a script in Python, the interpreter automatically adds the directory of that script to the import search paths. When you execute "main3.py", you're implicitly adding that directory to the search path. Pyright has no way of knowing this is the main entry point to a script though. You haven't indicated that it's an independent "executable". That's why Pyright's configuration mechanism allows you to identify the various "execution environments" within your code base.

Try this experiment.

• Starting with your sample code, fix the relative import as I mentioned above:

from . import import2

• At the end of main.py, add the following lines:

import subfolder.main3
import subfolder.main2

• Run python main.py

What you'll find now is that the relative import works fine, but the import import2 generates a runtime exception, as predicted by Pyright.

[oh-mycode]

What do you offer to do in my case - I have a few packages which include contains python modules with the same name.
It looks like this:

    module1.py
    main.py - inside here there is `import module1`
p2
    module1.py
    main.py - inside here there is `import module1`

if I add python.analysis.extraPaths path to p1 and p2 and in p2/main.py try to go to definition of module1
VS Code opens p1/module.py

[erictraut]

@oh-mycode, for a more complex situation like yours, you can define different "execution environments" in your pyrightconfig.json file, each with independent root directories and extra paths. For details, check out this documentation.

[mrucker]

Just wanted to add one more comment to this thread. I used python.analysis.extrapaths for the first time today and it worked great. All go to references started working and import resolution warnings went away. I didn't even have to restart vs code to get it to work.

[JosephTLyons]

I ran into this issue today and still don't quite understand why it is happening.

src
main.py
enums.py

in main.cpp

from enums import BrushSize, Tool, Window # Import "enums" could not be resolvedPylance (reportMissingImports)

The Python interpret says that this is valid, but PyLance does not. I'm using PyCharm and VS Code these days. PyCharm's automatics import command is what wrote that line of code for the import. I'm just not sure why PyLance keeps thinking its an error. Is it that I have to a set a path setting for these, in VS Code, for each project?

What's really weird is I have other projects that use this same type of import (where the file is in the same directory as main.py) and PyLance isn't throwing an error on those like it is in this project, despite it seeming to be exactly the same.

[JosephTLyons]

Out of all the languages I've worked in up to this point in my career I've found learning Python imports to be one of the most frustrating language features ever. I haven't quite been able to put my finger on why that is though

If you haven't used Rust, give it a try and see if its import system tops Python's in how frustrating it is...

[mrucker]

@JosephTLyons Given your code above I'm guessing your Pylance problem (i.e., why it complains sometimes and not others) has to do with your workspace root directory. The import itself is completely fine so long as you are running main.py using the python command on the command line.

[JosephTLyons]

@JosephTLyons Given your code above I'm guessing your Pylance problem (i.e., why it complains sometimes and not others) has to do with your workspace root directory. The import itself is completely fine so long as you are running main.py using the python command on the command line.

Ok. It just sank in. I realize what is being said now. Yeah, I'm remoting into a RaspberryPi, so my root is not the project's root, but the entire linux system, so PyLance doesn't understand the import from that location and tosses and error. This makes sense. If I remote into the project directory itself, then I have no errors. Thanks, that helped clear it up.

[jakebailey]

I'm assuming you remoted in via the SSH remote extension, right? I've seen it just plop the editor straight into /, which is unfortunate for projects that need you to be in a different folder (plus, I'm really guessing that's going to make VSC unhappy as it attempts to set up file watchers for the entire filesystem).

I'm actually surprised that it works; last I knew, the SSH remote support for VSC created a virtual file system known only to the editor, but didn't actually provide "real" paths to LSs.

[ewerybody]

Oh well. This seems to be a returning issue, right? I mean I see the point:
Not all python files are supposed to run as scripts and highlightig this issue promotes more explicit code 👍 and Pylance team deserves applause for coming forward like this!

On the other hand: At least for me this has never been an issue with PyDev or Jedi and is highlighting something that just works IF a file is run as script! ...

So I wonder if it would be feasable to handle imports in an if __name__ == '__main__':-block accordingly and highlight only these outside such a block as unresolvable 🤔 Isn't that a pretty good "script-indicator"?

[jakebailey]

@ewerybody See #253 for script specific things. We don't really want to copy methods that lead to false negatives in import resolution; jedi's import resolution is "friendly" but isn't strict at all as a result

[ewerybody]

Yeah that's what I was referring to with:

... highlightig this issue promotes more explicit code 👍 and Pylance team deserves applause for coming forward like this!

🙂 For the 3rd paragraph: Shall I post this over there again?

[jakebailey]

No, that's okay. I'm already aware of heuristics about trying to use these main blocks to deduce the context, but I'm not certain it's a good idea. I linked that issue if you wanted to follow support for script-related fixes.

[tapetersen]

While looking at if __name__ == "__main__": blocks can often tell if it's to be executable it's used for both execution as script and as a module using python -m package.path and absolute imports are resolved from the script dir in the former case but not the latter.

A decent heuristic that hasn't been mentioned though could be checking for a shebang line as that is only really needed and used if executed as a script.

[dutzi]

I created a file named pyrightconfig.json, placed it in my project's root directory, and populated it with:

{
  "executionEnvironments": [
    {
      "root": "packages/core",
      "extraPaths": ["./"]
    },
    {
      "root": "packages/parsers",
      "extraPaths": ["./"]
    }
  ]
}

Worked great!

[saryeHaddadi]

Hello, I ran into the same issue (multiple python files, each in its own folder, not the projet root folder, and each file performing import of scripts inside its own folder).
I feel that making more visible how to handle that situation by configuring a pyrightconfig.json would be nice. A sort of official howto for this.

This issue has been my frustation n°1, I am glad to have found a solution. Thank you.

[bharathwajan]

i am using anaconda(python 3.8.5) in vscode
i am importing the y.py to the x.py by saying # note : both my x and y are on the same folder named blog
structure of the folder is
blog
x.py
y.py
the statement i used for importing is

from .x import func_name

but i am having issues with the relative import saying that:
Exception has occurred: ImportError
attempted relative import with no known parent package
File "C:\Users\lab\bharathwajan\blog\x.py", line 2, in
from .models import data

[jakebailey]

@bharathwajan I don't think this is related; I believe you're saying that we are reporting no error, but when you run the code, there is an error, which is different than this issue. I'd appreciate that in a new report.

If you stuck a __init__.py in the folder, I believe it would fix your problem, but that new issue could be handled by us knowing that imports will fail without that file (if it does fix it as I believe it does).

[bharathwajan]

i already had a file called __init__.py in my folder but eventhough its
showing me error

On Wed, 13 Jan, 2021, 12:04 am Jake Bailey, [e-mail]
wrote:

@bharathwajan https://github.com/bharathwajan I don't think this is
related; I believe you're saying that we are reporting no error, but when
you run the code, there is an error, which is different than this issue.
I'd appreciate that in a new report.

If you stuck a __init__.py in the folder, I believe it would fix your
problem, but that new issue could be handled by us knowing that imports
will fail without that file (if it does fix it as I believe it does).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/pylance-release/issues/68#issuecomment-758854386,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKT3EI65L2OYXJJL5DHJGKTSZSI25ANCNFSM4OTKA6WQ
.

[heejaechang]

can you try setting this hidden option, in setting.json, "python.analysis.useImportHeuristic": true, and see whether it helps to resolve local imports? and whether it causes something to break? or whether it behaves differently than what you would expect?

[ricardoprins]

can you try setting this hidden option, in setting.json, "python.analysis.useImportHeuristic": true, and see whether it helps to resolve local imports? and whether it causes something to break? or whether it behaves differently than what you would expect?

For me it helped perfectly. Thanks!

[ajkleinjsq]

@heejaechang Thanks for the tip! I am testing this currently but I wanted to highlight a use case that may be related and was not solved by turning on the hidden setting:

In our codebase, we have a module's __init__.py file that re-exports another module with the statement from .othermodule import *. Whether or not this is best practice aside, I would still expect Pylance to correctly route back to the source code when I click on a reference in the codebase and select "Go to definition". Instead, I get routed back to the __init__.py file.

In addition, and potentially a side effect of this behavior, is that when trying to add type definitions to a function, Pylance errors with "Module not allowed in this context". So for example

def a_function() -> Tuple[ClassReference, bool] {
  pass
}

the ClassReference will show the error.

If I can provide any more info or clarification please let me know! Thanks for your consideration and work on this issue.

[jakebailey]

@ajkleinjsq I don't think that's related to this issue, and we'd appreciate a new issue with a repro.