Pylance-release: Pylance Gets Confused If __init__.py Doesn't Match Package Structure

[iwoloschin]

Environment data

• Language Server version: 2020.7.4 (pyright a65343ef)
• OS and version: macOS 10.15.6
• Python version (& distribution if applicable, e.g. Anaconda): Homebrew Python 3.8.5

Expected behaviour

I'm trying to use asyncssh.scp() (see below for full code snippet). I do not expect Pylance to raise any error on this line and the code itself works.

Actual behaviour

Pylance provides an error like this:

"asyncssh.scp()" has type "Module" and is not callablePylance (reportGeneralTypeIssues)

Code Snippet / Additional information

The problem is caused by asyncssh/__init__.py importing the coroutine scp from the module scp:
https://github.com/ronf/asyncssh/blob/master/asyncssh/__init__.py#L83

This code causes Pylance to throw an error:

import asyncssh

async def do_scp():
    await asyncssh.scp("test.file", "localhost:~/")

I'm not sure what the right answer here is. Arguably structuring an import like that is confusing and bad practice, but at the same time python is allowing __init__.py to override the package's file structure, so it seems that Pylance should allow the same. Consider the following in a python REPL:

>>> import asyncssh
>>> help(asyncssh.scp)
Help on function scp in module asyncssh.scp:

async scp(srcpaths, dstpath=None, *, preserve=False, recurse=False, block_size=16384, progress_handler=None, error_handler=None, **kwargs)
    Copy files using SCP
...

[jakebailey]

I think that the library as-written will break if any part of the code does import asyncssh.scp, as scp is both a submodule and member of asyncssh.

If someone _ever_ does import asyncssh.scp in any part of the code, then asyncssh's scp member is going to be overwritten with the module instead. It's not generically possible to know which is going to be available and when, as it can happen in _any_ part of the code, even a library which depends on asyncssh and just so happens to have a line like that.

[iwoloschin]

Is that true? If I try this in the python REPL it seems to "do the right thing" (if we consider following whatever __init__.py says the "right thing"!).

>>> import asyncssh.scp
>>> help(asyncssh.scp)
Help on function scp in module asyncssh.scp:

I do agree that this is not a great way to write a python package. In this case the most correct fix might be just moving scp.py to _scp.py and then the module becomes somewhat hidden, while the public bits are pulled out in __init__.py. Unfortunately, if it's not my code I can't change it, but I'm happy to make an issue with asyncssh for discussion.

[jakebailey]

That's curious. I was just going over something similar to this and saw the opposite behavior where I could export a variable, read it, then do another import, and read a module (in explaining why import resolution in Python is difficult...). Perhaps there some edge case here that's not recognized.

I would be curious if the same code works in MPLS (the old Microsoft LS).

[iwoloschin]

I don't remember seeing MPLS ever complain about this line, but I haven't touched this code in a while.

[erictraut]

Thanks for reporting the problem. This is a bug in Pyright. It's currently evaluating the type of scp as a union of a module and a function. That's technically the correct type of the symbol, since it is being assigned both of those types, but the type checker should take into account the fact that the module is always overwritten by the function in this case. I'll work on a fix.

[jakebailey]

Ah, I see how this is working. asyncssh is "doing the import", so asyncssh.scp is remembered as previously imported, and will never be imported again by any user (even when you write import asyncssh.scp, as that's really just import asyncssh then some finagling to get scp as a member if needed; the relativeness of the import likely makes this sort of thing possible).

All references to scp after that point can only access the assignment. Given the two assignments are happening within asyncssh, I don't think things will break by noting that it's always overwritten.

This is opposed to the canonical case I use to show how this gets complicated:

foo/__init__.py:

bar = 1234

def print_bar():
    print(bar)

foo/bar.py:

# something

main.py:

import foo
print(foo.bar) # prints 1234
foo.print_bar() # prints 1234
import foo.bar
print(foo.bar) # prints <module foo.bar>
foo.print_bar() # prints <module foo.bar>

Related are the cases I've given in microsoft/pyright#439; this is a special case of my first example as the side effect is immediately squashed by assigning a variable.

[iwoloschin]

Thanks for the quick response!

Out of curiosity, should these sorts of bugs be filed here (pylance) or in the pyright repo? I kind of assumed that if I'm encountering them in pylance I should file them here, but happy to use the pyright repo if that's preferred!

[jakebailey]

Here is fine, both are watched. Here is mostly preferred if you are mainly using Pylance.

(And if you report it here, you'll see when fixes are released in Pylance specifically when we close issues. Pyright has a different release schedule than Pylance; Pylance is weekly-ish where as pyright doesn't have a set schedule.)

[erictraut]

I've introduced a fix specifically for the "from .A import A" case, which addresses the problem you are seeing. This fix will be in the next version of Pylance and Pyright.

[jakebailey]

This issue has been fixed in version 2020.8.0, which we've just released. You can find the changelog here: https://github.com/microsoft/pylance-release/blob/master/CHANGELOG.md#202080-5-august-2020