Pylint: False positive and negative for E1101 with similarly named (sub-)modules.

[kaorihinata]

I've not been able to look into this too deeply, but I've come across an odd issue where a top-level module (which was never imported) is being preferred over a sub-module (in another top-level module.) I've confirmed that the issue has not been fixed in a preview release.

Steps to reproduce

Take the following file (pylint_example.py, in my case) as an example:

#! /usr/bin/env python3
"""This is an example of the top-level/sub-module issue."""
import asyncio
print(asyncio.subprocess.Process)
print(asyncio.subprocess.pwd)

Current behavior

Pylint seems to be erroneously checking asyncio.subprocess members against the top-level subprocess module. Note that Process (a member of asyncio.subprocess) fails, while pwd (a member of subprocess) succeeds:

************* Module pylint_example
pylint_example.py:4:6: E1101: Module 'subprocess' has no 'Process' member (no-member)

--------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: -6.67/10, +0.00)

Expected behavior

I assume the actual output should look something like this:

************* Module pylint_example
pylint_example.py:5:6: E1101: Module 'asyncio.subprocess' has no 'pwd' member (no-member)

--------------------------------------------------------------------
Your code has been rated at -6.67/10 (previous run: -6.67/10, +0.00)

pylint --version output

Result of pylint --version output:

pylint 2.7.2
astroid 2.5.1
Python 3.9.1 (default, Feb 18 2021, 20:12:08)
[Clang 12.0.0 (clang-1200.0.32.29)]

[Pierre-Sassoulas]

Thanks for creating the issue. I can reproduce this, and there is a Process in asyncio.subprocess.

[kaorihinata]

It's a bizarre one. I've been trying to reproduce it using modules in the current working directory, and while the particular combination of __all__, relative imports, and importing subprocess within asyncio.subprocess that the asyncio.subprocess module uses complicates the resulting AST, and adds an additional item to non_opaque_inference_results, I've not been able to achieve the exact same result.

When reproducing the actual bug, Module.asyncio.subprocess does not show up in non_opaque_inference_results. However, when I run a similar test using modules in the current directory, the correct module does still show up in non_opqaue_inference_results. If I can find the exact environmental conditions that are causing the module to never make it into non_opaque_inference_results then I'll update the ticket.

(Edit: This may actually be a PyCQA/astroid issue. If it is I can open a new one over there if need be.)

[kaorihinata]

Alright. I figured out how to reproduce it generically, and it may actually be a case of Python breaking its own rules.

example.zip

Essentially, asyncio's __init__.py imports its subprocess module using:

from .subprocess import *

That's it.

When you create a file-based module and attempt to perform a * import you do not end up with the module in your namespace. However, when you perform a * import (relatively) within a directory-based module you do end up with the module in your namespace. Essentially, subprocess (asyncio.subprocess, in this case) ends up in asyncio's module namespace, but it's not clear to me whether or not it actually should be given there's no accompanying from . import subprocess (which would actually satisfy pylint as well.)

[kaorihinata]

Just an update. I'm asking upstream to clarify whether or not this is their issue, as it certainly appears to be based on the documentation, as well as the behavior of both the import statement, and importlib in essentially all other circumstances. See: https://bugs.python.org/issue43477

[kaorihinata]

Well, it's taken almost a week, but the author has refused to take responsibility for the behavior in question, and is not interested in changing it either, even if it's inconsistent with how the same statement acts when used literally everywhere else.

He's essentially insisting that performing:

from .x import *  # `from .x import y` would also work.

...within a package (and only within a package) should be equivalent to:

from .x import *  # `from .x import y` would also work.
from . import x

He insisted it was part of the import process, but no part of the import process requires that the child be stored on the parent. I've checked all of importlib. I've checked the C components in Python/import.c. None of it requires that. Tearing out the chunk of code referenced in the issue causes essentially no change to the import process. It just requires that, if you want the child to exist in the namespace of the parent, you have to import it.

So, all of that aside, I assume this means that pylint is going to have to detect if a from import is occurring inside a package, then assume the existence of the submodule itself. I don't know what else can be done besides perhaps requesting that the official Python documentation be updated to reflect this inconsistency.

[Pierre-Sassoulas]

Thank you for investigating the issue so thoroughly @kaorihinata !