Mypy: MYPYPATH doesn't respect PEP420 namespace packages

[mr-rodgers]

Take this directory tree:

/path/to/src
└───namespace
    └───package
        │   __init__.py
        │
        └───__pycache__
                __init__.cpython-35.pyc

And this MYPYPATH:

> $env:MYPYPATH
/path/to/src

then running mypy -p namespace.package gives:

> mypy -p namespace.package
Can't find package 'namespace.package'

However, mypy works fine when you add an empty __init__.py into namepace (turning it into a regular package).

[gvanrossum]

I know... I'm not sure what to do about this. The algorithm from PEP 420 is quite complex (namespace packages are only considered if no "classic" package or module is found). The workaround of adding a dummy __init__.py is simple enough (you could also have a dummy __init__.pyi so that Python is not affected by this file).

I also expect that usually you'd want to create a separate set of stubs rather than type-check the original source code (though obviously this may vary per package).

[iScrE4m]

I'd like to bump this. In future, there will be python programmers who never heard of pre PEP420 era of __init__.py and that's good. What's not good is mypy forcing them to create dummy files.

[agronholm]

It might get the ball rolling if somebody explained why this is such a big problem for mypy.

[agronholm]

I just bumped into Jukka Lehtosalo and he said there shouldn't be any major issue blocking this. But given the easy workaround that exists for this, it'll likely require an outsider to write a PR.

[gvanrossum]

What's not good is mypy forcing them to create dummy files.

This issue is more complex than you think.

[iScrE4m]

I don't think it's trivial, I've read your previous comment - so I don't even think I'd be able to write a PR. I just think it's an issue worth looking into eventually

[richardmss]

I'm working on a project with a directory structure like the following:

./a
./a/c
./a/c/file1.py
./b
./b/c
./b/c/file2.py

Note that both, a and b, contain a subfolder called c. Adding an __init__ everywhere defines some modules multiple times (the actual project contains dozens folders) and things get hairy. Any progress on this issue would be much appreciated.

[gvanrossum]

Adding an __init__ everywhere defines some modules multiple times [...]

I presume you mean in each c folder but not in a or b right? So c is a namespace package and you have both a and b in PYTHONPATH or sys.path. (And when running mypy, you pass it both a and b, which will have the same effect -- or perhaps you pass it their parent directory or the current directory.)

With that setup, I agree you have a predicament. We will have to support namespace packages in mypy. At the very least it should be possible to do something like setting MYPYPATH=a:b and then run mypy -m c and have it find both parts of c.

[carljm]

Just a philosophical note here that might influence the way mypy handles this: in my experience in a Py3-only world, packages without __init__.py are not a special edge case for namespace packages, they are the default case. It's pretty unreasonable to expect a new Py3 developer to remember to drop empty __init__.py turds everywhere when their code demonstrably runs perfectly fine without them. It's hard to even make a sensible case in code review for why they should add __init__.py (other than "it's the way we've always done things in the old world, get off my lawn" or "the typechecker is buggy and won't find your code without it.") The presence of __init__.py becomes the special case, used only when you need some package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but AFAICS it's an inevitable one.

[gvanrossum]

Or perhaps this reflects your particular culture.

On Nov 25, 2017 21:48, "Carl Meyer" [e-mail] wrote:

Just a philosophical note here that might influence the way mypy handles
this: in my experience in a Py3-only world, packages without __init__.py
are not a special edge case for namespace packages, they are the default
case. It's pretty unreasonable to expect a new Py3 developer to remember to
drop empty __init__.py turds everywhere when their code demonstrably runs
perfectly fine without them. It's hard to even make a sensible case in code
review for why they should add __init__.py (other than "it's the way
we've always done things in the old world, get off my lawn" or "the
typechecker is buggy and won't find your code without it.") The presence of
__init__.py becomes the special case, used only when you need some
package-level code or imports.

This reversal may not have been an intended consequence of PEP 420, but
AFAICS it's an inevitable one.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/python/mypy/issues/1645#issuecomment-346986174, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMunwfsZJ03cqKrWM6ONIipLvzuh_ks5s6Pu5gaJpZM4IuKil
.

[carljm]

Or perhaps this reflects your particular culture.

Certainly possible. I'd be curious to hear from another Py3-only culture
which (convincing to new Python 3 developers) arguments are used in
favor of continuing the tradition of empty __init__.py files. I'm not
aware of a strong practical argument that can be made for them, but if
I'm missing a good one I may start using it ;)

[ethanhs]

This seems to be hitting quite a few people, so raising priority to high.

[JukkaL]

This seems to be one the most important Python features that mypy doesn't properly support. Some potential users have told me that they were unable to use mypy because of this.

[jgossage]

Unfortunately the work-around of adding dummy__init__.py files does not work when you are using tools that expect namespace packages to not have__init__.py files. This limitation is preventing me from getting full value from mypy.

[ilevkivskyi]

@jfgossage As a dirty trick you can try adding dummy __init__.pyi files (note i at the end), this might work.

[jgossage]

I am working with a structure like this:

   src
      wiseoldbird  (Python package)
          utilities (Python package)
              module1.py
              module2.py
  tests
        wiseoldbird (Python package)
             utilities (Python package)
                   test_module1.py
                   test_module2.py

Unfortunately if I try to use __init__.pyifiles in all the package directories I get the following mypy error:

tests/wiseoldbird/__init__.pyi: error: Duplicate module named 'wiseoldbird'

[warsaw]

As shown with #5575 I hit the same problem. I think mypy needs to somehow handle namespace packages and be prepared for an all Python 3 world.

[warsaw]

I want to point out why adding an empty __init__.pyi is not a viable solution. Remember that the original motivation for PEP 420 was to support downstream package distributors such as Debian. In these cases, files must be owned by exactly one platform package. Thus if foo is a namespace package, then something must own foo/__init__.py but since the foo directory isn't directly owned by any specific platform package, neither can foo/__init__.py. E.g. if foo/bar and foo/qux are subpackages, then which one owns foo/__init__.py?

Requiring foo/__init__.pyi just punts the problem down the field because it has exactly the same ownership problems as foo/__init__.py.

[gvanrossum]

Yeah, that's why we eventually implemented mypy --namespace-packages.

[nicktimko]

And for those that stumble on this issue via Google, that flag was added in 0.640, so make sure you're on that or later, then use the aforementioned --namespace-packages argument (permalink)

[warsaw]

FYI. For completeness, I'll mention that I recently realized that my patch only works for the top level namespace package. It doesn't help traversing in to nested sub-namespace packages. That doesn't affect us in practice so I'm not motivated to fix it.

[nickroeker]

It is the year 2020. Python 3 is the only supported major release of Python. Is using the non-default --namespace-packages flag still the recommended solution?

As a creator of various namespaced packages targeted for Python 3, this requires users of my packages to set this flag to get any type information out of my packages (requiring them to somehow know that my package is a namespace package, carrying all this baggage). Also as a developer of these packages, getting the __init__.pyi alternative to work in bdist_wheel seems downright impossible with Python 3, so I'm having a hard time building something that works on the major supported platform using a major supported feature in what is a pure python package.

I would open a PR to change the default but given the discussion here, the decision appears a lot more nuanced and personal than I currently understand, and I do not wish to upset the maintainers.

[ErwanDL]

I hope that I am not unaware of some context around this issue, but I agree with @nickroeker that it would make sense for the --namespace-packages flag to be the default behaviour for mypy, especially as the current behaviour is really confusing for Py3-only users like myself.

I very recently stumbled into an issue (https://github.com/python/mypy/issues/8497) that I believe was due to this (adding the --namespace-packages flag solved it), and being a "relatively new" Python user that has never used Python 2, I had a very confusing time understanding why I was expected to add an empty __init__.py in a namespace package for mypy to work.

Is the main obstacle to making this behaviour default the need for retrocompatibility with Python 2 ?

[jgossage]

I think that the question of the --namespace-packages flag needs to be
raised with the Python people. I am just a user like you.

On Fri, Mar 13, 2020 at 11:32 PM Erwan de Lépinau [e-mail]
wrote:

I hope that I am not unaware of some context around this issue, but I
agree with @nickroeker https://github.com/nickroeker that it would make
sense for the --namespace-packages flag to be the default behaviour for
mypy, especially as the current behaviour is really confusing for Py3-only
users like myself.

I very recently stumbled into an issue (#1645
https://github.com/python/mypy/issues/1645) due to this (adding the
--namespace-packages flag solved it), and being a "relatively new" Python
user that has never used Python 2, I had a very confusing time
understanding why I was expected to add an empty __init__.py in a namespace
package for mypy to work.

Is the main obstacle to making this behaviour default the need for
retrocompatibility with Python 2 ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/python/mypy/issues/1645#issuecomment-599004836, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACZGNN6Y63SWVGXVD5K6YTRHL3F3ANCNFSM4CFYVCSQ
.

--
Jonathan Gossage

[JukkaL]

Enabling support for namespace packages by default in Python 3 mode sounds like a good thing. Would somebody create a new issue about it? I haven't looked into this in detail, but we might need to fix some issues with namespace package support first.

[ErwanDL]

I can create an issue this afternoon 👍

[russelldavis]

Here's the issue that proposes supporting this by default: https://github.com/python/mypy/issues/8584

(Thanks @JukkaL -- I am very much in favor of the proposal.)

[aloosley]

I'm confused because I still get the same error when running mypy --namespace-packages . at the root of my repo that doesn't have a py file (mypy 0.782, python 3.7.7):

$ mypy --namespace-packages .
There are no .py[i] files in directory '.'

So what exactly is the work-around if --namespace-packages doesn't work?

I have several python packages I want to check under the root folder as well various folders and subfolders with python files.

I'm probably missing something simple and dumb. Thanks for the help.

[nickroeker]

@aloosley It's been a while so I'm not honestly sure anymore, but I _think_ for --namespace-packages to work you may still need a py.typed file in the right place. I _think_ the right place for that is in the non-namespace folders. For example, if you have package company.service, the service/ folder would need a py.typed file.

Aside from that, I find I generally need to run mypy from the outside. So instead of being in company/ and running mypy ./, go up a directory and run mypy company/.