Mypy: Warn about missing __init__.py

[JukkaL]

If a package doesn't have an __init__.py file, the error message from mypy isn't very helpful (e.g., Can't find package 'abc'), and sometimes users can't figure this out by themselves. If mypy can't find a package, it could see if the directory exists somewhere in the module search path without an __init__.py, and give a message that says this explicitly.

Examples of potential better messages:

$ mkdir foo
$ mypy -p foo
Can't find valid package 'foo' (foo/ has no __init__.py)
$ mypy -c "import foo"
<string>:1: error: Cannot find module named 'foo' (foo/ has no __init__.py)

[gvanrossum]

Not also that in Python 3, a directory without __init__.py can be a package (there's a PEP for that). But it's intended only for a certain type of "namespace" packages. There once was an issue requesting support for this, but I rejected it because it would break the "crawl up until root of package" algorithm that mypy uses when passed a file inside a package. Maybe though there should be a flag to change that behavior? (My current workaround is "add an __init__.pyi.)

[davidthornton]

I'm not seeing this message come through on release 0.521... I got stung by this, and wanted to prevent anyone else from having to go through this by reminding y'all 😀

[carljm]

At Instagram this behavior has been a common cause of new packages silently not being typechecked, and nobody realizing it until much later. Somebody will add a new package without an __init__.py, and it will work fine (because we are on Python 3, where __init__.py is not actually required), and they won't realize that their new code is not typechecked because mypy can't find it. We run with ignore-missing-imports, for other reasons we can't change at the moment, so it's totally silent.

It would be great to have an option to explicitly specify the package root and not require __init__.py files.

[gvanrossum]

I agree it would be nice to have a way to override this. The "crawl-up" behavior is only needed for files passed on the command line. @carljm -- do you pass files inside __init__-less packages on the command line? If not, maybe in PY3 mode we could just change the behavior when you set $MYPYPATH and implement a module search (when following imports) that's more similar to what Python 3 itself does.

BTW what exactly do you mean by "package root"? For a module m.py inside a package p, would that be the directory p or its parent?

[carljm]

BTW what exactly do you mean by "package root"? For a module m.py inside a package p, would that be the directory p or its parent?

I meant the parent; the directory that would be on sys.path at runtime. Probably "package root" was a bad choice of term; maybe "import root" is better. Finding this directory is the reason mypy crawls up looking for __init__.py files, right?

do you pass files inside __init__-less packages on the command line?

Not as part of any automated flow, but it's likely that a developer may do this locally when debugging some type errors; it would be confusing if it didn't work correctly. For our automated full typecheck we just run mypy . from our import root. (To be clear, we have a single "import root" directory with multiple top-level packages inside it. We don't currently set MYPYPATH or expect mypy to find any code outside that directory.)

maybe in PY3 mode we could just change the behavior when you set $MYPYPATH and implement a module search (when following imports) that's more similar to what Python 3 itself does.

Not sure I fully understand what you're suggesting here, but it seems like it might be possible to leverage MYPYPATH in a way that would work even if mypy is passed a file within a package; the logic would be something like "if there is a directory on MYPYPATH that is an ancestor of the file we were given, assume the directory on MYPYPATH is the "import root" and don't crawl up directories looking for __init__.py." Then we would just add our import root to MYPYPATH. This isn't 100% ideal, since it would require setting MYPYPATH where currently we don't need to, but that tradeoff would be worth it to solve this problem.

Another option that comes to mind is to assume (maybe opt-in based on a config flag) that the directory containing mypy.ini is the import root. This would work great for our case, but I'm sure there are projects it wouldn't work for.

[JukkaL]

What about something like this in the config file:

[mypy]
import_roots = .

[gvanrossum]

I don't think "root" is a common part of Python's terminology about imports (though it seems to be when talking about distributions). We usually talk about "import path", and it's always thought of as a list of directories; for individual directories we say "path entry".

I like to just add a flag (command line and in mypy.ini) that changes the behavior so that it doesn't try to crawl up the filename to find additional path entries, but instead searches the search path (current directory plus MYPYPATH) for the file and complains loudly if a file isn't on those. And perhaps the flag would also make it so that if you pass any directories on the command line if will treat those directories as path entries.

[carljm]

I like to just add a flag (command line and in mypy.ini) that changes the behavior so that it doesn't try to crawl up the filename to find additional path entries, but instead searches the search path (current directory plus MYPYPATH) for the file and complains loudly if a file isn't on those.

Presuming "searches the search path" implements Py3-compatible behavior (willingness to import packages without __init__.py) in PY3 mode, that sounds fine.

And perhaps the flag would also make it so that if you pass any directories on the command line if will treat those directories as path entries.

Not sure that makes sense. E.g. in our case, say the correct import path entry is /opt/instagram-server. If that's my cwd and I run mypy some/package/, that doesn't mean that I want to add /opt/instagram-server/some/package/ to the import path (doing so would be wrong), it just means I want to typecheck the files inside that particular package.

[gvanrossum]

OK, sounds good. Let's name the flag --namespace-packages (in mypy.ini, namespace_packages = True). I'm pretty busy with other stuff but I'd be willing to look at a PR.

[carljm]

Given the conclusion here, it seems this is a dupe of #1645.

[ethanhs]

This is a duplicate of #1645.