Mypy: [release 0.780] Source file found twice under different module names error.

Created on 3 Jun 2020  ·  22Comments  ·  Source: python/mypy

After yesterday's release 0.780, our CI starting failing with the message Source file found twice under different module names: 'grader.testlib.framework' and 'testlib.framework' but we only have one file named framework.py in grader/testlib/framework.py. This error does not show up in version 0.770.

question
Source
picture ayushr2
👍13

Most helpful comment

ALL: The solution is to use mypy -p <folder> instead of mypy <folder>. This is because PEP 420 support only works on import, not on files/directories passed on the command line. The -p takes a package name which is treated as an import.

picture gvanrossum on 23 Aug 2020
👍2 👀1

All 22 comments

How are you invoking mypy, and what does your directory structure look like? Also, please share your config file/command line flags if possible.

ethanhs picture ethanhs on 3 Jun 2020

Directory Structure:

/grader
              /testlib
                         /components
                         framework.py
              /testers
              .mypy.ini
              ...

Invocation:
mypy grader/ --config-file grader/.mypy.ini

Config .mypy.ini

# Global options:

[mypy]
python_version = 3.6
mypy_path = grader

[mypy-pytest]
ignore_missing_imports = True
ayushr2 picture ayushr2 on 3 Jun 2020

We're running into the same problem. I thought it might have been caused by mypy-protobuf, but it doesn't look like the OP was using that. I posted some details in the above issue, but here are some more:

Error message:

buildgrid/_protos/build/bazel/semver/semver_pb2.pyi: error: Source file found twice under different module names: 'buildgrid._protos.build.bazel.semver.semver_pb2' and 'build.bazel.semver.semver_pb2'
Found 1 error in 1 file (checked 144 source files)

Directory structure (all files outside of buildgrid/_protos omitted)

.
├── buildgrid
│   ├── _app
│   │   ├── bots
│   │   ├── commands
│   │   └── settings
│   ├── _protos
│   │   ├── __init__.py
│   │   ├── build
│   │   │   ├── __init__.py
│   │   │   └── bazel
│   │   │       ├── __init__.py
│   │   │       ├── remote
│   │   │       │   ├── __init__.py
│   │   │       │   └── execution
│   │   │       │       ├── __init__.py
│   │   │       │       └── v2
│   │   │       │           ├── __init__.py
│   │   │       │           ├── remote_execution.proto
│   │   │       │           ├── remote_execution_pb2.py
│   │   │       │           ├── remote_execution_pb2.pyi
│   │   │       │           └── remote_execution_pb2_grpc.py
│   │   │       └── semver
│   │   │           ├── __init__.py
│   │   │           ├── semver.proto
│   │   │           ├── semver_pb2.py
│   │   │           ├── semver_pb2.pyi
│   │   │           └── semver_pb2_grpc.py
│   │   ├── buildgrid
│   │   │   ├── __init__.py
│   │   │   └── v2
│   │   │       ├── __init__.py
│   │   │       ├── monitoring.proto
│   │   │       ├── monitoring_pb2.py
│   │   │       ├── monitoring_pb2.pyi
│   │   │       └── monitoring_pb2_grpc.py
│   │   ├── buildstream
│   │   │   ├── __init__.py
│   │   │   └── v2
│   │   │       ├── __init__.py
│   │   │       ├── buildstream.proto
│   │   │       ├── buildstream_pb2.py
│   │   │       ├── buildstream_pb2.pyi
│   │   │       └── buildstream_pb2_grpc.py
│   │   └── google
│   │       ├── __init__.py
│   │       ├── api
│   │       │   ├── __init__.py
│   │       │   ├── annotations.proto
│   │       │   ├── annotations_pb2.py
│   │       │   ├── annotations_pb2.pyi
│   │       │   ├── annotations_pb2_grpc.py
│   │       │   ├── http.proto
│   │       │   ├── http_pb2.py
│   │       │   ├── http_pb2.pyi
│   │       │   └── http_pb2_grpc.py
│   │       ├── bytestream
│   │       │   ├── __init__.py
│   │       │   ├── bytestream.proto
│   │       │   ├── bytestream_pb2.py
│   │       │   ├── bytestream_pb2.pyi
│   │       │   └── bytestream_pb2_grpc.py
│   │       ├── devtools
│   │       │   ├── __init__.py
│   │       │   └── remoteworkers
│   │       │       ├── __init__.py
│   │       │       └── v1test2
│   │       │           ├── __init__.py
│   │       │           ├── artman_remoteworkers.yaml
│   │       │           ├── bots.proto
│   │       │           ├── bots_pb2.py
│   │       │           ├── bots_pb2.pyi
│   │       │           ├── bots_pb2_grpc.py
│   │       │           ├── command.proto
│   │       │           ├── command_pb2.py
│   │       │           ├── command_pb2.pyi
│   │       │           ├── command_pb2_grpc.py
│   │       │           ├── remote_workers.yaml
│   │       │           ├── remoteworkers_gapic.yaml
│   │       │           ├── tasks.proto
│   │       │           ├── tasks_pb2.py
│   │       │           ├── tasks_pb2.pyi
│   │       │           ├── tasks_pb2_grpc.py
│   │       │           ├── worker.proto
│   │       │           ├── worker_pb2.py
│   │       │           ├── worker_pb2.pyi
│   │       │           └── worker_pb2_grpc.py
│   │       ├── longrunning
│   │       │   ├── __init__.py
│   │       │   ├── operations.proto
│   │       │   ├── operations_pb2.py
│   │       │   ├── operations_pb2.pyi
│   │       │   └── operations_pb2_grpc.py
│   │       └── rpc
│   │           ├── __init__.py
│   │           ├── code.proto
│   │           ├── code_pb2.py
│   │           ├── code_pb2.pyi
│   │           ├── code_pb2_grpc.py
│   │           ├── status.proto
│   │           ├── status_pb2.py
│   │           ├── status_pb2.pyi
│   │           └── status_pb2_grpc.py
│   ├── bot
│   │   └── hardware
│   ├── cleanup
│   ├── client
│   └── server
│       ├── actioncache
│       ├── bots
│       ├── capabilities
│       ├── cas
│       │   └── storage
│       │       └── index
│       │           └── sql_dialect_delegates
│       ├── execution
│       ├── operations
│       ├── persistence
│       │   ├── mem
│       │   └── sql
│       │       └── alembic
│       │           └── versions
│       └── referencestorage
├── data
│   ├── bin
│   └── config
│       └── grafana
│           ├── dashboards
│           │   └── buildgrid
│           └── datasources
├── dockerfile-scripts
├── docs
│   └── source
│       ├── data
│       ├── developer
│       └── user
└── tests
    ├── auth
    │   └── data
    ├── cas
    │   ├── data
    │   │   └── hello
    │   │       ├── docs
    │   │       │   └── reference
    │   │       └── utils
    │   └── index
    ├── integration
    ├── persistence
    ├── stress-testing-dockerfiles
    │   └── mnt
    │       ├── configs
    │       └── scripts
    └── utils

mypy.ini

[mypy]
plugins = sqlmypy
mypy_path = buildgrid/_protos

[mypy-botocore.*]
ignore_missing_imports = True

[mypy-alembic.*]
ignore_missing_imports = True

[mypy-grpc_reflection.*]
ignore_missing_imports = True

[mypy-janus.*]
ignore_missing_imports = True

mypy invocation (run from the root of the repository):

mypy buildgrid/
roitk picture roitk on 3 Jun 2020

@ayushr2 @roitk it seems that mypy is now flagging those files as duplicates because they are both searched by being in the MYPYPATH and the sources list. This seems correct to me. This may have to do with a change in how mypy searches namespace packages but I am not sure.

ethanhs picture ethanhs on 3 Jun 2020

I ran into this too, even though I'm not setting $MYPYPATH. It's dependent on exactly how I invoke mypy:

jelle@devshared28:~/ans/yo$ /home/jelle/ans/yo/.tox/py36/bin/python3.6 -m mypy --config-file /home/jelle/ans/yo/src/mypy.ini /home/jelle/ans/yo/src
src/__init__.py: error: Source file found twice under different module names: 'yo.src' and 'src'
Found 1 error in 1 file (checked 37 source files)
jelle@devshared28:~/ans/yo$ /home/jelle/ans/yo/.tox/py36/bin/python3.6 -m mypy --config-file /home/jelle/ans/yo/src/mypy.ini src
Success: no issues found in 37 source files

The config file doesn't have options that should be relevant, and the same error appears if I don't pass any config file.

JelleZijlstra picture JelleZijlstra on 4 Jun 2020

Hm, then perhaps this is a regression after all? I think this definitely merits further investigation.

ethanhs picture ethanhs on 4 Jun 2020

MYPYPATH (or mypy_path) should not point to a directory with an __init__.py file, as this can result in a package with an ambiguous name. Previously mypy sometimes accepted this. This could cause crashes so mypy now complains about it.

@ayushr2 If /grader/__init__.py exists, you've hit the above issue. You should perhaps point mypy_path to the parent directory of grader.

@roitk You have mypy_path that points to a directory with a __init__.py file, so mypy gets confused.

@JelleZijlstra Your issue might be different. Can you give a full example, including all relevant source files?

JukkaL picture JukkaL on 8 Jun 2020

I suspect Jelle's issue is also a spurious __init__.py and that it manifests differently with relative paths because of how we crawl up to find the base directory: https://github.com/python/mypy/blob/3adb2e952cf9e16d4d493440a1c7be3de4d214ef/mypy/find_sources.py#L101

hauntsaninja picture hauntsaninja on 10 Jun 2020

@hauntsaninja is correct. The directory structure in my example is as follows:

├── __init__.py
├── src
    ├── __init__.py
    ├── mypy.ini
    └── yo_bot.py

(plus numerous other files not relevant here). If I remove the outer __init__.py, the error goes away.

JelleZijlstra picture JelleZijlstra on 11 Jun 2020

I'm hitting this. It happens because I have to specify all the subfolders explicitly in order to get mypy to process them, since it doesn't process folders recursively (contrary to what its documentation says). e.g.

$ echo $MYPYPATH

$ mypy src test test/jobs
test/jobs/job_utils.py: error: Source file found twice under different module names: 'job_utils' and 'jobs.job_utils'  [misc]

If I turn off namespace_packages it seems to work, but it's unclear to me what the ramifications of that will be (the documentation is unclear to its practical purpose, but seems to imply this will cause some modules I import to not be recognised, and thus not have their use by my code typechecked?).

I've noticed for some time that it was double-reporting everything, implying it's been processing things twice, but it didn't originally do that - I'm not certain, but I think this was a mypy change, not something I did. Alas I don't know which version of mypy introduced that duplicate processing.

wadetregaskis-linkedin picture wadetregaskis-linkedin on 17 Jun 2020

mypy should recursively process your directories, and in fact it definitely is doing that to some extent, since it's clearly finding job_utils.py twice. Could you confirm whether mypy src test is checking job_utils.py (perhaps by introducing a type error)? Is there a tests/__init__.py?

hauntsaninja picture hauntsaninja on 17 Jun 2020

I found I had this problem when upgrading from 0.770 to 0.780 in multiple projects. When I removed MYPY_PATH from the environment variable and defining mypy_path in mypy.ini as a gobal fixed the issue.

httpstergeek picture httpstergeek on 17 Jun 2020

@wadetregaskis-linkedin You might be hitting #5759. Namespace package support is currently incomplete but a major update to namespace packages is being planned (#8584).

@httpstergeek It would be helpful if you can provide a simplified but complete example which reproduces your issue. It seems that things are working unexpectedly in some specific scenarios that we don't yet fully understand.

JukkaL picture JukkaL on 19 Jun 2020

@hauntsaninja, no, it definitely doesn't process directories recursively _unless_ they're modules (i.e. it gets 'src' and all its subfolders, because 'src' has a 'foo' subfolder with an __init__.py in it, but the associated 'test' folder contains no __init__.pys, at any level). At one point I believe it did - I used to just provide 'src' and 'test' arguments - but that was a long time ago. I don't recall all the details now, but there's been quite a struggle over various different versions of mypy to get it to just process these two folders recursively.

The duplicates only arise when I specify all the subfolders explicitly, and only for the subfolders I specify (since otherwise mypy ignores them).

IIRC it _also_ reports things twice from not just the 'test' folder but also 'src', implying that somehow merely mentioning both, and having the files in 'test' import those from 'src', somehow causes it to do a redundant pass.

wadetregaskis-linkedin picture wadetregaskis-linkedin on 19 Jun 2020

@JukkaL I also hit this issue after upgrade. I think it is the same issue reported here. Here is a minimal recreate (using CentOS 7):

mkdir -p lib/dir1/dir2/dir3/dir4
touch lib/dir1/dir2/dir3/file1.py
echo "import dir1.dir2.dir3.file1 as file1" > lib/dir1/dir2/dir3/dir4/file2.py

With mypy 0.781:
MYPYPATH=lib mypy --namespace-packages --strict-optional --strict lib/dir1/dir2/dir3/*
lib/dir1/dir2/dir3/file1.py: error: Source file found twice under different module names: 'file1' and 'dir1.dir2.dir3.file1'
Found 1 error in 1 file (checked 2 source files)

With mypy 0.770:
MYPYPATH=lib mypy --namespace-packages --strict-optional --strict lib/dir1/dir2/dir3/*
Success: no issues found in 2 source files

jvm3487 picture jvm3487 on 22 Jun 2020

I have the same issue, but just a single __init__.py in our main directory. If I remove the __init__.py it works fine.

richtong picture richtong on 3 Jul 2020

The same happens for https://github.com/typeddjango/djangorestframework-stubs

Output:

» mypy -p rest_framework-stubs
rest_framework-stubs/request.pyi: error: Source file found twice under different module names: 'rest_framework-stubs.request' and 'rest_framework.request'  [misc]
Found 1 error in 1 file (checked 48 source files)

When I delete __init__.pyi - it works.
It also work when I downgrade to pip install 'mypy==0.770' (my current solution)
Related https://github.com/typeddjango/djangorestframework-stubs/pull/75

sobolevn picture sobolevn on 3 Jul 2020

I experienced the opposite – adding an __init__.py solved the issue. With every folder being a package, this worked fine:

mypy app/mailers app/models app/services utils

We are gradually adding typechecks to the project, and the next step was to add the app/rest folder. It didn't contain an __init__.py, which caused this:

app/rest/errors.py: error: Source file found twice under different module names: 'errors' and 'app.rest.errors'

At first I tried moving errors.py away, but then mypy complained about other files under app/rest. So, adding app/rest/__init__.py helped.

Edit: this must be #5759, as we use mypy with namespace_packages = True.

killthekitten picture killthekitten on 1 Aug 2020

I am hitting this too today (mypy==0.782) after converting a library to use PEP 420 native namespacing. Here's the error message I'm seeing:

namespace/package/foo/bar.py: error: Source file found twice under different module names: 'package.foo.bar' and 'namespace.package.foo.bar'  [misc]

As per @richtong 's suggestion above, removing the __init__.py in our subpackage_a directory solved the problem.

Unfortunately, that __init__.py contains important information, so deleting it isn't a viable solution in my use case.

Aside: what's interesting is this error is only triggered for one source file, even though this error could exist in many places within the library.

I am coming here for one question and one suggestion:

  • Question: As a temporary workaround, how can one disable this warning on a per-module basis?

    • Placing # type: ignore or # type: ignore[misc] at the top of the namespace/package/foo/bar.py module is not desirable for me and doesn't actually suppress this message

    • I am unsure of how to add ignores for this to the config file, the below isn't working for me with setup.cfg:

[mypy-namespace.package.foo.bar]
ignore_errors = True
  • Suggestion: Provide a more verbose message for this error.

    • Ex: over two more lines, enumerate where exactly the duplicate source files were found, with line numbers

jamesbraza picture jamesbraza on 22 Aug 2020

@jamesbraza could you also share the command you use to run mypy?

killthekitten picture killthekitten on 22 Aug 2020

@killthekitten yeah I use the files section of a config file, and just invoke mypy via entering solely mypy. Currently, I have three args to files:

  • .: to type check setup.py in my repo's root
  • namespace: type check the actual package
  • tests: type check unit tests

I noticed that if I get rid of the ., that this error actually goes away. So a new workaround for me is to no longer use files, and in my CI config just invoke mypy two times:

  • mypy namespace tests
  • mypy setup.py

In my opinion, the more desirable workaround would be to make this error message suppressible.

Update: I am no longer hitting this error, as it turns out (found out through https://github.com/python/mypy/issues/9341) I was improperly invoking mypy. I needed to use -p and -m as opposed to files in the config file.

Perhaps that might be the root of your problem @ayushr2 ?

jamesbraza picture jamesbraza on 23 Aug 2020

ALL: The solution is to use mypy -p <folder> instead of mypy <folder>. This is because PEP 420 support only works on import, not on files/directories passed on the command line. The -p takes a package name which is treated as an import.

gvanrossum picture gvanrossum on 23 Aug 2020
👍2 👀1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Stiivi picture Stiivi  ·  3Comments
jstasiak picture jstasiak  ·  3Comments
mplanchard picture mplanchard  ·  3Comments
takeda picture takeda  ·  3Comments
Stiivi picture Stiivi  ·  3Comments