Pytest: get rid of "ImportMismatchError"

[hpk42]

Over on the tox issue tracker, there was a discussion about the irritating "ImportMismatchError" that pytest throws if one collects from the source tree but the imports actually come from some installed location (e.g. some virtualenv). This happens more often with tox i guess because the default way of testing with tox is creating and installing a package and then collecting from the source tree.

@jaraco and me agreed that it's probably best to try improve pytest so that it, say with pytest-4.0, will automatically redirect from source to install location. Initially and up until the next major release we could introduce a --collect-redirect-to-installed=yes option or so which works like this: if during collection we find a python package (a directory with an __init__.py for now) we check if importing it will actually get it from another location. If so, we redirect to the install location and also report this fact. Only problem is that with python3 one can have python packages without __init__.py files so the actual logic needs to take that into account as well (how many people are really using this, btw?).

[RonnyPfannschmidt]

@hpk42 proposal: ignoring the error altogether if the contents of the wanted and the imported file are the same (and adding a mention of install/reinstall/develop maybe)

[hpk42]

ignoring the error seems dangerous to me as the module under question might import or depend on other modules which are not the same at source versus install location. So i can change files at the source location, run tests and they all appear to be running fine while in fact they are just running against the last installed version.

[RonnyPfannschmidt]

thats why i made content equality a condition!

[The-Compiler]

But with what @hpk42 brought up, the condition would need to be module equality, plus equality for all modules that module imports - right?

[hpk42]

sure, i was referring to that case where a module source under test is equal to the imported version of it. It might still import something that isn't. Imagine e.g. a typical inline test layout with __init__.py files. The test file mypkg/tests/test_x.py might be the same at the source and at the actually installed location but the modules it itself imports would come from install whereas you assume they come from source. This is precisely what ImportMismatchError tries to detect.

[RonnyPfannschmidt]

well, from my pov we can either stop supporting broken folder layouts or we have to introduce broken hacks ^^ my proposal was such a hack

i mean this is one of the cases where we literally loose because there is no clean and backward compatible way to support this, so we pick the cost and pay it ^^

[hpk42]

I hold that the automatic redirection from source to install location is worth trying and could resolve this problem by default with pytest-4.0 (if not earlier -- nobody can really have relied on an ImportMismatchError appearing).

[jaraco]

Only problem is that with python3 one can have python packages without init.py files so the actual logic needs to take that into account as well (how many people are really using this, btw?)

In pypa/setuptools#250, I'm working to resolve issues with namespace packages, and the leading solution will require _all_ namespace package distributions to follow this model, even at the development source. And currently, any namespace packages installed by pip (or with setuptools --single-version-externally-managed install) will also have this characteristic (even on Python 2). That is all packages jaraco.* and zope.* and similar will not have __init__.py in the namespace package as installed by pip (not -e) and rely on the -nspkg.pth or PEP 420 (Python 3.3+) to create the package module.

[hpk42]

Is there a way to find out if a directory is the root of a package if there is no __init__.py?

[jaraco]

I thought you might ask that, and the answer is essentially no. Or rather, on Python 3.3+, any directory on sys.path is always the root of a package:

$ mkdir foo
$ python -c "import foo; print(foo.__path__)"
_NamespacePath(['/Users/jaraco/foo'])

Probably the packaging infrastructure should expose the "namespace packages" declared for a given distribution, such that you could enumerate all installed packages and determine the namespace packages they declare (and from there resolve the paths pertinent to those packages), but in my initial investigation, I don't see that metadata exposed for installed packages.

[jaraco]

@hpk42: Is it viable to actually perform the import and then inspect <module>.__path__? Or do you need to know if a directory is the root of a package by inspecting the file system only?

[RonnyPfannschmidt]

we need to figure the module name from the filesystem so its a bit tricky to guess the root

[hpk42]

@jaraco i guess it's possible. Directories are not themselves represented in the collection tree. So as soon as we collect a python module and would throw an ImportMismatchError we can instead deliver a message "redirecting collection of source location X to installed location Y" and accept the import instead of throwing. We probably want to only report this once for the root, though, not for each module.

[RonnyPfannschmidt]

for correct redirection we should compare a complete module tree before accepting it (imagine a installed package missing a module

[hpk42]

maybe this works: when importing a collected source python module which turns out to come from some other install-location we memorize a source-rootdir -> install-rootdir mapping and issue a report about the redirection. When we subsequently try to import another module based on a mapped root, we don't report it again but verify that the import will come from the install location and if not throw an ImportMismatchError (this should be very rare). We might also check at first redirection time that no module was imported already and was not mapped. Altogether requires a bit of state keeping but should be doable.

[nicoddemus]

I'm still mid-way digesting the discussion, so sorry if I'm asking something which is obvious to everyone involved.

I'm used to run tests in 3 scenarios, which is:

• tox
• setup.py develop
• configuring PYTHONPATH for conda (which is very similar to setup.py develop).

Of those, only when using tox an ImportMismatchError can happen and we know it is not actually an error because tox guarantees for the files to be the same since it was tox itself who copied them from source to destination.

In which situations an ImportMismatch error happens outside using tox? Some other tool or some workflow where that error is common and is actually an error, the fact that a module being imported was already imported under another name?

[hpk42]

regarding tox installing packages, it's possible that due to MANIFEST or other issues not all files are installed.

Import-Mismatch can happen out side tox e.g.: if you "pip install ." and then run tests with pytest in the case where you have tests on source files (app or tests) which were installed and which python will import from there. The same issue applied for PYTHONPATH.

[RonnyPfannschmidt]

@nicoddemus when people do pip install instead of pip install -e that can happen as well

[nicoddemus]

@hpk42 and @RonnyPfannschmidt thanks!

How about we add some way to force pytest to not raise the ImportMismatchError? This makes it possible to bypass this check on situations where we know for sure the error is a false positive, for example when running under tox. I'm thinking of something non-intrusive and simple to implement like PYTEST_IGNORE_IMPORT_MISMATCH=1. With it in pytest, tox could always set that environment variable behind the scenes automatically even if the user is not using pytest, which would solve the problem for all tox users.

While of course that's not the most elegant solution and does not address the underlying issue, it has some advantages to it:

1. Very simple to implement so we can get this out very soon: tox sets an environment variable, pytest reads it and does not raise the error;
2. Basically no maintenance burden;
3. Backward compatible, for tox and pytest;
4. Can be kept around even if we implement a proper solution later, because as this discussion shows the solution probably isn't easy and it might contain bugs itself.

This is not meant to solve the other user cases you described, but at least would solve the issue for a lot of tox users now, while we can discuss and refine a more long term solution. We might even decide to not document it at all, keep it as an "internal workaround" to at least alleviate tox users for now.

Just throwing this idea here to see what you guys think, don't mean to disrupt/stop the current discussion on how to treat the problem properly.

[RonnyPfannschmidt]

a accessible workaround for tox/env var use seems really sensible as a first step to enable users to test at all and leaves us time for working on a more detailed "propper" fix

im in favour, good idea :)

[nicoddemus]

@hpk42 and @jaraco, what are your thoughts on the PYTEST_IGNORE_IMPORT_MISMATCH workaround?

[wheerd]

I am also having this issue when using the Windows Subsystem for Linux and trying to run my tests there. Apparently the linux/windows paths get mixed up.

I am not sure if this is a separate issue as I am somewhat surprised how the windows path even gets in there in the first place. Here is the traceback (with shortened paths):

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/_pytest/config.py", line 325, in _getconftestmodules
    return self._path2confmods[path]
KeyError: local('/mnt/c/.../tests')

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/_pytest/config.py", line 356, in _importconftest
    return self._conftestpath2mod[conftestpath]
KeyError: local('/mnt/c/.../tests/conftest.py')

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/_pytest/config.py", line 362, in _importconftest
    mod = conftestpath.pyimport()
  File "/usr/local/lib/python3.5/dist-packages/py/_path/local.py", line 668, in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
py._path.local.LocalPath.ImportMismatchError: ('tests.conftest', 'C:\\...\\tests\\conftest.py', local('/mnt/c/.../tests/conftest.py'))
ERROR: could not load /mnt/c/.../tests/conftest.py

[jaraco]

what are your thoughts on the PYTEST_IGNORE_IMPORT_MISMATCH workaround?

That sounds reasonable to me. In my experience, it's only the tox-created environments that are experiencing the false positives.

[blueyed]

Another case where this happens when running tests in a Docker container, where the source gets bind-mounted as an image (into something like /srv, /app or /src typically), and you had run the tests on the host already.
In this case no automatic redirection would be possible.

[obestwalter]

I encounter that error when running tests in a vagrant project. Vagrant maps your project read/writable into the box and pytest gets confused about that. Removing *.pyc in between runs when switching between guest and host resolves that.

[jmagnusson]

I can confirm that obestwalter's tip works well for Docker also.

[jmf-mordis]

I also find this error running pytest inside a docker container. The obestwalter's tip works well for me, but i would like to add that in python3 you need to remove the __pycache__ folders.

[obestwalter]

in python3 you need to remove the __pycache__ folders.

They also just contain .pyc files, so getting rid of the folders is not necessary - just the contents.

e.g. rm -r **/*.pyc

[jayofdoom]

Is this issue still being addressed? The workaround that @obestwalter presented doesn't work for me, attempting to run py.test tests under tox.

[obestwalter]

@jayofdoom Can you tell us, which os, which versions and post the output that you get?

[jayofdoom]

This was on Ubuntu Trusty, I can't paste the output as it was in a closed source repo.

I have the problem resolved now, but in my case, I worked around by setting (in tox.ini)

[pytest]
testpaths = src

Once that was set, pytest stopped seeing the .tox/ dir, and installed package there, causing the error.

[letmaik]

I have a similar problem. I use tox to install a wheel and then run pytest, which works, but it fails with ImportMismatchError as soon as I add the --doctest-modules option. This issue seems to be the same as #539 and none of the workarounds here and there worked for me.

[obestwalter]

@letmaik I would be very surprised if a complete removal of all caches and python cruft would not solve the problem. If that really is the case, it would be helpful if you could provide a minimal reproducer or a link to the repository if it is publc and the exact steps to reproduce the problem.

[letmaik]

@obestwalter Here it is: https://github.com/letmaik/pytest-issue-2042 Just clone it and run tox on it as it is. I could track it down to a very specific issue which may be an edge case: Importing your module under test within conftest.py. I had done that as I needed the module within a fixture. I could work around the issue by delay-importing, meaning I moved the import into the fixture functions.

[blueyed]

Confirmed:

GLOB sdist-make: /tmp/pytest-issue-2042/setup.py
python create: /tmp/pytest-issue-2042/.tox/python
python installdeps: pytest
python inst: /tmp/pytest-issue-2042/.tox/dist/pytest-issue-2042-1.0.0.zip
python installed: py==1.4.34,pytest==3.2.2,pytest-issue-2042==1.0.0
python runtests: PYTHONHASHSEED='638212311'
python runtests: commands[0] | pytest -s -v --doctest-modules --ignore=setup.py
========================================= test session starts =========================================
platform linux -- Python 3.6.2, pytest-3.2.2, py-1.4.34, pluggy-0.4.0 -- /tmp/pytest-issue-2042/.tox/python/bin/python
cachedir: .cache
rootdir: /tmp/pytest-issue-2042, inifile:
collected 0 items / 1 errors                                                                           

=============================================== ERRORS ================================================
___________________________________ ERROR collecting src/foo/bar.py ___________________________________
.tox/python/lib/python3.6/site-packages/py/_path/local.py:680: in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
E   py._path.local.LocalPath.ImportMismatchError: ('foo.bar', '/tmp/pytest-issue-2042/.tox/python/lib/python3.6/site-packages/foo/bar.py', local('/tmp/pytest-issue-2042/src/foo/bar.py'))
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================= 1 error in 0.13 seconds =======================================
ERROR: InvocationError: '/tmp/pytest-issue-2042/.tox/python/bin/pytest -s -v --doctest-modules --ignore=setup.py'
_______________________________________________ summary _______________________________________________
ERROR:   python: commands failed

Without investigating this looks unexpected to me, too - and is therefore likely another issue..

@letmaik
You are affected by the same error in some regard, but appear to have found another corner case to trigger it.

[letmaik]

@blueyed Another effect I discovered is that with my work-around all tests, both docstring and regular tests, are importing from the source folder, not the installed location, which defeats the purpose of using tox in the first place. To clarify, importing foo and printing foo.__path__ within a test shows the path to be not in site-packages but in the source location. If I disable docstring testing, then the installed module is used when importing. A partial work-around is to run pytest twice, once for docstring testing only, and another time for regular testing (by passing an explicit folder to test, in my case foo for docstring tests vs tests for unit tests), so that at least the regular tests are based on the installed module.

[obestwalter]

@blueyed thanks for enlightening me. I see now that there a very different ways to trigger this problem and I was only aware of one them until now.

[nicoddemus]

@letmaik are your tests installed with your package? If so, I believe using --pyargs mypackage.tests will work as you expect including doctesting.

[letmaik]

@nicoddemus No they are not.

[brianbruggeman]

I encounter this issue frequently. I do have multiple environments (host machine, docker, vm, etc.) and python versions (python 2 & 3), but this also happens when switching branches on a git repo to checkout patches to our main branch. For each environment, because I develop, I typically install the python package in development mode, which creates a .egg-info file, in addition to populating the folder tree with python bytecode files and __pycache__ folders. For some time, I also had pytest installed on my host system outside of the virtual environment, and that also caused the same kind of error.

I've internally codified the following cleanup procedure which I (potentially) run each time I switch branches:

$ hash -r
$ find . -name '*.py[odc]' -type f -delete
$ find . -name '__pycache__' -type d -delete
$ rm -Rf .pytest_cache
$ rm -rf *.egg-info .cache .eggs build dist
$ pip install -e .

The above seems to take care of the odd behavior.

[emil-petersen]

With pytest / docker, I ran into this issue, but solved it by deleting .pyc files.

On Unix, an easy solution is to run the following command: find . -name "*.pyc" -delete

Hope this helps

[jacebrowning]

Would patching this in py make sense?

https://github.com/pytest-dev/py/blob/bb4d0d0ca5c4b18a4b1276e2ccae3adf7f472bfa/py/_path/local.py#L685-L686

So that the line reads: if not issame and not os.getenv('PY_IGNORE_IMPORT_MISMATCH'):

[RonnyPfannschmidt]

we would like to completely remove our need for pylib over time

[blueyed]

Still a good intermediate step likely to allow for ignoring it using an environment variable?!

[CRImier]

Affected, too. Would be great to have an environment variable to bypass the checks.

[nicoddemus]

Still a good intermediate step likely to allow for ignoring it using an environment variable?!

IMHO yes, that solution would probably be enough for most use cases and very simple to implement.

[vlcinsky]

@brianbruggeman I just run into exactly the same issue: copied project repository from older project and did not rebuilt the tox created virtualenvs. This resulted in two different conftest.py being at hand and reported by pytest. Finally, I was happy to see that error as it could easily mess up my tests.

[tom-dalton-fanduel]

For anyone finding this page as a result of it being the first search result for a combination of the error message text and 'tox', there is a tox document with some suggested fixes. I hope that by putting this note here, others will be saved the frustrating day I've had trying to fix this issue with tox/pytest interaction and importable tests.

The section "installed-versus-checkout version" here: http://tox.readthedocs.io/en/latest/example/pytest.html#known-issues-and-limitations has a good explanation and solutions for this error.

(All that is not to say I wouldn't like pytest to take care of this for me :D)

[scottschreckengaust]

Dropping the --doctest-modules switch removes the ImportMismatchError error, I have not figured out how to include yet...

[RonnyPfannschmidt]

@scottschreckengaust currently doctest-modules is fundamentally incompatible with testing against a installed package due to module search in checkout vs module usage from site-packages

[rathko]

When working with docker, one handy option is to add .dockerignore, which will exclude all __pycache__ folders from being sent to the docker image context:

**/__pycache__

[DannyHinshaw]

Just ran into this issue with trying to run pytests in a docker container. After removing the usual pyc suspects, found out pytest has it's own cache, .pytest_cache that I had missed in .dockerignore / .gitignore.

[jmbowman]

After struggling for a while to come up with a workaround for this that didn't involve moving a large number of files in the source tree, I found one that worked pretty well for that particular case. In tox.ini's [testenv] section:

changedir = {envsitepackagesdir}

There's no mismatch if the current directory is the site-packages directory into which the package has been installed. This avoids undesirable changes to file paths which mess up coverage reporting with other workarounds involving changedir and/or --pyargs.

[brianbruggeman]

@jmbowman
Does it make any difference if the package is installed in develop mode? (i.e. python setup.py develop or pip install -e .)

[jmbowman]

@brianbruggeman I didn't try telling tox to do that, because one of the goals of the test suite is to make sure we didn't forget to include any important files in the package manifest; running the code from the source location fails to accomplish that. Using an editable install is a valid workaround in cases where that isn't a concern, though (for example, if you're testing a service instead of a package).

[brianbruggeman]

@jmbowman I understand your reasoning there, and one of my use-cases overlaps the very specific one you've enumerated. I can easily see that type of a solution used in a release cycle at the end or possibly through something like Travis and/or CI.

But, I think we may have different use-cases. I don't want a development environment to restrict its use to tox. Said differently, I want a developer to have the option to install in develop mode and to run pytest at will without an issue... especially if they swap their mac with a docker instance. Additionally, I don't think pytest's successful invocation should rely on yet another tool.

[jmbowman]

At least in this particular repo, pytest was working fine against the source directory even before the workaround (didn't even need an editable installation for that). The ImportMismatchError only occurred when running pytest in tox. I tend to use pytest directly for debugging individual test failures or running quick sanity checks of the test suite, but tox in CI or for testing with dependencies that don't match the repo's usual dev setup (different Python or Django version, etc.)

I had actually verified that all the tests were passing locally in pytest after converting from nose, but Travis failed on the PR because it was using tox by default. I then reproduced the issue by running tox locally also, where I started experimenting with all the workarounds that have been suggested over the last few years. This was the only one I found that didn't involve a major reorganization of the repo or break coverage collection.

[christianmlong]

For me, the problem was caused by a missing -e argument to pip as part of [testenv] commands in tox.ini.

Here's my working version of tox.ini

[tox]
envlist = py36
skipsdist = True

[testenv]
usedevelop = True
commands =
    pip install -e '.[test]'
    py.test
deps =
    -rrequirements.txt

If I use pip install '.[test]' instead of pip install -e '.[test]', I get the ImportMismatchError.

I specify pip install -e '.[test]' as part of the test commands so that I can declare my test dependencies in only one place, in setup.py.

[obestwalter]

Hi @christianmlong, as a tox specific note: your tox.ini is equivalent to

[tox]
envlist = py36
skipsdist = True

[testenv]
usedevelop = True
extras = test
commands = pytest
deps = -rrequirements.txt

so you don't need to emit any pip command and can control how it is installed purely by setting usedevelop.

see extras
see py.test vs pytest

[gillesdouaire]

We use symlinks to make our Python virtual envs portable, and are stuck on this problem. Import error is raised because path A is different than path B, although both paths are functional.

Everything on Python works fine with our portable venvs, except when we try to use one of our plugins in this path A/path B pytests context.

And these delete *.pyc / __pycache__ solutions do not apply to our case.

We'd love a switch so that pytest ignores the mismatch, or just throws a warning/info.

[gillesdouaire]

and not os.getenv('PY_IGNORE_IMPORT_MISMATCH')

Tested in our context, and that would work well.

[nicoddemus]

With 3.8 we have better warnings support, would changing this from an error to a warning be an acceptable solution?

[blueyed]

Sure - much better than an error.
But isn't this coming from py? (https://github.com/pytest-dev/pytest/issues/2042#issuecomment-353369932)

[nicoddemus]

Oh right. We would need to change py for that to work (which I don't see as a problem).

[RonnyPfannschmidt]

every time we need to change pylib we should take a look at if we can implement it better and simpler (with pathlib* for example)

[nicoddemus]

Oh good point, might be worth trying to port pyimport. 👍

[brianbruggeman]

Warnings are good as long as there's a configurable way to silence them...

[nicoddemus]

@brianbruggeman

Warnings are good as long as there's a configurable way to silence them...

The warnings filter of pytest 3.8 should work (I'm not sure if you are asking or you are saying it is not possible to silence them).

[RonnyPfannschmidt]

import mismatch as is is pretty much a critical error not a warning ^^

[nicoddemus]

My suggestion was to reduce the pain when there are false-negatives, but I understand your point.

[RonnyPfannschmidt]

hmm, lets define "false negatives"

[nicoddemus]

pytest throws if one collects from the source tree but the imports actually come from some installed location (e.g. some virtualenv). This happens more often with tox i guess because the default way of testing with tox is creating and installing a package and then collecting from the source tree.

Those cases, where we are certain the imported files and the compiled files are the same.

[RonnyPfannschmidt]

they are at different location and different kind of plug-in have massive issues with that fact (and every time its happens, its a indication that the project has a broken folder structure)

[blueyed]

@RonnyPfannschmidt
Please read some of the above comments: it typically happens when running tests first withing Docker and then without - that is caused by the file paths typically being different inside the Docker container, regardless of the project layout.
Also using symlinks appears to cause it (https://github.com/pytest-dev/pytest/issues/2042#issuecomment-423601601) in general.
I think we should add support for py to ignore this in case some env variable is set. That is useful in general.

[RonnyPfannschmidt]

im all for fixing problems, but it sounds a lot like all we will be doing is hiding/ignoring

[blueyed]

Yeah, I see.
https://github.com/pytest-dev/py/pull/199 will allow to ignore it, and it needs to be enabled explicitly. This should allow for working around cases where py is wrong / getting in the way.

While you can (and should) fix the problem when run with Docker (find -name "*.pyc" -delete), there appear to be cases where this is not possible (symlink).
The symlink issue might be a bug in py after all (and should get reported/fixed there then).
It would be good if @gillesdouaire would report it with py.

[gillesdouaire]

thanks very much for the MR @blueyed . I'll report the symlink issue with py project.

[nicoddemus]

py-1.7.0 has been released which supports using PY_IGNORE_IMPORTMISMATCH=1 to suppress this error. While this is not ideal, it is at least a escape hatch for cases where it is getting in the way. Thanks @blueyed for the patch! 🙇

[gillesdouaire]

Thanks! We just used the latest py and it works very well in our context :)

[blueyed]

I think it would be worth to add a note about using the env to https://github.com/blueyed/pytest/blob/6bf4692c7d8100b19dc0178764398edbd7803701/src/_pytest/python.py#L487-L496 (if py >= 1.7).

[shreyasbapat]

I am facing these issues here, any way to supress it? https://circleci.com/gh/einsteinpy/einsteinpy/75?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

[blueyed]

@shreyasbapat
Use the env?