Pytest: --pyargs does not understand namespace packages

[pytestbot]

Originally reported by: Wolfgang Schnerring (BitBucket: wosc, GitHub: wosc)

---

The pyargs resolution does not understand namespace packages when the different contributing packages are installed as eggs (which is the layout used by zc.buildout and also pip install --egg, see http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages). It looks like this:

site-packages/
    zope.asdf.egg/
        zope/
            asdf/
                ...
    zope.qwer.egg/
        zope/
            qwer/
                ...

This is a well supported layout, which means import zope.asdf and import zope.qwer work just fine (the specification is something to the effect of, if there is more than one directory that claims to be package "zope", there is no guarantee which one you will actually get when you say import zope, but all subpackages will be accessible regardless).

But py.test --pyargs zope.asdf will work, while py.test --pyargs zope.qwer will say "file or package not found" (it might also be the other way around, so zope.qwer works, but zope.asdf doesn't).

This is because _pytest.main.Session._tryconvertpyarg does not actually rely on the Python import mechanism to do the resolution (I'm not sure why, I'm guessing it's to support collecting tests outside of packages?). Instead it splits the argument on dots and loads the parts from the import system, assuming their filesystem location is the one that matters -- which is incorrect. In the example, trying to resolve zope.qwer, it will first resolve zope, which results in a random matching entry, e.g site-packages/zope.asdf/zope/__init__.py. It then assumes that the rest of the name must exist below _this specific_ directory, thus never finding zope.qwer.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/478

[scottpurdy]

The relevant code is here.

Instead of iteratively importing from the top level package, could it try importing from the full path first and, if it fails, iteratively pop the rightmost element off until it succeeds in importing and then do the file lookup from there?

I'm happy to implement a solution if there is a reasonable option available.

[RonnyPfannschmidt]

Since this is a setuptools mess, how about supporting module.path:object.path just like entry points of setuptools?

[scottpurdy]

That seems fine to me. Does it work for packages though? I don't have a strong opinion on the discovery mechanism as long as it supports namespace packages.

[RonnyPfannschmidt]

not sure off-hand, i'll probably need a while before i can touch it due to volunteering time constraints

[RonnyPfannschmidt]

@scottpurdy is this sill an issue, with the gradual phasing out of eggs im thinking of letting this one be unless someone makes a pr

[scottpurdy]

Not super high priority for me or I'd help out. Are eggs really being phased out?

[RonnyPfannschmidt]

yes, its no longer part of the equation in modern packaging tools
the common standard way of installation is pip+wheel

[eeaston]

.. unless you want to support zip importing, which is really important for slow filesystems like NFS. This bug is related to https://github.com/pytest-dev/pytest/issues/1445 - the test collector doesn't support zipfile importing either.
If I get some time soon I'll have a go at fixing this, the only workaround at the moment I can find is:

easy_install --always-unzip zope.asdf.egg
py.test $(python -c "import pkg_resources; print pkg_resources.get_distribution('zope.asdf').location")

[enkore]

This is kinda easy to trip over when using --pyargs, since one forgotten __init__.py lets py.test miraculously gloss over packages.

If a fix is complex, maybe just print a hint? Right now it says:

ERROR: file or package not found: somepkg

When we're on a Python supporting NS packages and --pyargs was used it could say something like...

ERROR: file or package not found: somepkg (missing __init__.py?)

[RonnyPfannschmidt]

@enkore very good point

[DuncanBetts]

I have made a Pull Request #2081 for commit ba3c1d2. Apologies if I should have waited for further discussion.

[nicoddemus]

Apologies if I should have waited for further discussion.

Not at all. It's no harm opening a PR, even if to entice more discussion. But I think adding just the message should be enough for this issue for now.

[nicoddemus]

2085 (thanks @DuncanBetts) adds the warning message at least.