Pytest: Pathlib compat fails under Python 3.5

[khaeru]

pytest's compatibility code chooses to use pathlib2 except for Python 3.6. This means that pathlib2.Path objects are returned for fixtures/fixture methods like tmp_path_factory.mktemp(). When the user tests code using Python 3.5's built-in pathlib, the built-in package chokes on the objects from pathlib2.

Put the following in test_foo.py and execute using pytest under Python 3.5:

from pathlib import Path  # available in Python 3.5


def foo(p):
    p2 = p / 'foo.txt'
    print(p2.exists())


def bar(p):
    p2 = Path(p)
    print(p2.exists())


def test_foo(tmp_path_factory):
    foo(tmp_path_factory.mktemp('baz1'))


def test_bar(tmp_path_factory):
    bar(tmp_path_factory.mktemp('baz2'))

…this occurs:

$ pytest test_foo.py
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.5.2, pytest-4.4.0, py-1.5.2, pluggy-0.9.0
Matplotlib: 2.1.2
Freetype: 2.6.1
rootdir: /home/gidden/work/iiasa/message/ixmp, inifile: setup.cfg
plugins: mpl-0.9, cov-2.6.0
collected 2 items

test_foo.py .F                                                                                                                                                             [100%]

==================================================================================== FAILURES ====================================================================================
____________________________________________________________________________________ test_bar ____________________________________________________________________________________
tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7f0779c3f1d0>, _basetemp=PosixPath('/tmp/pytest-of-gidden/pytest-22'))
    def test_bar(tmp_path_factory):

>       bar(tmp_path_factory.mktemp('baz2'))
test_foo.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_foo.py:13: in bar
    p2 = Path(p)
/usr/lib/python3.5/pathlib.py:969: in __new__
    self = cls._from_parts(args, init=False)
/usr/lib/python3.5/pathlib.py:651: in _from_parts
    drv, root, parts = self._parse_args(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cls = <class 'pathlib.PosixPath'>, args = (PosixPath('/tmp/pytest-of-gidden/pytest-22/baz20'),)
    @classmethod
    def _parse_args(cls, args):
        # This is useful when you don't want to create an instance, just
        # canonicalize some constructor arguments.
        parts = []
        for a in args:
            if isinstance(a, PurePath):
                parts += a._parts
            elif isinstance(a, str):
                # Force-cast str subclasses to str (issue #21127)
                parts.append(str(a))
            else:
                raise TypeError(
                    "argument should be a path or str object, not %r"
>                   % type(a))
E               TypeError: argument should be a path or str object, not <class 'pathlib2.PosixPath'>
/usr/lib/python3.5/pathlib.py:643: TypeError
======================================================================= 1 failed, 1 passed in 0.13 seconds =======================================================================

The same command succeeds under Python 3.6.

Discovered by @gidden

[nicoddemus]

Hi @khaeru, thanks for the report.

We decided to use pathlib2 on py<3.6 because the standard pathlib was missing a few methods IIRC. @RonnyPfannschmidt knows the details.

A workaround is to use Path(str(p)) instead of Path(p), but I believe you already know that.

[khaeru]

Hi @nicoddemus — yes, that's what we're doing for now.

However, it was somewhat cryptic, so still took us some sleuthing to figure out was going on. Only when the pytest-returned pathlib2 objects hit pathlib code (e.g. "bar" in the snippet above) does this happen; in other cases ("foo") it does not.

For less experienced pytest users, it might help to:

• explicitly state in the documentation which of pathlib/pathlib2 is used, and/or
• catch/re-raise with an explanatory message/hint.

[RonnyPfannschmidt]

Its a unfortunate fact that pathlib pre python 3.6 is a broken mess

[nicoddemus]

I will see if we can drop pathlib2 on Python 3; I think we can live with the missing bits in Python 3.4 and 3.5.

[nicoddemus]

Hi @khaeru,

We have decided that reverting to standard pathlib in Python 3.5 will be worse overall because it will break many test suites due to the API change in the tmp_path fixture.

We have updated the docs for some time now explaining pathlib2 objects are returned in Python 3.5.

Thanks again. :+1:

[khaeru]

Hi @nicoddemus —thanks for the valiant attempt to overcome that upstream brokenness!

Hopefully anyone who encounters the same problem can stumble upon this issue and learn that they might need to explicitly cast to pathlib.Path or pathlib2.Path on 3.5.