Pytest: deprecation warnings when imported on python 3.7 beta

[irmen]

Python 3.7 will introduce a few new API deprecation warnings. They're triggered when importing pytest on python 3.7 beta:

11:20 $ python3.7 -Wall 
Python 3.7.0b2+ (heads/3.7:3b4c6b16c5, Mar 22 2018, 23:55:22) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import  pytest
/home/irmen/.local/lib/python3.7/site-packages/_pytest/assertion/util.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sequence
/home/irmen/.local/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:6: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/home/irmen/.local/lib/python3.7/site-packages/_pytest/mark/structures.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import namedtuple, MutableMapping as MappingMixin
>>> pytest.__version__
'3.5.0'
>>> 

pytest version: 3.5.0
python version: cpython 3.7 built from source (3.7 branch)
OS: debian 9

[pytestbot]

GitMate.io thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/2191 (add an option to warn about Python 3.x possible incompatibilities), https://github.com/pytest-dev/pytest/issues/2118 (pytest triggering deprecation warnings in 3.0.5), https://github.com/pytest-dev/pytest/issues/2327 (capsys not capturing on Python 2.7), https://github.com/pytest-dev/pytest/issues/2674 (SystemError: 'finally' pops bad exception with python 3.7 (master branch)), and https://github.com/pytest-dev/pytest/issues/2917 (Asserting repeated warning does not work in Python 2.7).

[nicoddemus]

Thanks @irmen for the report!

[RonnyPfannschmidt]

this si a easy fix - we need to try to import from collections.abc first

[feuillemorte]

collections.abc is easy, but the problem with changing imp to importlib: pytest stops with error:

RuntimeError: maximum recursion depth exceeded while calling a Python object

[RonnyPfannschmidt]

lets fix the easily fixable first then ^^ i wont have time to investigate soonish

[RonnyPfannschmidt]

i just tried to fix imp importing, its not actually easy as things are named differently and it needs help on python3

we might need to drop python2 support or spend quite some time for a compat layer
an alternative would be using importlib2 on python2, but that might come with an own can of worms

[irmen]

For Serpent, I fixed it with a simple version check like this:
https://github.com/irmen/Serpent/commit/a0bb00881504487629c0470fbca5df1ef9bbb652#diff-e4eda821e02f4c665914be5d8a0a7a5dR23

[RonnyPfannschmidt]

@irmen thats good for the abc one ^^ up for making that pr for pytest module?

[irmen]

Sure I'm not familiar with the Pytest code base yet but this shouldn't be too hard.

[irmen]

note that at least more_itertools which is a dependency, has the same deprecation warning. I guess I'll be stalking them too soon

[irmen]

I'm not able to fix the deprecation warnings from using the imp module in assertion/rewrite.py there's a bit too much wizardry going on there

[RonnyPfannschmidt]

@irmen correct, its quite tricky, please dont try unless you are fluent - i fear we might need to port to importlib/importlib2 in general

[nicoddemus]

I agree with @RonnyPfannschmidt, that bit is quite tricky I'm afraid.

[hroncok]

We've got a lot of tests failures in Fedora because of this:

testing/python/approx.py FFFFFFFFFFFFFFFFFFFFFFFFFFFss.FFFFFss           [ 80%]

...

E           DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

We run the testsuite as:

PATH=/builddir/build/BUILDROOT/pytest-3.5.1-1.fc29.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin:/builddir/.local/bin:/builddir/bin
PYTHONPATH=/builddir/build/BUILDROOT/pytest-3.5.1-1.fc29.x86_64/usr/lib/python3.7/site-packages
/builddir/build/BUILDROOT/pytest-3.5.1-1.fc29.x86_64/usr/bin/pytest-3.7 -r s testing

Any hack to get rid of such failures? I've been trying to use -W ignore::DeprecationWarning but it didn't help. Thanks

full log

[nicoddemus]

@hroncok this has been released in 3.5.1 already, see: https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst#trivialinternal-changes

Not sure why you are still seeing this warnings, here's the code:

https://github.com/pytest-dev/pytest/blob/8e17e322535b36d01cf6ba1199f8a812858062c1/_pytest/compat.py#L41-L47

[hroncok]

This is 3.5.1, that's the weird thing.

[hroncok]

this is in master:

https://github.com/pytest-dev/pytest/blob/8e17e322535b36d01cf6ba1199f8a812858062c1/_pytest/python_api.py#L429

[hroncok]

testing with:

diff --git a/_pytest/compat.py b/_pytest/compat.py
index a5fa037..1a4b750 100644
--- a/_pytest/compat.py
+++ b/_pytest/compat.py
@@ -40,11 +40,11 @@ MODULE_NOT_FOUND_ERROR = 'ModuleNotFoundError' if PY36 else 'ImportError'

 if _PY3:
     from collections.abc import MutableMapping as MappingMixin  # noqa
-    from collections.abc import Sequence  # noqa
+    from collections.abc import Mapping, Sequence  # noqa
 else:
     # those raise DeprecationWarnings in Python >=3.7
     from collections import MutableMapping as MappingMixin  # noqa
-    from collections import Sequence  # noqa
+    from collections import Mapping, Sequence  # noqa


 def _format_args(func):
diff --git a/_pytest/python_api.py b/_pytest/python_api.py
index 8e09a4a..838a4a5 100644
--- a/_pytest/python_api.py
+++ b/_pytest/python_api.py
@@ -426,7 +426,7 @@ def approx(expected, rel=None, abs=None, nan_ok=False):
        __ https://docs.python.org/3/reference/datamodel.html#object.__ge__
     """

-    from collections import Mapping, Sequence
+    from _pytest.compat import Mapping, Sequence
     from _pytest.compat import STRING_TYPES as String
     from decimal import Decimal

(Edited to add Mapping to compat)

[RonnyPfannschmidt]

that one needs to go into 3.6

[nicoddemus]

@hroncok would you like to open a PR? If we can get this into 3.6 it would be great.

[hroncok]

I'm still testing a build, because the first patch was incomplete. A PR is incoming, just want to confirm it fixes the problem.

[hroncok]

succeeded!

[RonnyPfannschmidt]

@hroncok please provide a pr - we'd like to include in the next release which is due today or early tommorow

[hroncok]

I said I'm working on it. Here it is https://github.com/pytest-dev/pytest/pull/3497

[nicoddemus]

We can close this now, the imp import is already tracked by #1403.

Thanks everyone!

[schlomo]

Also reported in https://github.com/boto/botocore/issues/1615