pytest 5.4+ doesn't handle IsolatedAsyncioTestCase from CPython core.

[cjw296]

So, https://github.com/pytest-dev/pytest/pull/5734 causes pytest to incorrectly skip some async tests, specifically those based on https://docs.python.org/3/library/unittest.html#unittest.IsolatedAsyncioTestCase or code like it

See https://github.com/pytest-dev/pytest/pull/5734#issuecomment-599023047 for real world failure, the mock backport has some of its own code in this area, to cater for Python 3.6 and 3.7:
https://github.com/testing-cabal/mock/blob/b5ce0a5c4d372b77deff46fec8edf974d7d1f875/mock/backports.py#L38-L82

For now, I'll just pin to pytest<5.4, but would be interested to know in how this will be fixed.

[cjw296]

Here's a minimal reproducer for Python 3.8 that works on pytest 5.3, but not on 5.4:

from unittest import IsolatedAsyncioTestCase


class AsyncArguments(IsolatedAsyncioTestCase):

    async def test_something_async(self):

        async def addition(x, y):
            return x+y

        self.assertEqual(await addition(2, 2), 4)

[graingert]

Here's a minimal reproducer for Python 3.8 that works on pytest 5.3, but not on 5.4:

from unittest import IsolatedAsyncioTestCase


class AsyncArguments(IsolatedAsyncioTestCase):

    async def test_something_async(self):

        async def addition(x, y):
            return x+y

        self.assertEqual(await addition(2, 2), 4)

What happens if you run a failing test? Eg: assert await addition (2, 2) == 3?

[cjw296]

$ pytest mock/tests/test_reproducer.py 
========================================================= test session starts =========================================================
platform darwin -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/chris/vcs/git/mock, inifile: setup.cfg
plugins: cov-2.8.1
collected 1 item                                                                                                                      

mock/tests/test_reproducer.py F                                                                                                 [100%]

============================================================== FAILURES ===============================================================
_________________________________________________ AsyncArguments.test_something_async _________________________________________________

self = <mock.tests.test_reproducer.AsyncArguments testMethod=test_something_async>

    async def test_something_async(self):

        async def addition(x, y):
            return x+y

>       self.assertEqual(await addition(2, 2), 3)
E       AssertionError: 4 != 3

mock/tests/test_reproducer.py:11: AssertionError

IsolatedAsyncioTestCase's run method does all the async stuff, in the same way an async plugin would:

        def run(self, result=None):
            self._setupAsyncioLoop()
            try:
                return super().run(result)
            finally:
                self._tearDownAsyncioLoop()

Remember: this is a unittest.TestCase subclass, so it can do lots in that run method...

[RonnyPfannschmidt]

thanks for the details, i believe we should ignore async for unittest subclasses, i will try to fix this afternoon

[cjw296]

I think it might also make sense to have a setting to turn this off...

[RonnyPfannschmidt]

@cjw296 there is another issue at play, i made a opt-out specifically for unit-test and get

unittest/test_unittest_asyncio.py::AsyncArguments::test_something_async
  /home/ronny/Projects/pytest-dev/pytest/.env/lib/python3.8/site-packages/pluggy/callers.py:187: RuntimeWarning: coroutine 'AsyncArguments.test_something_async' was never awaited
    res = hook_impl.function(*args)

unittest/test_unittest_asyncio.py::AsyncArguments::test_something_async_fails
  /home/ronny/Projects/pytest-dev/pytest/.env/lib/python3.8/site-packages/pluggy/callers.py:187: RuntimeWarning: coroutine 'AsyncArguments.test_something_async_fails' was never awaited
    res = hook_impl.function(*args)

[RonnyPfannschmidt]

https://github.com/pytest-dev/pytest/pull/5996 is the real origin of the issue, a fix will need some more time to understand

[cjw296]

Okay, if it helps, there’s no massive rush on this, pinning to the older version is fine for the foreseeable future...

[grigi]

I have the same issue with using https://github.com/Martiusweb/asynctest with CPython 3.7 in my project.

[nicoddemus]

5996 is the real origin of the issue, a fix will need some more time to understand

Indeed, that's a very complicated (and emotional) thread.

Would dropping support for --pdb for TestCase-based tests be an option? It would greatly simplify the code.

[RonnyPfannschmidt]

I have a fix in mind but caught a cold in between, I'll take a look later

[nicoddemus]

Thanks, get well soon!

[blueyed]

Any update here?
pytest-django is basically waiting on this getting fixed / and/or reverted.

[RonnyPfannschmidt]

currently i'm dealing with the corona fallout, i have no idea when i will be able to commit time to this again

[nicoddemus]

@RonnyPfannschmidt if you have the time, can you share what you have in mind? I might take a stab at this then.

[RonnyPfannschmidt]

@nicoddemus i have a working implementation i will push now, will add tests slightly later

[RonnyPfannschmidt]

see the pr

[blueyed]

While #6927 enables them again, they still do not work as expected (due to https://github.com/pytest-dev/pytest/issues/6947).

I.e. this fails:

diff --git i/testing/example_scripts/unittest/test_unittest_asyncio.py w/testin>
index 16eec1026..4e6925c46 100644
--- i/testing/example_scripts/unittest/test_unittest_asyncio.py
+++ w/testing/example_scripts/unittest/test_unittest_asyncio.py
@@ -1,7 +1,13 @@
 from unittest import IsolatedAsyncioTestCase  # type: ignore


+teardowns = []
+
+
 class AsyncArguments(IsolatedAsyncioTestCase):
+    async def asyncTearDown(self):
+        teardowns.append(self)
+
     async def test_something_async(self):
         async def addition(x, y):
             return x + y
@@ -13,3 +19,6 @@ async def addition(x, y):
             return x + y

         self.assertEqual(await addition(2, 2), 3)
+
+    def test_teardowns(self):
+        assert len(teardowns) == 2

From https://docs.python.org/3/library/unittest.html#unittest.IsolatedAsyncioTestCase.asyncTearDown:

This is called even if the test method raised an exception

[RonnyPfannschmidt]

good catch