Pytest: Asserting repeated warning does not work in Python 2.7

[jgosmann]

Running the following test code with Python 2.7 will fail because for some reason the warning is not emitted a second time:

import warnings
import pytest

def warn():
    warnings.warn(UserWarning("test"))

def test_warning():
    warn()
    with pytest.warns(UserWarning):
        warn()

The output:

============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/jgosmann, inifile:
collected 1 item

test2.py F

=================================== FAILURES ===================================
_________________________________ test_warning _________________________________

    def test_warning():
        warn()
        with pytest.warns(UserWarning):
>           warn()
E           Failed: DID NOT WARN. No warnings of type (<type 'exceptions.UserWarning'>,) was emitted. The list of emitted warnings is: [].

test2.py:13: Failed
=============================== warnings summary ===============================
test2.py::test_warning
  /home/jgosmann/test2.py:7: UserWarning: test
    warnings.warn(UserWarning("test"))

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 failed, 1 warnings in 0.01 seconds =====================

Running with Python 3.6.1 succeeds:

============================= test session starts ==============================
platform linux -- Python 3.6.1, pytest-3.2.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/jgosmann, inifile:
collected 1 item

test2.py .

=============================== warnings summary ===============================
test2.py::test_warning
  /home/jgosmann/test2.py:7: UserWarning: test
    warnings.warn(UserWarning("test"))

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 0.00 seconds =====================

Note that this also occurs across different tests, e.g. when parametrizing:

@pytest.mark.parametrize('i', range(2))
def test_warning(i):
    if i % 2 == 0:
        warn()
    else:
        with pytest.warns(UserWarning):
            warn()

When running tests with pytest-xdist in non-deterministic order this can cause random test failures with Python 2.7.

[jgosmann]

It seems a workaround is to add warnings.simplefilter('always') to pytest_configure in conftest.py.

[The-Compiler]

That looks expected to me - that's just how warnings work in Python by default :wink:

[jgosmann]

Can you point me to some part of the documentation that would explain that behaviour? The default warning filter section for 3.6 and 2.7 only talk about ignored warnings (that are not UserWarning). When I start a Python interpreter and repeatedly do warnings.warn('a') I get only a single warning in both Python versions. So it seems that there is a once filter on UserWarnings by default? But that does not explain why in pytest with Python 3.6 I do not have to set an always filter to get all warnings.

[jgosmann]

Also, the aforementioned workaround does not reliably work in conjunction with pytest-xdist. (But I cannot manage to produce a test case, it seems to only fail on Travis-CI.)

[The-Compiler]

I can't explain this off-hand I'm afraid.

[sloria]

We're running into this as well in the marshmallow test suite.

Here's a failed build on travis: https://travis-ci.org/marshmallow-code/marshmallow/builds/324608482

The test in question looks like this:

def test_strict_is_deprecated():
    with pytest.warns(DeprecationWarning):
        class StrictUserSchema(Schema):
            name = fields.String()

            class Meta:
                strict = False

    class UserSchema(Schema):
        name = fields.String()

    with pytest.warns(DeprecationWarning):
        UserSchema(strict=True)

The problem only exists in 2.7--the test passes fine in Python 3.

[sloria]

Update: The test still fails when using pytest.warns(DeprecationWarning) but it does not fail if we use pytest.deprecated_call():

def test_strict_is_deprecated():
    with pytest.deprecated_call():
        class StrictUserSchema(Schema):
            name = fields.String()

            class Meta:
                strict = False

    class UserSchema(Schema):
        name = fields.String()

    with pytest.deprecated_call():
        UserSchema(strict=True)

[nicoddemus]

pytest.deprecated_call() goes out of its way to ensure it can capture the warnings, while pytest.warns uses warnings.catch_warnings underneath so that might explain the difference in behavior (see recwarn.py).

I would suggest to change the code to use deprecated_call to catch deprecation warnings, but I agree it might seem confusing.

[jgosmann]

This problem also occurs for non-deprecation warnings. Maybe pytest.warns should capture warnings in the same way as pytest.deprecated_call?

[jccurtis]

I added the filter workaround mentioned by @jgosmann to the package level pytest config: (setup.cfg in this case) as explained in the docs:

# setup.cfg
[tool:pytest]
filterwarnings = always

[asottile]

good news, looks like #4104 will fix this :)

[asottile]

via #4104