Pytest: Option to treat warnings as errors

[ghostsquad]

as related to #812

I like the idea of "no tests found" as a warning, and a config option to treat warnings as errors. That way, you can build up your CI without tests at first, and maybe with a few other "problems" that just bubble up as warnings. Then as you button things up, switch over to "treat warnings as errors".

[The-Compiler]

There should probably be some mechanism to ignore specific errors then - I'm still using various plugins which raise deprecation warnings (pytest-capturelog, pytest-bdd), and I don't have the time to fix them all (mainly an issue for pytest-bdd).

[nicoddemus]

There should probably be some mechanism to ignore specific errors then

Agreed, perhaps something like this?

[pytest]

# turn all pytest-warnings in errors
warnings_as_errors=all

# a few warnings by code: I9 and C1
warnings_as_errors=I9,C1

Btw, @RonnyPfannschmidt @hpk42, what does the "code" warnings mean? Here are a few examples for illustrations:

# cacheprovider.py
warn(code='I9', message='could not create cache path %s' % (path,)
warn(code='I9', message='cache could not write path %s' % (path,))

# config.py
warn("I2", "could not load setuptools entry import: %s" % (e,))

# junitxml.py
request.config.warn(code='C3', message=msg, fslocation=request.node.location[:2])

[RonnyPfannschmidt]

im not sure myself, unfortunately i couldnt partake in the design of the warnings system in detail

[The-Compiler]

@nicoddemus I think a way to say "all except ..." makes more sense - in my case I'd like to have _all_ warnings as errors, except a few deprecation warnings I can't fix easily.

[RonnyPfannschmidt]

by now i think the warning codes where a misstake, and would rather see something like the warning classes of the warnings module

[The-Compiler]

I agree - some plugins also use custom pytest warnings, and it's hard to think of a good warning code.

[The-Compiler]

I think this should be done after a bigger refactoring of the warning system like @RonnyPfannschmidt mentioned. I'm not sure if that can be done in a backwards-compatible way, so I'm tenatively changing the milestone for this one to 4.0.

[nicoddemus]

Related to #2452

[Zac-HD]

Since version 3.8, pytest uses Python's standard warnings mechanisms. You can treat warnings as errors in any of the normal ways, and also by running pytest -Werror or equivalent entries in a config file.

[nicoddemus]

@Zac-HD just wanted to mention that I really appreciate the issue triaging you have been doing in the repository lately!

[Zac-HD]

Just clearing stale questions and labelling untagged issues has gone a surprisingly long way :smile:

[nicoddemus]

Just clearing stale questions and labelling untagged issues has gone a surprisingly long way

Definitely!