Pytest: Make `--strict`/`--strict-markers`/`xfail_strict` consistent

[webknjaz]

_(It's a follow-up for https://twitter.com/codewithanthony/status/1263130307848335360)_

While reading the doc about markers, I encountered mentions of --strict/--strict-markers CLI args and an example for adding them via addopts. It was not clear if they are the same or not so I added both. Since some options support being specified as both a config setting and a CLI option, I first tried setting them outside of addopts (it didn't work).
I then recalled that there's also xfail_strict and compared the behaviors. xfail_strict turns out to be a config-only thing while --stict/--strict-markers are CLI-only. Later I learned that --stict==--strict-markers.

It's not obvious why they exist in these specific forms but it seems like it'd be beneficial to adjust this inconsistency and maybe add a few more clarifications to the docs.

WDYT?

• [x] a detailed description of the bug or suggestion
• [ ] output of pip list from the virtual environment you are using
• [ ] pytest and operating system versions
• [ ] minimal example if possible

[nicoddemus]

Thanks @webknjaz!

Indeed this is all a bit confusing, and the result of options growing organically in pytest over the years. 👍

[webknjaz]

Do you think that it'd be reasonable to support these options in both CLI and config?
Also, should --strict mean --strict-markers+xfail_strict?

[nicoddemus]

Here's the long story. 😁

--strict and --strict-markers are aliases (the latter is more recent), and both are independent from xfail_strict.

--strict-markers: makes unknown markers raise errors. This was historically named --strict, which is confusing because it is too broad, while it only applies to markers. The --strict alias remains for backward compatibility only (so we should definitely hunt and change all instances of --strict to --strict-markers).

xfail_strict: this changes the default value of the strict parameter of pytest.mark.xfail.

Historically (nobody remembers exactly when, I think), xfail changed from failing the test suite if the test passes (meaning, you assume a test should fail, but if it passes, then there's something wrong so the test suite fails so you will notice), to just show the test as x in the terminal if an xfail-marked test passes, but otherwise does not fail the test suite.

We believe this was changed so pytest.mark.xfail could be used to mark "flaky" tests, which fail on occasion, in a way to not break the test suite (but again I'm not sure).

Later on, some users correctly point out that they have true "xfail" tests, meaning that if the test passes, the test suite should fail (for example, they test against a known bug of a third party library that makes a certain test fail; if an upgrade causes the test to pass, then they want to know about it loudly, so the test suite fails accordingly). To allow for that use case, we included the strict parameter to pytest.mark.xfail, defaulting to False (again for backward compatibility). Using @pytest.mark.xfail(strict=True) will have this behavior.

The xfail_strict ini option changes the default value of the mark, so users can apply their preference broadly across the project, rather than having to enforce it in every xfail mark.

---

pytest is a very mature and stable project, and has grown options/docs/features organically through the years. This leads sometimes to some options not being so clear as they should, as well as a ton of backward compatibility concerns.

That's why this days we are very careful about introducing new APIs because we know how long we have to end up supporting those, as well as having backward compatibility in place.

Again, thanks for bringing this up, this is another of the areas we could use some improvement. 👍

[DahlitzFlorian]

Hi @nicoddemus! Should we add a short note about this to the documentation, the preference fpr --strict-markers, and that --strict might be extended in the future? I'm fairly new to pytest and would be happy to work on this!

[nicoddemus]

Hi @DahlitzFlorian,

Thanks for the offer to work on this, we appreciate it a lot!

Should we add a short note about this to the documentation, the preference fpr --strict-markers, and that --strict might be extended in the future?

Probably it is worth adding a historical note somewhere, but there are no plans to extend --strict in the future (I believe it would only lead to more confusion).

Feel free to open a PR and we can discuss the fine details over there. 👍

[webknjaz]

there are no plans to extend --strict in the future (I believe it would only lead to more confusion).

What about adding a new option like --strict-mode that would influence any strictness related defaults?

[nicoddemus]

Not at the moment (I'm not saying it is not a good idea, just that there are no plans for it hehehe).

Oh and I was citing --strict (just --strict), other strict-related options of course are possible (--strict-mode like you suggested).

[nicoddemus]

Closing this in favor of https://github.com/pytest-dev/pytest/issues/7503.

Thanks again @webknjaz!