Pytest: automatically register marks used in -m

[graingert]

I'd like to be able to mark a test as @pytest.mark.only then run the suite with pytest -m only without the --strict-markers warning triggering.

Then in CI running the suite with pytest will generate a strict-markers warning

[graingert]

it should be a case of parsing the "-m" expression to bytecode and whitelisting markers based on the LOAD_GLOBAL opcodes

[RonnyPfannschmidt]

https://pypi.org/project/pytest-only/

[RonnyPfannschmidt]

also your proposed solution is an absolute no go that falls flat for other normal usages

[The-Compiler]

also your proposed solution is an absolute no go that falls flat for other normal usages

Why? It seems like a reasonable idea to me to automatically whitelist markers being used with -m, as we can assume they're not a typo.

(It'd really help if you could present a rationale for statements like this in general - I feel like people need to ask for clarifications for your comments a lot, and providing some details rather than an absolute statement without further explanations would save both you and other people time.)

[RonnyPfannschmidt]

the problem is, that they are only valid if the expression is there,
aka, they turn back invalid if the expression is no longer used

in addition to that the proposed implementation is needlessly complex

given that the intent here seems to recreate the "only mark" feature that already was implemented in a plug-in, im opposed

also good call out, i need to provide better and more context way more often

[blueyed]

@graingert
Related: https://github.com/pytest-dev/pytest/issues/2872

(Just for reference: I'm using --mark-fixture myself (https://github.com/blueyed/pytest/pull/98, https://github.com/blueyed/pytest/commit/6837251511a0cdb40202ff094af4eaa6df415c8e).
I find it very useful to have "dynamic marks" created on demand, based on used fixtures - without any code changes.)

[graingert]

@blueyed are we considering this ticket and my suggested approach valid?

[blueyed]

@graingert cannot really say. I've re-opened it since it was closed by mistake (wanted to close my own issues only).

[Zac-HD]

FWIW I think this is a reasonable feature to have - the idea of the strict mark handling is to catch typos. This proposal doesn't weaken that at all, and it's a nice UX improvement for this workflow.

The implementation should be fairly self-contained too, just get the command-line marks and register them.

[nicoddemus]

Just so I'm clear: the proposal is for marks in -m to not trigger warnings, correct?

Wouldn't that also open the possibility for typos in the command-line with -m, just as it is possibly to mistype mark names in tests, which is the whole point of the warning?

[Zac-HD]

If you have the same typo'd mark name on the command-line (or config file adopts) as in your code, that would indeed suppress the warning.

I don't see this as a big problem though; if it's on the commandline you'll get the warning a little later, and if it's in a config file that's unfortunate but basically equivalent to registering a mark (badly).

To be clear I don't think that these outcomes are good, but I do think they're a price worth paying to support the -m only workflow that prompted this issue.

[nicoddemus]

I see, thanks:

To be clear I don't think that these outcomes are good, but I do think they're a price worth paying to support the -m only workflow that prompted this issue.

That sounds good at first glance, but for any test suite that wants to use this feature, they will eventually have to mark tests with pytest.mark.only, at which point it makes sense for them to register the mark in pytest.ini anyway, otherwise they will always get the warning... which defeats the purpose of -m automatically registering marks. Or am I missing a use case?

[nicoddemus]

And another point: all it does is generate a warning, which can then be ignored by the user. 🤷‍♂

[nicoddemus]

(I've changed the title to better reflect the intent, IMO)

[Zac-HD]

pytest-only is a plugin to "only run tests marked with @pytest.mark.only. If none are marked, all tests run as usual."

so in normal operation you don't need the only mark registered... but "the plugin should register their marker" is a fine resolution to me, with "users can register it" as a workaround until then.

[nicoddemus]

pytest-only is a plugin to "only run tests marked with @pytest.mark.only. If none are marked, all tests run as usual."

Yes, but I suspect the plugin registers pytest.mark.only, no?

[Zac-HD]

Nope, it doesn't. So perhaps we should close this issue in favor of encouraging pytest-only to register the mark?

[Zac-HD]

(misclick, oops!)

[nicoddemus]

Nope, it doesn't. So perhaps we should close this issue in favor of encouraging pytest-only to register the mark?

Definitely: https://github.com/theY4Kman/pytest-only/pull/8

[nicoddemus]

Yeah if the whole point of the issue was using pytest-only, then I think this can be closed as the fix lies in theY4Kman/pytest-only#8. 👍

[graingert]

For me this feature is for one off marks that I won't push to source control

I see, thanks:

To be clear I don't think that these outcomes are good, but I do think they're a price worth paying to support the -m only workflow that prompted this issue.

That sounds good at first glance, but for any test suite that wants to use this feature, they will eventually have to mark tests with pytest.mark.only, at which point it makes sense for them to register the mark in pytest.ini anyway, otherwise they will always get the warning... which defeats the purpose of -m automatically registering marks. Or am I missing a use case?

[nicoddemus]

For me this feature is for one off marks that I won't push to source control

But you can still do it, you will only see a warning...

[graingert]

But I have strict marks enabled

[nicoddemus]

If you are adding just a few marks temporarily to the code to only execute them, I think it is OK to also comment out the strict marks setting in your ini file.

We need to keep in mind that every extra feature incurs maintenance that we will need to carry around/support for many years; also, this feature also adds an exception that we will have to explain to users in the docs, and worse, might surprise users which mistype marks in the -m option and then suddenly their strict marks option is ignored from under them.

So IMHO this feature is not worth the added complexity, less in terms of code and more in terms of user-facing functionality that we have to explain.

[graingert]

What about a --no-strict-marks?

[nicoddemus]

To that I'm not opposed to immediately, because it would be a new option which is explicit about what it does.

But adding this option just in light of this rare use case is still dubious, I think. As I said, IMO if you are already modifying the code locally to add a bunch of temporary @pytest.mark.only marks around, why not do another one line change and comment out temporarily the error option in your filterwarnings setting?

Also, if you do this often enough that you feel this should be a common workflow, installing pytest-only wouldn't solve it for you?

[graingert]

Tbh once pytest-only is fixed my specific use case is handled

To that I'm not opposed to immediately, because it would be a new option which is explicit about what it does.

But adding this option just in light of this rare use case is still dubious, I think. As I said, IMO if you are already modifying the code locally to add a bunch of temporary @pytest.mark.only marks around, why not do another one line change and comment out temporarily the error option in your filterwarnings setting?

Because adding the extra marks will cause CI to fail but commenting pytest.ini will cause CI to incorrectly pass

[nicoddemus]

Because adding the extra marks will cause CI to fail but commenting pytest.ini will cause CI to incorrectly pass

Fair enough, there's opportunity for mistakes there.

But given that pytest-only solves the problem for you, I would put this on hold indefinitely (unless others who have commented in this thread think otherwise).

[theY4Kman]

Hey, y'all, I merged @nicoddemus's PR. Thank you for opening it, improving the plugin, and your consideration for the community :)