Junit5: Run only "untagged" tests

[calvertdw]

I request the ability to run only "untagged" tests.

Most tests will run with default arguments and should not require @Tag("fast"). Other tests require special needs or take a long time so we tag them (i.e. @Tag("slow")). I would prefer not to require the majority of tests to be tagged at all.

Current workarounds include parsing the code for all declared instances of @Tag("*") or using the Launcher#discover method in the build phase, such that all found tags can be excluded.

Deliverables

• Provide a keyword (i.e. "all") for the excludeTags property to exclude all tests that are tagged OR some solution providing the equivalent result.

[marcphilipp]

all won't work because someone might already be using a tag with that name.

We could introduce special tag expressions for "no tags" and "at least one tag", e.g. none() and !none().

@junit-team/junit-lambda WDYT?

[sbrannen]

all won't work because someone might already be using a tag with that name.

I agree that all is not an option.

We could introduce special tag expressions for "no tags" and "at least one tag", e.g. none() and !none().

I had a similar idea: _great minds think alike_. 😉

How about tagged() and !tagged()?

[sbrannen]

Though in terms of the tag expression parsing, it might be better to avoid the use of parentheses.

So perhaps something like $tagged and !$tagged?

We'd just need to make sure we pick special characters that were not previously valid (i.e., perhaps already used by somebody).

Disclaimer: I haven't looked at the exact syntax parsing recently. So the above suggestion may potentially be bogus. 😇

[marcphilipp]

If I'm not mistaken, $tagged would currently be considered a valid tag (expression). I added the parentheses, because none() is currently invalid.

How about none() and any()?

[calvertdw]

As a note, my current workaround is to use the Launcher#discover method as part of my build, such that all found tags can be excluded. Our specific implementation of this can be found here.

[sbrannen]

If I'm not mistaken, $tagged would currently be considered a valid tag (expression).

Yeah... as I hinted at, my proposal could well have been bogus, and it in fact is.

In hindsight, we really should have been more restrictive with valid characters in tag names (see org.junit.platform.engine.TestTag.RESERVED_CHARACTERS). Oh well... too late now.

I added the parentheses, because none() is currently invalid.

How about none() and any()?

How about none() and all()?

As in...

• excludeTags( all() )
• includeTags( none() )

[marcphilipp]

I think any() would be better because it is a tag expression that matches all tests that have _any_ tag while none() matches tests that have _none_.

[sbrannen]

I think any() would be better because it is a tag expression that matches all tests that have _any_ tag while none() matches tests that have _none_.

OK. I can agree with you on that, when you phrase it like that. 😉

[calvertdw]

To make it more clear:

any() would match all tests that have at least one tag.
none() would match all tests that have exactly zero tags.

[calvertdw]

Still looking for this! It's nice for running tests locally in IntelliJ w/o Gradle. It'd be snappy to pass a tag expression to run untagged tests.

[marcphilipp]

Tentatively slated for 5.6 M2 solely for the purpose of _team discussion_.

[marcphilipp]

Team Decision: Let's go with any() and none().

@calvertdw Would you be interested in submitting a PR?

[signed]

Should the special terms play nice with operators or be treated separately?

• ! none()
• ! any()
• none() | somethingelse
• any() | somethingelse (you could just remove something else)
• any() | none()
• any() & none()
• any() & somethingelse (you could just remove any in this case)

Or is only this allowed and the previous examples are all invalid input?

• none()
• any()

Treating the 1st sample block as invalid inputs makes the implementation way easier.

[mmerdes]

I would favour simplicity over completeness

[marcphilipp]

If it considerably simplifies the implementation, I'm fine with only supporting none() and any() as standalone expressions for now.

[mmerdes]

In progress

[calvertdw]

I think none() | somethingelse would quickly become a useful expression but adding standalone expressions for none() and any() are already a huge improvement. I am fine with either.

Consider the documentation and support in factoring the complexity. Not supporting expressions might be confusing to users.

[calvertdw]

I see @mmerdes has taken this up, but I might be able to assist given some pointers in the right direction. What classes/interfaces are relevant? In general, what's a good way to do it?

[mmerdes]

@calvertdw
Thanks for the offer! I will let you know if I encounter any roadblocks :)

[tcfurrer]

I just wanna say "Thank You" to everyone who worked on this enhancement. My project's Gradle scripts got simplified significantly when I was finally able to make use of the new any() and none() capability of JUnit 5.6. It's very useful!