Junit5: Support repeatable annotations for built-in conditions

[rocketraman]

Junit should support multiple condition annotations on classes and tests. One use case for this is a test that should not run in two situations: the first is that it should never run in the CI environment (CI=true), and the second is if a dev sets another environment variable (SKIP_X=true). One approach is to use the @DisabledIfEnvironmentVariable twice.

The only solution currently is to do @DisabledIfEnvironmentVariable(named = "SKIP_X", matches = "true"), and then set SKIP_X in the CI environment. However, with this approach we lose some self-documentation -- the explicit CI=true check documents the fact that this test doesn't run currently in the CI environment, but may in the future, and is an easy way to find all such cases. It is also far less maintainable, as each such case requires modifying the CI configuration settings.

Deliverables

• [x] Support multiple condition annotations on classes and tests

[sbrannen]

Tentatively slated for 5.5 M1 for _team discussion_

[sbrannen]

Proof of concept: _in progress_

[sbrannen]

The proof of concept can be seen here:

https://github.com/junit-team/junit5/compare/issues/1793-multiple-condition-annotations

[marcphilipp]

Team Decision: use new repeatable annotation lookup for execution condition annotations

to be determined later: use new repeatable annotation lookup for registering multiple @ArgumentSource annotations of the same type for parameterized tests.

[sbrannen]

Update: the latest work on this can be seen in the following branch for anyone interested.

https://github.com/junit-team/junit5/commits/issues/1793-repeatable-conditions

[sbrannen]

@junit-team/junit-lambda, I have added the _team discussion_ label in order to discuss the semantics we want for the proposed feature.

Some topics to consider:

• Repeated annotations like @EnabledIfEnvironmentVariable and @EnabledIfSystemProperty could be supported with AND or OR semantics but not both simultaneously (at least not with the current approach). Developers could naturally desire one or the other: the "best" semantics are therefore debatable.
• We currently support OR semantics for annotations like @EnabledOnOs since we accept multiple OSes via an array. Thus, we may choose to simply align with that existing feature and apply OR semantics to repeated annotations for environment variables and system properties; however, doing so would break from the traditional semantics of individually processed conditions.
• Treating annotations such as @EnabledOnOs and @EnabledOnJre as repeatable annotations with AND semantics doesn't make sense, since the values are mutually exclusive -- for example, it doesn't make sense to say that a test is enabled on Mac _and_ on Windows at the same time (i.e., for the currently executing test suite) since the evaluation of those conditions would always result in disabled && enabled (if executed on either a Mac or Windows OS) which equates to disabled.

So, before we complete this work, we need to decide on the desired semantics for all such built-in conditions.

[sbrannen]

@DisabledIfEnvironmentVariable(named = "CI", matches = "true")
@DisabledIfEnvironmentVariable(named = "SKIP_X", matches = "true")

@rocketraman, given the above, are you proposing those checks be treated with AND or OR semantics?

[sormuras]

Same same as if you declare

@DisabledIfEnvironmentVariable(named = "CI", matches = "true")
@DisabledOnJre(JAVA_8)

today.

[rocketraman]

@sbrannen I was definitely proposing OR semantics for repeatable annotations. With the example given in the OP, I'd like to disable in either situation: if CI=true, OR SKIP_X=true. As @sormuras says, this is also consistent with the behavior of multiple non-repeated annotations.

[marcphilipp]

Team Decision:

1. Treat individual annotation declarations as individual execution conditions.
2. Don't add support for @EnabledOnOs, @DisabledOnOs, @EnabledOnJre, and @DisabledOnJre as repeatable annotations at this time.

[sbrannen]

Update: further work has been pushed to the aforementioned feature branch.

[sbrannen]

For anyone interested, I've opened a PR: #2094

[sbrannen]

This is now in master. Feel free to try it out in the upcoming 5.6 M2 snapshot builds.