Junit5: Support @ExtendWith at package level and module level

[iamdanfox]

Context

I've written an extension which deterministically 'shards' my tests to ensure they run on exactly one CircleCI node (by hashing the name into CIRCLE_NODE_TOTAL buckets), allowing me to dial up parallelism and make my builds go faster.

However, it's kinda annoying having to add my annotation @CircleCiSharding to every single test class. I considered using the automatic registration functionality (-Djunit.jupiter.extensions.autodetection.enabled=true), but I don't like the risk that some other obscure test dependency will then magically start applying its own extension to my tests. I'd prefer to retain 100% control of exactly what I enable, it's just quite verbose with annotations on every class at the moment.

Another option is to have a single TestBase which all other tests inherit from, but it's very easy to accidentally forget to use this when writing a new test.

Proposal

My suggestion would be to allow people to write a package-info.java:

@ExtendWith(CircleCiShardingExtension.class)
package com.foo.bar;

I'm imagining this would apply the extension to all tests within this package. This satisfies my goals because:

• I don't have to enable magical auto-registration, so dependency changes can't change the behaviour of my tests
• newly authored test classes will just 'do the right thing' without devs having to remember to my @CircleCiSharding annotation

[sormuras]

...and also support ElementType.MODULE -- I can't find an open issue for supporting modules -- but I'm sure we discussed before.

[iamdanfox]

:+1: One more alternative would be to allow whitelisting the specific extension that would be automatically discovered:

-Djunit.jupiter.extensions.autodetection.whitelist=CircleCiShardingExtension,OtherExtension

This would eliminate my concern that extensions from deep in my dependency graph could suddenly start affecting my tests after a version upgrade (as the whitelist would be fully code-reviewable).

[sbrannen]

...and also support ElementType.MODULE -- I can't find an open issue for supporting modules -- but I'm sure we discussed before.

The team has debated allowing extension registration at the package level several times but never decided in favor of it. The same applies to registration at the module level.

The general consensus has been that it's "too magical" and non-intuitive for the average developer.

One of the challenges is that it's difficult to _navigate_ to the package level configuration. At least in Eclipse, you cannot control-click on a package name to navigate to it's package.info.java file, and if you hover over the package name with the mouse... you see the Javadoc but not the annotations for the package.

[sbrannen]

👍 One more alternative would be to allow whitelisting the specific extension that would be automatically discovered:

I'm actually in favor of that and have suggested it previously. I think that would be a useful feature in general.

[iamdanfox]

@sbrannen I'd be fine with either solution. In my assessment of the 'magicness' of the various solutions, I think the whitelisting system prop is marginally more magic:

• -Djunit.jupiter.extensions.autodetection.enabled=true: 9/10 magic - with this approach it's pretty much impossible to predict behaviour changes at code-review time, because just changing a version number of one of your deps may bring in a _transitive_ that has the right META-INF file.
• whitelisting extensions using a system property: 6/10 magic - the system property that enables extensions will be configured in your _build_ tool config (i.e. gradle files or maven xml), not in .java files. There's also a risk that people opening stuff in IDEs don't set this up correctly, resulting in different behaviour from the command line vs Eclipse/IntelliJ.
• package-info.java: 5/10 magic - IntelliJ navigates in the same way you describe as eclipse, but at least I just have to look in my src/main/java folder to see what extensions are configured
• test base inheritance: 2/10 magic - if I see a class that extends TestBase, I can't directly see what's going on, but it's just one click away.
• explicit annotations on all classes: 1/10 magic - the name of the extension is right in front of my face during code review

[sbrannen]

whitelisting extensions using a system property: 6/10 magic - the system property that enables extensions will be configured in your _build_ tool config (i.e. gradle files or maven xml), not in .java files. There's also a risk that people opening stuff in IDEs don't set this up correctly, resulting in different behaviour from the command line vs Eclipse/IntelliJ.

Although JUnit Platform configuration parameters _can_ be set via a JVM system property, I'd say that's a worst practice for a parameter like this one.

For this kind of config param, the best practice is to declare it in junit-platform.properties in the root of the classpath (i.e., src/test/resources).

That way it's in version control and honored in the build and by the IDE.

So for me, that's not very magical.

[sormuras]

[...] but at least I just have to look in my src/main/java folder to see what extensions are configured

That's why I like this solution.

Perhaps we may support both?

[iamdanfox]

Oh the junit-platform.properties sounds _much_ nicer than the -D system property approach, and it satisfies my concerns with the IDE thing too!

[marcphilipp]

@iamdanfox Would you mind changing the title and description to describe adding the new config parameter?

[sbrannen]

We should find some synergies between the proposal here and #1990.

[sormuras]

...I really missed the module-info.java support when trying to work-around #2194 -- which for the time being was resolved by attaching annotations on each test class.