Bazel: Feature Request: Provide a standard rule for JUnit testing

[carl-mastrangelo]

Description of the problem / feature request:

The java_test rule is really under-powered. By default, it can only run a single class, which forces users into one of four unpleasant scenarios:

1. Make a java_test rule per class. This leads to excessively large BUILD files, and the risk of accidentally forgetting to add a newly created test to the BUILD file. Example

2. Write a hacky Starlark rule to read in all the source .java files and generate a Suite class. This is repetitive, error prone, and can't be generally reused. It also depends on the source files, rather than working with a java_library. Example

3. Make a single Suite runner and manually add each class without the help of a rule. This combines the problems of both 1 and 2. Example

4. Make a fully featured Classpath scanning test runner and supporting code. This is the most amount of work, but the most likely to behave correctly. Bazel itself uses this with ClasspathSuite. However, this doesn't appear to be standard or reliable. It requires a AllTests.java file for each test directory. Lastly, it doesn't make sense for the Bazel community to have to replicate this amount of work just to run tests.

Therefore I am requesting a standardized TestRunner to be included with Bazel, which can scan the srcs and deps for classes annotated with @RunWith. (and possibly be scoped to a Java package.) At Google, there was an internal rule that did this, and it was a pleasure to use: junit4_test_suite. I think this would be an excellent starting point for making Bazel Java development easy.

I realize there are other testing frameworks, but JUnit is the biggest, and it would be a practical trade-off to give it a more first class status.

Feature requests: what underlying problem are you trying to solve with this feature?

Make it trivial to write JUnit tests

What's the output of bazel info release?

release 1.0.0

[davido]

Duplicate of: https://github.com/bazelbuild/bazel/issues/2539.

[carl-mastrangelo]

@davido Can you raise the priority of this somehow? This seems like an ongoing problem (all the way back to 2016), and it hasn't received much attention.

[davido]

I cannot raise the priority. I'm a regular Bazel user, like all others.

Having said that, Gerrit Code Review project that I also contribute to, extracted re-usable Bazel building blocks called bazlets. One of bazlets is junit_tests, located here: [1], so that you can say something like:

junit_tests(
    name = "gerrit-oauth-provider_tests",
    srcs = glob(["src/test/java/**/*.java"]),
    tags = ["oauth"],
    deps = [
        ":gerrit-oauth-provider__plugin_test_deps",
    ],
)

Others are maven_jar, javadoc, etc. So 100+ gerrit plugins are loading and using them. Example: [2]. Here where the bazlets are loaded: [3].

[1] https://github.com/GerritCodeReview/bazlets/blob/master/tools/junit.bzl
[2] https://github.com/davido/gerrit-oauth-provider/blob/master/BUILD#L26-L33
[3] https://github.com/davido/gerrit-oauth-provider/blob/master/WORKSPACE#L3-L8

[jin]

Closing as dupe of #2539.

[davido]

FWIW, there is also ready to use gen_java_tests macro available in https://github.com/google/bazel-common/blob/master/testing/test_defs.bzl.

And there is also a TODO in Dagger sources to switch to using that, in https://github.com/google/dagger/blob/master/test_defs.bzl#L22.