Bazel: Exception running bazel coverage on java tests

[robfig]

Description of the problem / feature request:

"bazel coverage" produces an exception internal to bazel

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel coverage --instrumentation_filter=//src/com/corp/profiles //test/com/corp/profiles:unit
...........
INFO: Analysed target //test/com/corp/profiles:unit (171 packages loaded).
INFO: Found 1 test target...
ERROR: /Users/robfig/alpha/src/com/corp/profileservice/validation2/error/BUILD:3:1: Building src/com/corp/profileservice/validation2/error/liberror.jar (3 source files) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -XX:+TieredCompilation '-XX:TieredStopAtLevel=1' -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9-dev-r4023-3.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/profileservice/validation2/error/liberror.jar-2.params)
java.nio.file.DirectoryNotEmptyException: bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.postVisitDirectory(JacocoInstrumentationProcessor.java:153)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.postVisitDirectory(JacocoInstrumentationProcessor.java:143)
    at java.nio.file.Files.walkFileTree(Files.java:2688)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.recursiveRemove(JacocoInstrumentationProcessor.java:141)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:79)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:140)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //test/com/corp/profiles:unit failed to build
INFO: Elapsed time: 111.911s, Critical Path: 39.00s
FAILED: Build did NOT complete successfully

Executed 0 out of 1 test: 1 fails to build.

$ bazel test //test/com/corp/profiles:unit
INFO: Analysed target //test/com/corp/profiles:unit (0 packages loaded).
INFO: Found 1 test target...
Target //test/com/corp/profiles:unit up-to-date:
  bazel-bin/test/com/corp/profiles/unit.jar
  bazel-bin/test/com/corp/profiles/unit
INFO: Elapsed time: 26.749s, Critical Path: 22.90s
INFO: Build completed successfully, 139 total actions
//test/com/corp/profiles:unit                                            PASSED in 5.5s

Executed 1 out of 1 test: 1 test passes.

The directory being complained about does not exist under output_path

$ bazel info
output_path: /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out

$ ls /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion
ls: /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/coverage-metadata/com/corp/profileservice/completion: No such file or directory

Feature requests: what underlying problem are you trying to solve with this feature?

Run java tests with coverage enabled

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I don't have an open source reproducer. I was hoping the stack trace would be enough

What operating system are you running Bazel on?

Mac OS High Sierra

What's the output of bazel info release?

release 0.9.0

Have you found anything relevant by searching the web?

This appears to be the main discussion about coverage, but it seems to indicate that java coverage is expected to be in working order
https://github.com/bazelbuild/bazel/issues/1118

[robfig]

Testing it on another package, I see a different exception

java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline

Full output:

$ bazel clean --expunge
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel coverage --instrumentation_filter=//src/com/corp/util/function //test/com/corp/util/function/...

.
INFO: Analysed 2 targets (20 packages loaded).
INFO: Found 1 target and 1 test target...
FAIL: //test/com/corp/util/function:unit (see /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/test.log)
INFO: From Testing //test/com/corp/util/function:unit:
==================== Test output for //test/com/corp/util/function:unit:
+ [[ -z bazel-out/darwin-fastbuild/bin/external/bazel_tools/tools/test/LcovMerger ]]
+ COVERAGE_LEGACY_MODE=
+ [[ -z bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files ]]
+ ROOT=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__
+ [[ bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files != /* ]]
+ export COVERAGE_MANIFEST=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files
+ COVERAGE_MANIFEST=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.instrumented_files
+ export COVERAGE_DIR=_coverage/test/com/corp/util/function/unit/test
+ COVERAGE_DIR=_coverage/test/com/corp/util/function/unit/test
+ [[ _coverage/test/com/corp/util/function/unit/test == /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__* ]]
+ COVERAGE_DIR=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test
+ mkdir -p /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test
+ COVERAGE_OUTPUT_FILE=bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ [[ bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat == /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__* ]]
+ COVERAGE_OUTPUT_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ export JAVA_COVERAGE_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test/jvcov.dat
+ JAVA_COVERAGE_FILE=/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/_coverage/test/com/corp/util/function/unit/test/jvcov.dat
+ export COVERAGE=1
+ COVERAGE=1
+ export BULK_COVERAGE_RUN=1
+ BULK_COVERAGE_RUN=1
+ [[ ! -n '' ]]
+ for name in '"$LCOV_MERGER"'
+ [[ ! -e bazel-out/darwin-fastbuild/bin/external/bazel_tools/tools/test/LcovMerger ]]
+ [[ -n '' ]]
+ cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__
+ /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
JUnit4 Test Runner
.E.E.E.E.E.E
Time: 0.07
There were 6 failures:
1) test_mapping_nonEmpty(com.corp.util.function.OptionalBooleanTest$ExplicitTests)
java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline
    at com.corp.util.function.OptionalBoolean.$jacocoInit(OptionalBoolean.java)
    at com.corp.util.function.OptionalBoolean.coerce(OptionalBoolean.java)
    at com.corp.util.function.OptionalBooleanTest$ExplicitTests.test_mapping_nonEmpty(OptionalBooleanTest.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:89)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
    at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:112)
    at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:144)
    at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:82)
Caused by: java.lang.ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 42 more

(MORE SIMILAR FAILURES HERE)

FAILURES!!!
Tests run: 6,  Failures: 6


BazelTestRunner exiting with a return value of 1
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2018-01-11 15:41:08 --

+ TEST_STATUS=1
+ touch /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
+ [[ 1 -ne 0 ]]
+ echo --
--
+ echo Coverage runner: Not collecting coverage for failed test.
Coverage runner: Not collecting coverage for failed test.
+ echo The following commands failed with status 1
The following commands failed with status 1
+ echo /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
/private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/bazel-sandbox/3966943955256125145/execroot/__main__/bazel-out/darwin-fastbuild/bin/test/com/corp/util/function/unit.runfiles/__main__/test/com/corp/util/function/unit
+ exit 1
================================================================================
INFO: Elapsed time: 14.977s, Critical Path: 2.30s
INFO: Build completed, 1 test FAILED, 34 total actions
//test/com/corp/util/function:unit                                       FAILED in 0.6s
  /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/test.log

Executed 1 out of 1 test: 1 fails locally.

$ bazel test --instrumentation_filter=//src/com/corp/util/function //test/com/corp/util/function/...
INFO: Analysed 2 targets (0 packages loaded).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 1.166s, Critical Path: 0.85s
INFO: Build completed successfully, 9 total actions
//test/com/corp/util/function:unit                                       PASSED in 0.5s

Executed 1 out of 1 test: 1 test passes.

[robfig]

The first exception does not occur if I pass "--jobs 1", so presumably this is a bug in the interaction with concurrent workers. If you have any guidance or links as to how to debug / develop a fix for it I'd be happy to try to put together a pull request. Looking at the second one..

[robfig]

Any feedback on this?

[robfig]

I tried with bazel 0.10.0 and these errors still occur in the same way

[robfig]

Running with --experimental_java_coverage solves the NoClassDefFoundError but not the java.nio.file.DirectoryNotEmptyException. At least this means I can run tests with --jobs 1 and have them succeed. Wonderful.

[robfig]

(Looking at it again, I figured out that the NoClassDefFoundError was caused by the java_test depending on a java_binary target (accidentally) instead of java_library. It began working after correcting that.)

[iirina]

Are there any more coverage errors you get now with --experimental_java_coverage?

[robfig]

Yes, the concurrency-related error still occurs as before when I run coverage on the ~25 test suite targets in our util directory. It does not occur with --jobs=1. Is that sufficient to diagnose?

[iirina]

Can you tell me more about your setup? How are your tests/libraries structured?

[robfig]

I have about 30 test packages under util, with a representative illustration of one in
https://github.com/robfig/bazel_coverage_repro

and I'm running them like

$ bazel coverage --experimental_java_coverage
   --instrumentation_filter=//src/com/corp/util
   test/com/corp/util/...

which passes (as expected) because it's only one test. I believe if I created 30 (or at least a number of) different packages it would reproduce.

Here is a latest example of failure, where one failed to build, and the rest were skipped:

ERROR: /Users/robfig/alpha/src/com/corp/util/lock/BUILD:3:1: Building src/com/corp/util/lock/liblock.jar (6 source files) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__ && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9-dev-r4023-3.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/util/lock/liblock.jar-2.params)
java.nio.file.NoSuchFileException: bazel-out/darwin-fastbuild/coverage-metadata/com/corp/util/time/Timestamps.class.uninstrumented
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1126)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.visitFile(JacocoInstrumentationProcessor.java:147)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$2.visitFile(JacocoInstrumentationProcessor.java:143)
    at java.nio.file.Files.walkFileTree(Files.java:2670)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.recursiveRemove(JacocoInstrumentationProcessor.java:141)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:79)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:157)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
INFO: Elapsed time: 77.067s, Critical Path: 46.09s
FAILED: Build did NOT complete successfully
//test/com/corp/util/function:unit                              (cached) PASSED in 4.5s
  /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/test/com/corp/util/function/unit/coverage.dat
//test/com/corp/util/addressutil:addressutil                          NO STATUS
//test/com/corp/util/appspartners:integration                         NO STATUS
//test/com/corp/util/base:unit                                        NO STATUS
//test/com/corp/util/categories:unit                                  NO STATUS
//test/com/corp/util/crypto:unit                                      NO STATUS
//test/com/corp/util/csv:unit                                         NO STATUS
//test/com/corp/util/elasticsearch:unit                               NO STATUS
//test/com/corp/util/flags:unit                                       NO STATUS
//test/com/corp/util/fuse:unit                                        NO STATUS
//test/com/corp/util/geo/country:unit                                 NO STATUS
//test/com/corp/util/geo/geocoding:unit                               NO STATUS
//test/com/corp/util/geo/geocoding/providers:unit                     NO STATUS
//test/com/corp/util/geo/ip:unit                                      NO STATUS
//test/com/corp/util/hash:unit                                        NO STATUS
//test/com/corp/util/http:unit                                        NO STATUS
//test/com/corp/util/i18n:i18n                                        NO STATUS
//test/com/corp/util/objectcache:unit                                 NO STATUS
//test/com/corp/util/protobuf:unit                                    NO STATUS
//test/com/corp/util/retry:unit                                       NO STATUS
//test/com/corp/util/serialization/formenc:unit                       NO STATUS
//test/com/corp/util/serialization/js:unit                            NO STATUS
//test/com/corp/util/serialization/xml:unit                           NO STATUS
//test/com/corp/util/sql:unit                                         NO STATUS
//test/com/corp/util/subscriptionfeatures:integration_bazel           NO STATUS
//test/com/corp/util/text:unit                                        NO STATUS
//test/com/corp/util/time:unit                                        NO STATUS
//test/com/corp/util/uaparser:uaparser_unit                           NO STATUS
//test/com/corp/util/youtube:unit                                     NO STATUS

Executed 0 out of 30 tests: 1 test passes, 1 fails to build and 28 were skipped.

Based on this error:

• It was building src/com/corp/util/lock
• As part of that, it was deleting the bazel-out/darwin-fastbuild/coverage-metadata directory
• com/corp/util/time/Timestamps.uninstrumented was under that directory, and it was deleted by a different worker between the time the file system walker listed the directory and the time it tried to delete that file.
• Note that com.corp.util.time IS NOT a dependency of com.corp.util.lock, so it's clearly incorrect for the worker doing coverage of util.lock to delete files used by the util.time worker.

This is supported by the fact that running the coverage using --jobs=1 causes it to work.

Does that seem plausible? Or is it something in our setup / bazel.rc that is causing the test suites to get mixed up?

[iirina]

Yes, it's totally plausible. I'll try to reproduce. Thanks for the info!

[iirina]

I managed to get a couple of errors that seem related with the bazel java tests. Both commands are successful if used with --jobs=1.

bazel coverage src/test/java/com/google/devtools/build/lib:java-rules-tests

ERROR: /Users/elenairina/bazel/src/main/java/com/google/devtools/build/lib/vfs/BUILD:34:1: Building src/main/java/com/google/devtools/build/lib/vfs/libvfs.jar (26 source files) and running annotation processors (AutoCodecProcessor, OptionProcessor) failed (Exit 1)
java.nio.file.NoSuchFileException: bazel-out/darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/vfs/_javac/vfs/libvfs_classes/com/google/devtools/build/lib/vfs/Root$PathRoot.class -> bazel-out/darwin-fastbuild/coverage-metadata/com/google/devtools/build/lib/vfs/Root$PathRoot.class
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:127)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor$1.visitFile(JacocoInstrumentationProcessor.java:104)
    at java.nio.file.Files.walkFileTree(Files.java:2670)
    at java.nio.file.Files.walkFileTree(Files.java:2742)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.instrumentRecursively(JacocoInstrumentationProcessor.java:102)
    at com.google.devtools.build.buildjar.instrumentation.JacocoInstrumentationProcessor.processRequest(JacocoInstrumentationProcessor.java:89)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:157)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //src/test/java/com/google/devtools/build/lib:java-rules-tests failed to build

bazel coverage --experimental_java_coverage src/test/java/com/google/devtools/build/lib:java-rules-tests :

ERROR: /Users/elenairina/bazel/src/main/java/com/google/devtools/build/lib/windows/jni/BUILD:49:1: Building src/main/java/com/google/devtools/build/lib/windows/jni/libprocesses.jar (1 source file) failed (Exit 1)
java.io.FileNotFoundException: /private/var/tmp/_bazel_elenairina/f2f47067c84d3ffa2025ad2073174c18/execroot/io_bazel/bazel-out/darwin-fastbuild/coverage-metadata/com/google/devtools/build/lib/windows (No such file or directory)
    at com.google.devtools.build.buildjar.jarhelper.JarHelper.copyEntry(JarHelper.java:176)
    at com.google.devtools.build.buildjar.jarhelper.JarCreator.execute(JarCreator.java:234)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.buildJar(SimpleJavaLibraryBuilder.java:160)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:134)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Target //src/test/java/com/google/devtools/build/lib:java-rules-tests failed to build

[robfig]

Can I ask why this is not a higher priority? It seems like sort of a big issue in running coverage, and I was under the impression bazel considered java coverage to be production ready. For example, we have to run all of our coverage with j=1, which is a LOT slower than when running the equivalent bazel test at full parallelism. This results in none of the teams wanting to use it.

Or, is there a workaround that I'm not seeing? How do others run coverage without specifying j=1?

[dhalperi]

Still happening on 0.13.0

[robfig]

Still happening on 0.14.0

@iirina , since I'm not sure you saw my previous question, can you advise why it's a low priority when it seems like a major blocker for calling java coverage "production ready"? How do folks in the bazel project run coverage? I can't imagine you pass --jobs 1

[iirina]

Hi Rob,

I'm sorry for the silence and the terrible state of coverage. A bit of coverage context: Bazel rolled-out coverage in a poor, minimalist way and unfortunately it was not production ready (see #1118). This quarter we had the resources to prioritize working on coverage and we now have a design doc for improving it. Also unfortunately, Bazel doesn't run coverage for itself ATM. That will change (see one of the requirements in the doc). All in all, the situation is bad and we are now trying to make it better.

I'm working on a fix for this issue right now. Given that the first RC was already cut for Bazel 0.15.0 already started, the fix should be in 0.16.0.

[robfig]

Hi Irina,
Thanks for the response! The design doc looks very good, and I'm excited at the progress. This particular issue preventing us from being able to reliably run bazel coverage is the main pain point in the current situation, and we can migrate to the new format when it's ready. I'm very pleased to hear that a fix is being worked on, and I'll probably cut a personal release as soon as the fix goes in.
Many thanks,
Rob

[iirina]

@robfig The fix is now in. It would be great if you could cut a RC that includes 4001dddc38b07fe0ee86f6a9e95ff4a92276bb2d and see if the issue still persists. I tested it with bazel coverage src/test/java/com/google/devtools/build/lib:java-rules-tests and didn't get breakages anymore.

[robfig]

@iirina The concurrency-related error has indeed gone away! That's a big step and will fix the vast majority of our issues. Thank you very much!

I do still see the other error from the original report. It seems to be triggered by instrumentation of an error-prone check.

Here's the scenario:

# //src/com/corp/util/i18n/BUILD
java_library(
    name = "i18n",
    srcs = glob(["*.java"]),
    exported_plugins = [
        "//src/com/corp/util/i18n/bugpatterns",
    ],
   ...  

and

# //src/com/corp/util/i18n/bugpatterns/BUILD
java_plugin(
    name = "bugpatterns",
    srcs = glob(["*.java"]),
   ...  

Here's the full error:

$ ../bazel/bazel-bin/src/bazel coverage test/com/corp/util/...

ERROR: /Users/robfig/alpha/src/com/corp/util/callable/BUILD:3:1: Building src/com/corp/util/callable/libcallable.jar (1 source file) failed (Exit 1): java failed: error executing command
  (cd /private/var/tmp/_bazel_robfig/1db08896ecff7e5e96a745981c3b77e6/execroot/corp && \
  exec env - \
    LC_CTYPE=en_US.UTF-8 \
  external/local_jdk/bin/java -Xbootclasspath/p:external/bazel_tools/third_party/java/jdk/langtools/javac-9+181-r4173-1.jar -jar external/bazel_tools/tools/jdk/JavaBuilder_deploy.jar @bazel-out/darwin-fastbuild/bin/src/com/corp/util/callable/libcallable.jar-2.params)
An exception has occurred in the compiler ((version info not available)). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.util.ServiceConfigurationError: com.google.errorprone.bugpatterns.BugChecker: Provider com.corp.util.i18n.bugpatterns.I18nNonConstant could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
    at com.google.errorprone.scanner.ScannerSupplier.fromBugCheckerClasses(ScannerSupplier.java:71)
    at com.google.errorprone.ErrorPronePlugins.loadPlugins(ErrorPronePlugins.java:47)
    at com.google.errorprone.ErrorProneAnalyzer.lambda$scansPlugins$0(ErrorProneAnalyzer.java:76)
    at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:164)
    at com.google.errorprone.ErrorProneAnalyzer.finished(ErrorProneAnalyzer.java:152)
    at com.google.devtools.build.buildjar.javac.plugins.errorprone.ErrorPronePlugin.postFlow(ErrorPronePlugin.java:110)
    at com.google.devtools.build.buildjar.javac.BlazeJavaCompiler.flow(BlazeJavaCompiler.java:112)
    at com.sun.tools.javac.main.JavaCompiler.flow(JavaCompiler.java:1363)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:959)
    at com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:100)
    at com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:142)
    at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:96)
    at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:90)
    at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:110)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder$2.invokeJavac(SimpleJavaLibraryBuilder.java:121)
    at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:54)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:124)
    at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:132)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.processRequest(BazelJavaBuilder.java:105)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.runPersistentWorker(BazelJavaBuilder.java:67)
    at com.google.devtools.build.buildjar.BazelJavaBuilder.main(BazelJavaBuilder.java:45)
Caused by: java.lang.NoClassDefFoundError: org/jacoco/agent/rt/internal_b0d6a23/Offline
    at com.corp.util.i18n.bugpatterns.I18nNonConstant.$jacocoInit(I18nNonConstant.java)
    at com.corp.util.i18n.bugpatterns.I18nNonConstant.<clinit>(I18nNonConstant.java)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 24 more
Caused by: java.lang.ClassNotFoundException: org.jacoco.agent.rt.internal_b0d6a23.Offline
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 32 more
INFO: Elapsed time: 18.145s, Critical Path: 8.44s
INFO: 14 processes: 10 darwin-sandbox, 1 local, 3 worker.
FAILED: Build did NOT complete successfully

It seems like coverage instrumentation should ignore java_plugin targets?

[iirina]

@iirina The concurrency-related error has indeed gone away! That's a big step and will the vast majority of our issues. Thank you very much!

Yay!

I do still see the other error from the original report. It seems to be triggered by instrumentation of an error-prone check.

Can you open a new issue for this bug and cc me on it? In the meantime could you also try running coverage with --experimental_java_coverage and tell me if you get the same error? Thanks!

[robfig]

Sadly, I do get the same error. Opened #5426, thank you!