Bazel: `bazel test` cannot find the `MANIFEST` file

[Profpatsch]

With bazel 0.17rc1 on GNU/systemd/Linux 4.14:

Please take a moment to look at the repro here: https://github.com/Profpatsch/repros/tree/master/bazel-test-manifest-not-found

Repro Description:

Running it works as expected:

$ bazel run //:test-manifest-file
INFO: Analysed target //:test-manifest-file (0 packages loaded).
INFO: Found 1 target...
Target //:test-manifest-file up-to-date:
  bazel-bin/test-manifest-file
INFO: Elapsed time: 0.214s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:test-manifest-file
-----------------------------------------------------------------------------
Manifest:
__main__/test-manifest-file /home/philip/.cache/bazel/_bazel_philip/fed589667d20cb7d0606716b1ab88ded/execroot/__main__/bazel-out/k8-fastbuild/bin/test-manifest-file
__main__/test.sh /home/philip/code/repros/bazel-test-manifest-not-found/test.sh
YAY

But when actually running the target as a test:

$ bazel test //:test-manifest-file
INFO: Analysed target //:test-manifest-file (0 packages loaded).
INFO: Found 1 test target...
FAIL: //:test-manifest-file (see /home/philip/.cache/bazel/_bazel_philip/fed589667d20cb7d0606716b1ab88ded/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test-manifest-file/test.log)
Target //:test-manifest-file up-to-date:
  bazel-bin/test-manifest-file
INFO: Elapsed time: 0.248s, Critical Path: 0.09s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//:test-manifest-file                                                    FAILED in 0.1s
  /home/philip/.cache/bazel/_bazel_philip/fed589667d20cb7d0606716b1ab88ded/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test-manifest-file/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

$ cat /home/philip/.cache/bazel/_bazel_philip/fed589667d20cb7d0606716b1ab88ded/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test-manifest-file/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:test-manifest-file
-----------------------------------------------------------------------------
Manifest:
cat: /home/philip/.cache/bazel/_bazel_philip/fed589667d20cb7d0606716b1ab88ded/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/k8-fastbuild/bin/test-manifest-file.runfiles/MANIFEST: No such file or directory

Which means the environment variable is not set correctly inside of bazel test.

Discussion

As I understand it, MANIFEST files are the only way to reliably access runfiles at runtime in a cross-platform way. Thus, I’d like to be able to use manifest-based runtime discovery inside of bazel run as well as bazel test, and consider the above a bug.

This issue might be pointing out the same problem, but it neither mentions the discrepancy between run and test, nor does it point to a minimal reproduction.

Solutions until this bug is fixed

What’s the recommended way to route around this bug until it has been fixed? According to the other report, this line sets up the corresponding variable:
https://github.com/bazelbuild/bazel/blob/2a2aa7ab31367da47f8449059fdfcddfba3a20f1/tools/test/test-setup.sh#L106

cc @laszlocsomor

[laszlocsomor]

Sorry about the long silence! Yes, this looks like a bug; I'm looking for the reason for the discrepancy between test and run.

The bug is fixed at HEAD (02432e19306dae4cbab7db929e57516f2bba204a): Bazel is consistent for run and test because on Linux the RUNFILES_MANIFEST_FILE envvar is empty. Which is correct because there's no MANIFEST file under $RUNFILES_DIR.

Using the MANIFEST is not in fact the only way to reliably access runfiles -- Bazel has a built-in runfiles library that implements runfile lookup. Here's how to use it: https://github.com/bazelbuild/bazel/blob/02432e19306dae4cbab7db929e57516f2bba204a/tools/bash/runfiles/runfiles.bash#L15
(Unfortunately it's about 20 lines of boilerplate code that I haven't found a way to make shorter.)

[laszlocsomor]

https://github.com/bazelbuild/bazel/commit/0b3197987d47456c478c599fbc548bc669c71f0b was the patch that fixed this behavior in Bazel at HEAD, i.e. to only export RUNFILES_MANIFEST_FILE when the manifest exists.

[laszlocsomor]

With Bazel 0.18.0, in https://github.com/Profpatsch/repros/tree/master/bazel-test-manifest-not-found, at https://github.com/Profpatsch/repros/commit/db2db2396d5c980bae1506f6d6105b19952a17f5, the following happens on Linux:

• RUNFILES_MANIFEST_FILE is unset, both with bazel test and bazel run
• RUNFILES_DIR is set, both with bazel test and bazel run

On Windows, also with 0.18.0:

• bazel test succeeds, because RUNFILES_MANIFEST_FILE is set
• bazel run prints the manifest's contents and "YAY"

[Profpatsch]

Is there any reason we don’t use MANIFEST-based discover on Linux as well? As far as I can see that approach is superior in every way.

[laszlocsomor]

There's one caveat of the manifest-based approach: the binary must be aware of it and use the Runfiles Library, and cannot expect a tree of runfiles.

With the directory-based approach, the binary can expect that its runfiles are under $RUNFILES_DIR (or $TEST_SRCDIR in case of tests). This is the original runfiles behavior, and how the Google-internal version of Bazel works.

[laszlocsomor]

So long as the binary is aware it needing to use the runfiles library, I agree that the manifest-based discovery is better because it does not require creating symlinks.

[laszlocsomor]

I just thought of another advantage of supporting runfiles trees: it's the natural approach for deploying binaries -- create the tree of runfiles somewhere on disk, set RUNFILES_DIR to its path, then run the binary.

Of course you can do the same with a manifest file -- just add to the previous steps the writing of a manifest and the setting of RUNFILES_MANIFEST_FILE -- but it just requires a bit more work.

In other words, both approaches have pros and cons, and for now both are supported.

FYI, you can disable generating runfiles trees to speed up builds using the --[no]experimental_enable_runfiles flag. See: https://github.com/bazelbuild/bazel/blob/a6deb76cd3d86e9ca51cdbdb2f93f8dfbbace1cd/src/main/java/com/google/devtools/build/lib/analysis/config/BuildConfiguration.java#L926