Bazel: sandboxfs fails to mount _tmp for tests

[kornholi]

With the following repro:

$ cat WORKSPACE
$ cat test.sh
$ chmod +x test.sh
$ cat BUILD
sh_test(
    name="sh_test",
    srcs=["test.sh"]
)

bazel test fails with sandboxfs enabled:

$ bazel test --experimental_use_sandboxfs --test_output=all --sandbox_debug :sh_test
INFO: Invocation ID: cb555d4f-dd5b-4cdd-b988-542b80b84676
INFO: Analysed target //:sh_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
INFO: Mounting sandboxfs instance on /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/sandboxfs
FAIL: //:sh_test (see /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/execroot/__main__/bazel-out/k8-fastbuild/testlogs/sh_test/test.log)
INFO: From Testing //:sh_test:
==================== Test output for //:sh_test:
src/main/tools/linux-sandbox.cc:154: linux-sandbox-pid1 has PID 26510
src/main/tools/linux-sandbox-pid1.cc:175: working dir: /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/sandboxfs/10
src/main/tools/linux-sandbox-pid1.cc:194: writable: /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/linux-sandbox/1/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:194: writable: /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/linux-sandbox/1/execroot/__main__/_tmp/8d9cc406864e2af179464ec538959378
src/main/tools/linux-sandbox-pid1.cc:198: "mount(/home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/linux-sandbox/1/execroot/__main__/_tmp/8d9cc406864e2af179464ec538959378, /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/linux-sandbox/1/execroot/__main__/_tmp/8d9cc406864e2af179464ec538959378, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory
src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 1
================================================================================
Target //:sh_test up-to-date:
  bazel-bin/sh_test
INFO: Build complete; unmounting sandboxfs...
INFO: Elapsed time: 0.167s, Critical Path: 0.06s
INFO: 1 process: 1 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//:sh_test                                                               FAILED in 0.1s
  /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/execroot/__main__/bazel-out/k8-fastbuild/testlogs/sh_test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

The _tmp directory exists in execroot:

$ ls -la /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/execroot/__main__/_tmp/8d9cc406864e2af179464ec538959378
total 8
drwxrwxr-x 2 kornholi kornholi 4096 Feb 19 20:46 .
drwxrwxr-x 5 kornholi kornholi 4096 Feb 19 20:46 ..

And bazel run runs successfully (although that does not run sandboxed anymore?):

$ bazel run --experimental_use_sandboxfs --test_output=all --sandbox_debug :sh_test
INFO: Invocation ID: 724216b4-ddc9-4f45-95be-f422429d0f68
INFO: Analysed target //:sh_test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: Mounting sandboxfs instance on /home/kornholi/.cache/bazel/_bazel_kornholi/99e6b8dc7d2e3c8f8ca2272e8162fc83/sandbox/sandboxfs
Target //:sh_test up-to-date:
  bazel-bin/sh_test
INFO: Build complete; unmounting sandboxfs...
INFO: Elapsed time: 0.100s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:sh_test
-----------------------------------------------------------------------------

This seems to only occur with bazel test, as building a pretty complex target caused no errors with sandboxfs.

@jmmv any ideas if this is a Bazel or a sandboxfs issue?

What operating system are you running Bazel on?

Ubuntu 18.04 + Bazel 0.22.0 + sandboxfs 0.1.0

[jmmv]

I think this is a Bazel issue. Note that the failure paths do not have /sandboxfs/ in them so they are outside of the mount point. Will need to dig a bit to understand what's happening.

[Globegitter]

@jmmv Any ideas on this? We are running into symlink issues (see https://github.com/facebook/jest/issues/5356) with the testing framework jest (https://jestjs.io/) and while we have patches to fix this on our end and I am planning to submit a fix to jest directly I would also be curios to see if just running tests with sandboxfs fixes the issues (which I am expecting it would).

[rohansingh]

@Globegitter Slightly unrelated, but could you describe the patches you have to fix the issue for Jest? I'm having the same issue with it and my workaround (dereferencing all symlinks) is not at all performant.

[Globegitter]

@rohansingh you can see some discussion on this here: https://github.com/ecosia/bazel_rules_nodejs_contrib/issues/4. If you find me on the bazelbuild Slack I can also send you our latest set of patches for 24.7.1 as well as some further ones we needed for coverage support.

[jmmv]

Confirmed that this is still a problem with the newest sandboxfs release (0.2.0) and latest Bazel. I'm guessing... I never tried running tests on sandboxfs, so there is probably some stupid lingering bug in there...

[al-tu]

hello. any progress on this issue? It's actually a blocker, would have switched to a sandboxed strategy otherwise.
Using: sandboxfs ver 0.2.0, bazel ver 3.4.1, testing with gtest

[Mathiasdm]

Note sure if it helps narrow this down, but: we run Bazel within a docker container with CentOS 7.
If our host machine is also CentOS 7, we don't see this issue. If our host machine is Arch, we do see this issue.

It might be related to the kernel version?
Edit: note that on CentOS 7, without sandboxfs, we use processwrapper-sandbox. On Arch, it looks like linux-sandbox is used.

[Mathiasdm]

The issue also occurs on CentOS 8 (and I assume on any Linux machine that uses linux-sandbox instead of processwrapper-sandbox), blocking our upgrade.

[tcm-marcel]

We run into the same issue described above (on Ubuntu) and figured out it only happens when using the linux-sandbox execution strategy.

Therefore, our workaround is to execute all test actions with the slightly less restrictive processwrapper-sandbox. This can be achieved by this .bazelrc entry:

test:linux --strategy TestRunner=processwrapper-sandbox

This has the advantage, that any compilation actions belonging to tests are still executed with linux-sandbox (if possible) and only the actual testing action uses the other execution strategy.