Bazel: cant build with bazel and ccache

[kreuzerkrieg]

Description of the problem / feature request:

Bazel fails to build tcmalloc with message

ccache: error: Failed to create temporary file for /home/user/.ccache/tmp/string_uti.stdout: Read-only file system

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I guess no special steps needed. ccache works fine, I'm able to build without using sudo with CMake, for example

What operating system are you running Bazel on?

Ubuntu 20.04

What's the output of bazel info release?

release 3.5.0

Have you found anything relevant by searching the web?

semi-related
https://github.com/bazelbuild/bazel/issues/1540
but I installed the Bazel using apt

Any other information, logs, or outputs that you want to share?

worked as expected once I ran Bazel under sudo, but I guess it is not the expected behavior
let me know if you need any additional information

[jiridanek]

My guess is that bazel sandboxing is blocking access. If that is the case, try building with --sandbox_writable_path=$HOME/.ccache

Your workaround using sudo suggests sandboxing may not be to blame, but the option may be worth trying anyways. If file permissions on your file system are to blame, then try sudo chown -R $USER:$USER $HOME/.ccache before running bazel. There may be files/directories owned by root user in your .ccache, which then cannot be written into by regular user.

[jiridanek]

It is usually useful to run bazel with -s --sandbox_debug --verbose_failures and include the output from this.

[kreuzerkrieg]

It is usually useful to run bazel with -s --sandbox_debug --verbose_failures and include the output from this.

sandbox_debug.txt

here you go

[kreuzerkrieg]

My guess is that bazel sandboxing is blocking access. If that is the case, try building with --sandbox_writable_path=$HOME/.ccache

Your workaround using sudo suggests sandboxing may not be to blame, but the option may be worth trying anyways. If file permissions on your file system are to blame, then try sudo chown -R $USER:$USER $HOME/.ccache before running bazel. There may be files/directories owned by root user in your .ccache, which then cannot be written into by regular user.

as of sudo chown -R $USER:$USER $HOME/.ccache cmake with gnumake or ninja works fine with this folder without any sudo, so I guess chown should make no difference, am I missing anything?

[jiridanek]

so I guess chown should make no difference, am I missing anything?

Your reasoning makes sense. I just personally prefer one bold try with chown over doing much up-front thinking about causes; I prefer to do that afterwards. That's all. edit: and if you mix running commands with sudo and without, its easy to create temp files owned by root in various places.

Sandboxing can be switched off, or you may try less powerfull sandbox, by doing either --spawn_strategy=standalone or --spawn_strategy=processwrapper-sandbox. This is what I'd certainly try, besides --sandbox_writable_path=$HOME/.ccache I previously suggested.

Previous instances of similar issues I was able to find were due to sandboxing

• https://stackoverflow.com/questions/49178997/bazel-failed-to-create-temporary-file
• https://stackoverflow.com/questions/52370202/how-to-get-bazel-ccache-and-sandboxing-to-work-together-ccache-read-only-file

edit: in any case, you've attached the verbose log. That should be enough information for somebody more knowledgeable about bazel to figure things out. You could maybe add the command you use to run bazel, and how you tell it to use ccache, as further information.

[jiridanek]

BTW, bazel has a background daemon. It is sometimes useful to kill it and restart it, because it might be remembering some now out-of-date state, with bazel shutdown. Probably not the case here, but its helpful to think of this when you want to try some steps again with a completely clean state.

[kreuzerkrieg]

sudo chown -R $USER:$USER ~/.ccache same result
bazel shutdown same result
bazel build --spawn_strategy=standalone //tcmalloc/... worked as expected

Dont get it, am I the only one who have a problem with sandboxing? it is quite fresh machine, 3 or 4 weeks, I remember having the same problem with previous machine with Ubuntu 18.04. Mystery...

[jiridanek]

In that case, what does using the sandbox writable path argument, that is --sandbox_writable_path=$HOME/.ccache do?

[kreuzerkrieg]

worked too

[jiridanek]

Awesome. That's expected behavior. Sandboxing limits the writable paths the build can write into. If you want ccache to work from inside a Bazel build with sandboxing turned on, you have to add $HOME/.ccache to the write allowlist. This is how the other Bazel+ccache issues I found turned out, too.

One thing I don't understand is why sudo also ended up, in effect, disabling the sandbox.

[kreuzerkrieg]

then it is odd the problem is not easily searchable on the net, since it sounds as something quite usual.
BTW, where is I'm whitelisting .ccache folder? I dont want to (remember and) add it to command line each time I run bazel

[jiridanek]

You can create config file $HOME/.bazelrc with the option, https://docs.bazel.build/versions/master/guide.html#bazelrc-syntax-and-semantics.

[susinmotion]

It looks like this is a documentation request at this point, to clarify that users should pass --sandbox_writeable_path to enable ccache with sandboxing. Does that sound right to you?

[kreuzerkrieg]

Whatever discoverable by google search will do, I guess

[blackliner]

I do have problems with that solution. If I add --sandbox_writable_path=/home/my_user/.ccache to the %workspace%/.bazelrc file, it works.
But since not every developer has ccache installed, I cannot blindly add that option to the repo. So I added it to the $HOME/.bazelrc, but it is not picked up by bazel... Any thoughts?

EDIT: sorry, not even adding it to the %workspace%/.bazelrc helps, it must be passed when running bazel test .... This changed within the last few bazel versions :-(

[jiridanek]

build --sandbox_writable_path=/home/user/.ccache

in bazel.rc should work. What does not work is using env variables, such as $HOME in .bazel.rc. https://github.com/bazelbuild/bazel/issues/10904. And env variables there never worked, as far as I remember.

[blackliner]

Thanks, I think I forgot the build...