Bazel: Option to set outputRoot to ex /tmp/bazel instead of ~/.cache/bazel

[4ZM]

I would like to set the outputRoot to somewhere outside the home directory (and that the location is specified on command line or in the .bazelrc.

I don't want to set the outputUserRoot or the outputBase, but the top level outputRoot. I still want the username and workspace hash directories under my new outputRoot directory.

Environment info

• Operating System:
  Linxu (Ubuntu)

• Bazel version (output of bazel info release):
  Tried without finding solution on 0.8.0

Have you found anything relevant by searching the web?

Ref bazel-discuss Google group question about this)

[iirina]

Assigning @lberki since he did some changes to this area.

[lberki]

What's "the top level outputRoot"? --output_user_root seems to be exactly what you want: that's where workspace hash directories and the files extracted from the Bazel binary go.

[4ZM]

According to https://docs.bazel.build/versions/master/output_directories.html:

the The outputRoot directory is ~/.cache/bazel

I want it to be /tmp/bazel in my environment (I still want different users to get different directories inside that directory).

[plopresti]

This would be a useful feature.

I am trying to install bazel on a multi-user system. Our users' home directories are on NFS and have limited space. I would like to configure bazel once for all users to put per-user state on a local file system.

Setting --output_user_root in the master bazelrc (/etc/bazel.bazelrc) does not work, because then all users will share an output user root. There is no mechanism, as far as I can tell, to use something like %user% or $USER in the master bazelrc.

I believe I either need such a mechanism or (more simply) an option for setting the top-level outputRoot.

[indrgun]

We are setting the output_user_root in .bazelrc local in the root workspace.
startup --output_user_root

However, in $HOME/.cache/bazel is still created and occasionally, it is filing up the quota for a person home directory causing bazel build to fail.

Is there a way to specify different .cache other than the home directory?

[dgrnbrg]

This is also an important feature for my use case--I need to specify a different path than $HOME/.cache/bazel, but I want each user to have their own unique path.

[plopresti]

Yeah it's kind of ridiculous there is no way to prevent this tool from littering $HOME.

In fact the only work-around, to my knowledge, is to set $HOME itself

[unicell]

How about changing outputRoot through $TEST_TMPDIR?

https://docs.bazel.build/versions/master/output_directories.html#documentation-of-the-current-bazel-output-directory-layout

[justincely]

changing $TEST_TMPDIR seems to have other implications than _just_ changing the output directory. setting $HOME did work for me though.

[adam-azarchs]

This is an issue for our team, due to a perhaps odd collision of circumstances:

1. Our team deploys bazel to an location which is mounted via nfs on the machines we develop on, to ensure we're all using the same version.
2. Our home directories are on nfs as well. Bazel does not play very nicely with having its output root on nfs (in particular, performance is atrocious due to the latency of metadata operations on nfs)
3. We've provisioned a local drive on the (shared) machine we usually build on to serve as a location for the bazel roots. This mount is exported via nfs read-only to the other machines.

This means we want to set the output root, but specifically to one machine and not specifically to one repository. Between that and the nfs-shared install location for bazel itself, the wrapper script solution becomes problematic. Changing $HOME is a non-starter for a variety of reasons - it would break way too many other things.

[plopresti]

@adam-azarchs

Well, you do not need to change $HOME globally.

Our situation is similar to yours. A simple "env HOME=/path/to/local/folder bazel ..." does the trick, at least for compiling Tensorflow. That's what we do.

It would still be nice to see this fixed, of course.

[adam-azarchs]

That might work for me personally but isn't something I can easily enforce for the rest of the team.

[plopresti]

@adam-azarchs

If you are the sysadmin, there is no such thing as "something you cannot easily enforce".

#!/bin/sh

export HOME=/path/to/local/storage/$USER

exec /actual/path/to/bazel "$@"

Name this script "bazel", make it executable, and put it in /usr/local/bin (or whatever your local custom is for everybody's PATH). Move the actual bazel binary to /actual/path/to/bazel outside of everybody's usual PATH.

Now anybody who runs "bazel whatever --with --some --arguments" will invoke Bazel with HOME set to a local folder.

Sure, this is a hack. But since this issue has been open for a year and a half, it is a reasonable bet that the developers simply do not care. Another option is to fix it yourself, open a pull request, and hope they take it.

[adam-azarchs]

I'm not the sysadmin and I don't have root. What I do control is the directory that everyone runs their toolchains out of, but it's an NFS mount that is shared on all machines. I did end up using a wrapper script, but it now needs to inspect the HOSTNAME and behave differently depending on which machine it runs on, which makes me feel dirty, and can't be maintained together with our other machine configuration scripts the way the machine-specific bazelrc is. Also, I am using --output_user_root instead of setting $HOME, because having HOME in an incorrect place breaks far too many other tools which end up running as child processes of bazel.

[luxe]

Is symlinking an option?
ln -s /tmp/bazel ~/.cache/bazel

[adam-azarchs]

That would require having access to people's home directories. For myself I have put a regular file in ~/.cache/bazel to catch mistakes in my wrapper script. The wrapper script to set --output_user_root is a functioning workaround for now, but it would be much cleaner if we could just set --output_root globally.

[TheButlah]

This would be really useful in certain CI scenarios. For example, I use bazel with gitlab, and gitlab can only pass things along to later stages if they are within the working directory of the job (symlinks will not be followed). Moving outputRoot would allow this to save several minutes of my CI times.