Bazel: Google Cloud Storage Cache Strategy

[pspeter3]

Description of the problem / feature request / question:

I know that Bazel is using Hazelcast as the cache engine but it would be nice to be able to support other cache engines, especially for CI environments. In this case I would like to be able to use a Google Cloud Storage bucket.

Anything else, information or logs or outputs that would be helpful?

I'd be willing to try and hack on this myself given some pointers on where to start and the best way to enable this.

[kchodorow]

@philwo, could you give an update on where we are with this?

[pspeter3]

Thanks @kchodorow for introducing! I think this would make CI builds substantially faster for our team.

[philwo]

@pspeter3 I think your idea is interesting. I'd be happy to do code review and/or give some guidance if you want to give implementing it a try.

The main problem currently is that Bazel has no good mechanism for consulting a cache prior to actually executing an action. @ulfjack and me are working on refactoring Bazel's execution pipeline to make it much easier to plug new mechanisms in. It might still take us 2-3 months, until it's all done, though.

There's also a lot of work going on to create a combined remote execution / caching strategy for Bazel (in this package: https://github.com/bazelbuild/bazel/commits/master/src/main/java/com/google/devtools/build/lib/remote) - I'll be honest, most of our own effort will for now be spend there, as we want to finish and polish one remote strategy, before adding support for even more different ones.

However, this shouldn't stop you from trying something, if you feel like hacking on Bazel's code. :)

Here are some pointers and a small "getting started" explanation to how this could work:

When Bazel builds something, it basically executes actions along a dependency graph, where each action has a defined set of input files and output files. Most of the actions that Bazel runs are so called SpawnActions. They're called like this, because the action involves executing an external program (the "Spawn", defined via its command-line, working directory and a few other attributes) that will produce the required outputs from the inputs. For example, this could be a C++ compiler that compiles *.c files to *.o files, but it could also be the "zip" utility that produces an output.zip from a number of input files.

Now, Bazel has a number of "strategies" (classes implementing the SpawnActionContext interface) that it can use to execute a SpawnAction - for example, in the simplest case, it could simply run the Spawn by executing the command-line that makes it up (e.g. "gcc -o hello hello.c") - this one is called the StandaloneSpawnStrategy (https://github.com/bazelbuild/bazel/blob/e6eca1f6c779eb04f776a27082da0a708b234e18/src/main/java/com/google/devtools/build/lib/standalone/StandaloneSpawnStrategy.java). Or it could create a sandbox directory somewhere, then execute the Spawn inside and move the output files out. Or it could send all the input files + the command-line to a remote system (compile farm), execute it there and download the output files. The only thing that Bazel cares about after it used a SpawnActionContext to execute a SpawnAction is that the output files are there. How you get them is up to you.

Which strategy it uses is defined by the --spawn_strategy= flag (https://github.com/bazelbuild/bazel/blob/e6eca1f6c779eb04f776a27082da0a708b234e18/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRulesModule.java#L51). If you add a new SpawnActionContext, you'd just give it a name (like "cloudcache") and run "bazel build --spawn_strategy=cloudcache //my:target".

And this is where you could hook in with your caching mechanism: If you implement a SpawnActionContext that in its execute method calculates a hash value of the SpawnAction that it is supposed to run, tries to download the results from Google Cloud Storage, and if that succeeds, you just put the output files where they're supposed to go. If you get a cache miss, you'd have to actually execute the action (maybe delegate to StandaloneSpawnStrategy or LinuxSandboxedStrategy) and when that succeeded, upload the results.

If you have any questions, feel free to write to https://groups.google.com/forum/#!forum/bazel-discuss or reply here.

[pspeter3]

@philwo thank you for the explanation!

[dslomov]

@philwo could you triage this request? What is the right component / priority? Thanks!

[hhclam]

I would advise against implement a separate spawn strategy for caching purposes. A spawn strategy is heavy-weight approach like Philipp said. Computing a hash in spawn strategy just before execution is far from efficient. It will be better if the cache is consulted much earlier.

Bazel can talk to a Hazelcast memory cluster for caching. I intend to remove this path in favor of REST interface (See https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/RemoteOptions.java#L23) for simple caching needs and gRPC API for optimized performance and use in conjunction with remote execution. You can also implement the gRPC API for caching if this is what you choose, see this thread for details: https://groups.google.com/d/msgid/bazel-discuss/3ea578a9-83c5-43f9-8f92-9ba5ef98bd6f%40googlegroups.com

[pspeter3]

This sounds substantially easier and like I do not need to modify Bazel. When do you expect this work to land?

[hhclam]

The --rest_cache_url feature is already in Bazel 0.4.4. The documentation for using it and better support for nginx and apache will land soon.

You can try with Bazel 0.4.4 today:

bazel --host_jvm_args=-Dbazel.DigestFunction=SHA1 build --spawn_strategy=remote --rest_cache_url=http://server-address:port/cache

Change http://server-address:port/cache to a REST enpoint that supports GET, PUT and HEAD and returns 200 on success.

[pspeter3]

Thanks @hhclam!

[Steve-Munday]

According to the official docs, this feature is supported now. Likely this issue can be closed?

[philwo]

@Steve-Munday Yep, I think so! Thanks for pointing it out.

I'll let @buchgr have the honor of closing this issue. :)

[buchgr]

oh wow, I didn't realize we had a bug for that :-). CLOSED!