Bazel: Have a generic, easy-to-setup distributed caching

[damienmg]

Build up on Alpha Lam's initial version to have a viable version for all users of Bazel.

@philwo is synchronizing all the efforts in that perspective.

[tfarina]

@hhclam fyi

[ittaiz]

Hi,
What's the status of this issue?

[philwo]

@ittaiz We have prototype implementations of distributed caching and remote execution in Bazel now:

Distributed caching: https://github.com/bazelbuild/bazel/commit/79adf59e2973754c8c0415fcab45cd58c7c34697
Remote execution: https://github.com/bazelbuild/bazel/commit/a1a79cb0a1880e825618c6440219680f27faa0b7

Here is some documentation: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/README.md

We will continue to improve this and welcome feedback :) Please file bugs and assign to @hhclam if you encounter issues with these features.

[ittaiz]

got you.
Couple of questions:

1. Are there any current thoughts on improvements or waiting for feedback? Asking since I saw some thoughts in the google group discussion
2. Can we have the readme's contents incorporated to the bazel.io site? I looked for it there and couldn't find it. Also maybe update the roadmap that this is done. People just skimming the roadmap might not know it's there (I knew something is there by deep diving into the code).
3. If I have thoughts they should be written here? Should they start in bazel-discuss and "graduate" here?

Thanks!

[damienmg]

It is still experimental so I think we need to at least discuss on the future of the Bazel interface before documenting it (if we going to change it a lot, it is better to not put too much documentation for now).

[ittaiz]

ok. I think when people evaluate bazel vs pants vs buck the distributed caching and execution plays a big part.
Having a note saying it's already there in as a prototype and can be used by early adopters, willing to be broken, can encourage people. I'm such a user.

[damienmg]

Yeah that sounds reasonable. We should at least do a blog post with basic
documentation of both feature. But we need to sync with Alpha who did
almost all the work.

On Tue, May 17, 2016 at 12:53 PM Ittai Zeidman [email protected]
wrote:

ok. I think when people evaluate bazel vs pants vs buck the distributed
caching and execution plays a big part.
Having a note saying it's already there in as a prototype and can be used
by early adopters, willing to be broken, can encourage people. I'm such a
user.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/904#issuecomment-219684155

[ittaiz]

The discussion about where this feature should proceed will be in bazel-dev?

[damienmg]

We are planning to have a live discussion then we will follow up on
bazel-dev.

On Tue, May 17, 2016 at 1:21 PM Ittai Zeidman [email protected]
wrote:

The discussion about where this feature should proceed will be in
bazel-dev?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/bazelbuild/bazel/issues/904#issuecomment-219689523

[ittaiz]

looking forward to it

[LavaScornedOven]

Just as a feedback from a random C++ developer, missing distributed cache is a deal breaker for me, especially when I consider that I already got through pain of learning make, QMake, CMake, gradle, and what not, and doing it anew just to get another way to build locally... It just doesn't justify the effort.

[softprops]

Any progress on this?

[damienmg]

Not really, documentation is still missing...

[ittaiz]

@damienmg not to belittle documentation but does that mean the code itself is ready in your opinion?

[damienmg]

To use for remote caching yes, but it still has no server side. Hazelcast can be used but I am not sure about the stability.

I also prototyped a local disk cache yesterday, see https://cr.bazel.build/8133

[damienmg]

(side-note: the protocol is still not stable, so we might change stuff in the future)

[ittaiz]

Thanks for the clarification. Do you expect distributed actions to be supported any time soon?

[damienmg]

It is in the same state that caching, no server-side, no documentation still subject to change
As said on IRC to @abergmeier, the protocol itself should work.

[dabrahams]

Hi,

The FAQ that links here says: "The open source Bazel code runs build operations locally. We believe that this is fast enough for most of our users, but work is underway to provide distributed caching."

I think that's going to lead to a lot of disappointment. People that are considering adopting Google's build tool are highly likely to already be frustrated by long build times because they have (smaller-but-still-)very large projects to build. That's certainly how I arrived here, and having cluster-based operation _actually supported_ is what could make it worth an investment to evaluate bazel in-depth.

So, please consider this encouragement to push the feature over the line to completion. Thanks!

[philwo]

@dabrahams The feature (remote caching, remote execution and a combination of both) is definitely coming and there's a lot of work related to it going on. We still need some time, though.

[softprops]

Update: were using remote caching in ci at meetup using a mix of the nginx recipie

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/README.md#nginx-with-webdav-module

With a custom proxy pass to handle http puts that return 200 response ses (I think this is master is fixed to handle nginx's 201)

Things seem to be working really well.

[asa]

Can you post your nginx config for this? (handling the 201 with proxy_pass)

[ulfjack]

I just wanted to update this. All of my fixes should be in the future 0.5.0 release, except the fix for #2843, though not everyone will be affected by that. There are some bugs in 0.4.5 that may silently fall back on errors or miss certain outputs (though I don't think that you'll get corrupt outputs).

[ulfjack]

The fix for #2843 actually made it in.

[brunobowden]

@ulfadams - thanks and can clarify the new behaviour? Automatic fallback?
Congirable timeout?
On Fri, May 19, 2017 at 8:30 AM Ulf Adams [e-mail] wrote:

The fix for #2843 https://github.com/bazelbuild/bazel/issues/2843
actually made it in.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/904#issuecomment-302734962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AApdq-K5Fw5kWIphsaEeqfOoGkBnxLJNks5r7bWlgaJpZM4HaO3x
.

[ulfjack]

I think you're referring to #2964. I haven't made any progress on that. I'm currently in NYC, and will only be able to work on it the week after next.

[softprops]

@ulfjack looking forward to that. Our temporary work around is a wrapper script that curls the cache endpoint to test for a 200 as an isitup test before pretending bazel cache args. A first class fallback would be much more robust!

[softprops]

@asa

Can you post your nginx config for this? (handling the 201 with proxy_pass)

Sry for the super long delay. our remote cache impl is closed source but I can share our nginx config

we run our cache in k8s as a pod with nginx and a little rustlang app that handles the uploads ( the proxy_pass http://localhost:1337; bit is the rust app )

Our nginx config looks something like

    log_format timed_combined '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" '
      '$request_time $upstream_response_time $pipe';
    error_log stderr;
    access_log /dev/stdout timed_combined;
    server {
      listen 80;
      return 301 https://$host$request_uri;
    }
    server {
      listen 443 ssl;
      ssl_certificate /etc/nginx/certs/tls.crt;
      ssl_certificate_key /etc/nginx/certs/tls.key;
      ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers         HIGH:!aNULL:!MD5;
      client_max_body_size 5000M;
      location / {
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/auth/htpasswd;
        proxy_pass http://localhost:1337;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
      }
      location /protected {
        internal;
        root  /cache;
      }
    }

The rustlang app handles uploads but less nginx serve the blobs via returning X-Accel-Redirect headers for the content. We force clients through basic auth over https for now because that's about as much security you can pack into the --rest_cache_url flag. The proxy pass app also returns 200 on puts because bazel expects it in 0.4.5. I believe the list of response codes for puts was expanded in unreleased versions.

[dabrahams]

I'm sure the answers seem obvious to some, but AFAICT there's no definition of “remote cacheing” or “remote execution” anywhere. For example, what exactly is being cached, and what difference does that make to builds? I could (roughly) guess what “remote execution” means, but the one page of documents mentions it only in the title. There's no description of how to set it up or test it as far as I can tell. Currently there's so little explicit information available about either of these features that it's hard to tell whether I should be more than mildly intrigued.

One thing I'm wondering, in particular, is whether I need to set up the same build environment on all the machines in the cluster (like distcc) or whether something like icecream's chroot with archived build environment strategy is being used.

[ulfjack]

The preliminary docs are here:
https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/README.md

They are still work in progress.

Remote caching allows Bazel to reuse results from previous builds both your own as well as your coworkers (with the caveat that local toolchains need to be the same), for ~10x faster builds in our experience. Remote execution is another ~10x on top of that if you have enough machines. YMMV.

If you check in your toolchains into your repository, then Bazel uploads the files to the remote machine. If you use local toolchains, you need to match the local environment exactly (more like distcc), but we're also looking at docker containers for remote linux builds (more like icecream). We're looking at providing docker images / hermetic toolchains for at least a few languages to get everyone started quickly. MacOS and Windows support are both still work in progress, and may additionally have legal restrictions.

(Uploading to the remote machine only happens if the machine doesn't have the files cached already.)

[apobbati]

Is this issue still alive? Any work being done on it actively?

[buchgr]

Yep. Please take a look at https://docs.bazel.build/versions/master/remote-caching.html

[hhclam]

Nicely written!

[jgavris]

I concur, very nice work. Also +1, Google Cloud Storage is relatively easy to setup, and saves you from the hassle of having to run a service of your own!

[apobbati]

Awesome! Thanks for sharing

[PeterHeja]

Just stumbled into this issue from https://bazel.build/faq.html#does-bazel-require-a-build-cluster page, which list this feature as incomplete.
I think the completion of this feature should be reflected in the faq.