Bazel: Protobuf repo recompilation sensitivity

[pcj]

As discussed on the mailing list https://groups.google.com/forum/#!topic/bazel-discuss/UKLK7Y-jsW4 the protobuf repo is notoriously sensitive to recompilation. Presumably something in the environment used in the action key for protobuf C++ compilation leads to cache invalidation and hence recompiling protobuf from source.

Given protobuf libs are often base dependencies for many others, leads to degraded user experience.

Repro steps include taking any workspace that has a proto dependency and source ~/.bashrc, watch protobuf recompile.

Observed up to and including bazel 0.20.0.

/assign @hlopko

[hlopko]

Can you double check that:

git clone https://github.com/bazelbuild/rules_cc
cd rules_cc
bazel test //tools/migration/...
source ~/.bashrc
bazel test //tools/migration/...

results in recompilation? What is the minimal content of the ~/.bashrc file that triggers the issue? What platform are you on?

[pcj]

Looks like it's the PATH. Any changes to PATH will trigger recompilation.

Perhaps this is working as intended, or perhaps we can split it and use a set to dedup entries to normalize the path to the minimal necessary set.

Is it possible to exclude PATH here and rely solely on the toolchain?

[pcj]

$ uname -a
Linux bina 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[hlopko]

Hmmm then https://github.com/bazelbuild/bazel/issues/6648 should actually help you (afaik). Can you try with bazel 0.21?

[hlopko]

or with 0.20 and with --incompatible_strict_action_env

[pcj]

Yes! Tested on bazel 0.21.0 and indeed no longer recompiles with PATH change. I guess I will go ahead and close this. The migration path from 0.19.2 to 0.20.0 and 0.21.0 is a little challenging (for this same reason), but clearly beneficial. Thanks for your help @hlopko.

[hlopko]

Awesome, thanks Paul. Can you pls share this solution on the mailing list?

[hlopko]

@pcj What toolchain do you use? Was it the Bazel's autoconfigured C++ toolchain?

[hlopko]

Also, on what platforms do you observe this?

[pcj]

@hlopko I had observed it on linux & darwin before bazel 0.21.0. Now that I'm on 0.21 I'm really not seeing this anymore.

For most projects I'm not using a special toolchain, so yes, the autoconfigured one.

[buchgr]

I believe that I found the root cause. It looks like this issue is not specific to C++, but that almost all actions (spawn actions, c++ compile, c++ link, java compile) in Bazel declare a dependency on PATH independent of whether these actions actually use them. It's probably just most obvious with c++ actions as they typically take quite long.

[keith]

That would make sense with our solution to this which was to launch bazel with env -i and only pass very limited variables (and a static PATH variable)

[meisterT]

@buchgr friendly ping

[buchgr]

@meisterT what's up? :)

[meisterT]

I have heard from bazel users that this is still a pain point. So I wanted to check in if you are working on a fix and when we can expect it to land?

[buchgr]

Yes I am working on a proposal to fix it. I don't want to make promises on when it will land. 1-2 quarters.

[sudoforge]

I currently suffer from the protobuf targets rebuilding (a dependency of bazelbuild/buildtools (specifically, buildifier), but cannot reproduce the issue by re-sourcing my shell's rc file. To be honest, I'm really not sure how to debug this further other than "it happens every 5-10 minutes of inactivity".

[Anduril-Xu]

Its been annoying for a while.

In my setting, I have several languages under the same workspace, python, java, javascript/typescript. I use Intellij for most of the development, but every time I build one of the targets in one of the languages, the bazel cache seems to be invalidated for other languages, thus triggering a rebuild of some packages, most notably protobuf. It is only 10-20 seconds each time, but very annoying when I do cross language development.

Is this expected? anyway to not rebuild protobuf?

EDIT:
my bazel version was 0.29.1. the latest 3.2.0 also does not solve the problem

EDIT again:
seems like using prebuilt binaries works, e.g. https://github.com/google/startup-os/blob/master/WORKSPACE#L142-L154

[franciscocpg]

Its been annoying for a while.

In my setting, I have several languages under the same workspace, python, java, javascript/typescript. I use Intellij for most of the development, but every time I build one of the targets in one of the languages, the bazel cache seems to be invalidated for other languages, thus triggering a rebuild of some packages, most notably protobuf. It is only 10-20 seconds each time, but very annoying when I do cross language development.

Is this expected? anyway to not rebuild protobuf?

EDIT:
my bazel version was 0.29.1. the latest 3.2.0 also does not solve the problem

EDIT again:
seems like using prebuilt binaries works, e.g. https://github.com/google/startup-os/blob/master/WORKSPACE#L142-L154

Hi @Anduril-Xu
Were you able to configure your workspace to use prebuilt binaries? I have a go monorepo and haven't had success yet :disappointed:

[sudoforge]

@franciscocpg Are you still experiencing this? Have you tried using --incompatible_strict_action_env?

Failing that, or if you simply desire to use prebuilt binaries directly without future debugging, you should be able to simply load the binaries via http_file. Have you tried following the example that was linked to in google/startup-os?

[franciscocpg]

Hi @sudoforge

I want to use prebuilt binaries and have asked about it here: https://github.com/bazelbuild/rules_go/issues/2684.

But I haven't tried http_file yet. Will give it a try.