Bazel: Sharing Java outputs across platforms (Linux/Mac)

[gkossakowski]

Following up on https://github.com/bazelbuild/bazel/issues/4558#issuecomment-367597808
I'm starting this thread to discuss options for sharing JDK-derived (java rules, scala rules, etc.) outputs across platforms. The JDK itself is designed to be cross-platform so bytecode can be shared. However, the current design of bazel makes it difficult to share outputs across platform. See the linked ticket for details.

The linked ticket suggests a hacky workaround. All binaries that are not registered with bazel or WORKSPACE are not participating in the fingerprint calculation. The idea: set up bash scripts in a workspace that dynamically route to either linux or macos java binaries (they are not hermetic). Register these scripts and override local_sdk to use these scripts instead of what ships with bazel.

Questions:

• can the override be done so java rules pick up custom scripts for jdk resolution?
• will this actually make fingerprints coincide on on mac/linux?

Finally, how far are we from having a native support in bazel for sharing jdk outputs across platforms?

[ittaiz]

Greg,
I was also trying to crack this nut but I realized there are a lot of other variables in play. You have other actions which use native binaries which also need to be overridden (zip maybe). Additionally what will you do about Scrooge?
My point is you’ll need to maintain this for every small part.
What we’re currently mulling over (but haven’t gotten around to executing) is having Mac people seamlessly delegate to a local docker container which will be Unix and will share the cache

[gkossakowski]

Interesting. Did you figure out a way to ask bazel for the list of all files that participate in finger print calculation? I'd love to get a sense how difficult that nut is to crack.

The docker idea is interesting. Are you thinking of running bazel within docker? How would that work? Would you somehow pass the commands over docker barrier? Would IntelliJ work in that setup?

[ittaiz]

Re first question- no.

Re docker-
Yes, run bazel in docker. The rough idea is to have a script named bazel
which will call “docker run Bazel” with the arguments.
There are of course rough edges of mounts and such but they’re probably
solvable.
This should also please IntelliJ since it will just call Bazel as is.
On Tue, 27 Feb 2018 at 19:03 Grzegorz Kossakowski [e-mail]
wrote:

Interesting. Did you figure out a way to ask bazel for the list of all
files that participate in finger print calculation? I'd love to get a sense
how difficult that nut is to crack.

The docker idea is interesting. Are you thinking of running bazel within
docker? How would that work? Would you somehow pass the commands over
docker barrier? Would IntelliJ work in that setup?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4714#issuecomment-368950174,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFx4ldFKrx2hrHpRmUctgEJue-PhRks5tZDUzgaJpZM4SUKiz
.

[gregestren]

Re: the first question, you mean the fingerprint calculation for Bazel's action cache?

What specific kind of cross-platform caching are you interested in? Are you only interested in cross-platform hosts (building a Java library on a Mac or Linux machine, but in both cases targeting the same output architecture, i.e. --host_cpu is different but --cpu is the same)? Or are you also interested in cases where --cpu differs?

I've been prototyping platform-independent Java compilation by making input and output paths consistent. In other words, when --cpu=k8, all outputs today look like:

bazel-out/k8-fastbuild/my/project/some.output

This is unconditionally true: for C++ actions (where this makes sense), Java compilation (where it doesn't), and everything else. Since command input and output paths are part of Bazel's cache, this alone breaks meaningful cross-platform caching.

I've discovered some interesting results in my prototype, but the main gist is fixing this isn't trivial. One particular problem is generated code, e.g. if some Java source comes from a proto then host architecture info could make its way into that source. Another complication is select, which could swap out one of the source files in an arbitrarily platform-dependent way. Annotation processors could also be a problem.

But you imagine many classes of Java compilation don't use protos or select, etc. So at least in those cases these problems should be avoidable. The broader challenge is how to accurately identify these cases.

[ittaiz]

Yes, that’s the one.

We’re interested in java/Scala but also proto generation and hopefully
we’ll get to js/node/go/general docker containers in the quarters.
That’s why the docker solution sounds interesting since you can base the
container on the same image as CI agents and if we can get a smooth and
performant protocol it should be good
On Tue, 27 Feb 2018 at 20:20 Greg [e-mail] wrote:

Re: the first question, you mean the fingerprint calculation for Bazel's
action cache?

What specific kind of cross-platform caching are you interested in? Are
you only interested in cross-platform hosts (building a Java library on a
Mac or Linux machine, but in both cases targeting the same output
architecture, i.e. --host_cpu
https://docs.bazel.build/versions/master/user-manual.html#flag--host_cpu
is different but --cpu
https://docs.bazel.build/versions/master/user-manual.html#flag--cpu is
the same)? Or are you also interested in cases where --cpu differs?

I've been prototyping platform-independent Java compilation platform by
making input and output paths consistent. In other words, when --cpu=k8,
all outputs today look like:

bazel-out/k8-fastbuild/my/project/some.output

This is unconditionally true: for C++ actions (where this makes sense),
Java compilation (where it doesn't), and everything else. Since command
input and output paths are part of Bazel's cache, this alone breaks
meaningful cross-platform caching.

I've discovered some interesting results in my prototype, but the main
gist is fixing this isn't trivial. One particular problem is generated
code, e.g. if some Java source comes from a proto then host architecture
info could make its way into that source. Another complication is select
https://docs.bazel.build/versions/master/be/functions.html#select,
which could swap out one of the source files in an arbitrarily
platform-dependent way. Annotation processors
https://docs.bazel.build/versions/master/be/java.html#java_plugin could
also be a problem.

But you imagine many classes of Java compilation don't use protos or
select, etc. So at least in those cases these problems should be
avoidable. The broader challenge is how to accurately identify these cases.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4714#issuecomment-368975428,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF5DQF8WzSkzzFBaf2ICUix_t4SyPks5tZEdYgaJpZM4SUKiz
.

[gkossakowski]

My question about action digest/key calculation is about understanding the following scenario.

I'm trying to build a tiny example java target on two different machines that are supposed to be configured exactly the same.
Here's the code: https://github.com/gkossakowski/bazel-scala-example/tree/master/example-java

On one machine:

$ ./bazel build //example-java/...
INFO: Analysed target //example-java:example-java (8 packages loaded).
INFO: Found 1 target...
Target //example-java:example-java up-to-date:
  bazel-bin/example-java/libexample-java.jar
INFO: Elapsed time: 2.108s, Critical Path: 1.05s
INFO: Build completed successfully, 3 total actions
$ bazel dump --action_cache
Warning: this information is intended for consumption by developers
only, and may change at any time.  Script against it at your own risk!

String indexer content:

size = 4
bazel-out/host/internal/_middlemen/external_Slocal_Ujdk_Cjdk-default <==> 2
bazel-out/stable-status.txt <==> 0
bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params <==> 1
bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar <==> 3

Action cache (5 records):

0, bazel-out/stable-status.txt:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 10b1cf011f90321fb732fc449a053ac9

      packed_len = 40

1, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params:
      actionKey = 7c3c5f3be3f28b07a362ebebab153e55
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = c0865d70c0cf9fc7e8d2f3c3c3752e1d

      packed_len = 72

2, bazel-out/host/internal/_middlemen/external_Slocal_Ujdk_Cjdk-default:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 17ae06b3c4fb3fae5589a7f2cc7345a7

      packed_len = 40

3, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar:
      actionKey = c09910e3dce7ec078babc0586f2ae44f
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 656d73b79c3084253e24d32bda210efd

      packed_len = 72

and then on another:

$ ./bazel dump --action_cache
Warning: this information is intended for consumption by developers
only, and may change at any time.  Script against it at your own risk!

String indexer content:

size = 4
bazel-out/host/internal/_middlemen/external_Slocal_Ujdk_Cjdk-default <==> 2
bazel-out/stable-status.txt <==> 0
bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params <==> 1
bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar <==> 3

Action cache (5 records):

0, bazel-out/stable-status.txt:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 680d3aa439f2e7ed379d65e8418edcff

      packed_len = 40

1, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params:
      actionKey = 7c3c5f3be3f28b07a362ebebab153e55
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = c0865d70c0cf9fc7e8d2f3c3c3752e1d

      packed_len = 72

2, bazel-out/host/internal/_middlemen/external_Slocal_Ujdk_Cjdk-default:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 2df4b787734c4ed41f36d8181688a1ac

      packed_len = 40

3, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar:
      actionKey = c09910e3dce7ec078babc0586f2ae44f
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 087b248a91b01e3675efd87d9a9edded

      packed_len = 72

You can see that digest for external_Slocal_Ujdk_Cjdk-default is different on both. Why? How I can see what goes into computing that digest? Which file of jdk is different that caused digest to be different?

The action key for libexample-java.jar is the same but their digests are different. I think it's due to local_jdk digests being different.

[gregestren]

external_Slocal_Ujdk_Cjdk-default comes from here, which you can see reduces to four other file groups from the JDK.

To see which files these ultimately resolve to, you can run:

$ bazel query 'kind("source file", deps(@local_jdk//:jdk-default))’
@local_jdk//:lib/tools.jar
@local_jdk//:lib/sa-jdi.jar
@local_jdk//:lib/packager.jar
...

I believe all these files are staged for all compilation actions, i.e. here, which Bazel collects from jdk-default here.

One way to see which system files these resolve to:

$ bazel info | grep execution_root
execution_root: /private/var/tmp/_bazel_user/install/057f65b62d3a9f08e742ae63568ed3e5/execroot/a
$ cd /private/var/tmp/_bazel_user/install/057f65b62d3a9f08e742ae63568ed3e5/execroot/a
$ cd external/local_jdk/bin
$ pwd -P
/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/bin

[gkossakowski]

I followed your instructions to go to execroot and then:

$ cd external/local_jdk
$ find -L -type f -exec md5sum '{}' \; | sort -k2 > ~/tmp/"$(hostname -s).localjdk.md5"

And then I got:

$ diff *.localjdk.md5
112c112
< f17fcf0a5569896a71f5ff6016619f3e  ./jre/lib/amd64/server/classes.jsa
---
> 08d402cc1a2a1a943a14a9c886def3d3  ./jre/lib/amd64/server/classes.jsa
166c166
< 719a2fa00e75a11b3bf4110c12ca2594  ./jre/lib/security/cacerts
---
> 626cdc4888a02cd1d287bf3873e597c4  ./jre/lib/security/cacerts

I was sure I did similar steps yesterday and didn't see any difference. Thanks for the instructions, this is handy. A suggestion: could we have a slow mode in bazel that automates what I just did?

Aside for that, I don't understand why these files are different on the two machines. I suspect that we've built jdk out of sources and somehow that build is not reproducible. These files are large binaries so it's hard to tell what's the problem. Is there an escape hatch that let's me say "oh, i'm sorry that these files are different but can you just carry on pretending they're not"?

[gkossakowski]

Both cacerts and classes.jsa appear to be jvm-internal stores so I don't think bazel should take them into account when calculating digests. Thoughts?

[jodim]

Re: running bazel inside a docker container, you may be interested in https://github.com/nadirizr/dazel .

[gkossakowski]

I tried the idea of using bazel build just from the docker container to enforce reproducibility but that doesn't work due to localjdk including non-reproducible bits: #4769

[gkossakowski]

I filed a separate issue for bazel's own build not being reproducible on Mac: https://github.com/bazelbuild/bazel/issues/4770
This blocks the remote cache sharing between Mac machines; not sure if intentionally or by design, though.

[gregestren]

I wouldn't be surprised if @local_jdk//:jdk-default is overly broad, especially if this is inherited from the Google implementation where platforms are more consistent and extra files aren't as big a deal.

We've encountered this kind of issue in the past for other toolchains and have often successfully mitigated by simply declaring these rules more finely or breaking them down into sub-rules which are more selectively opted into actions that need them. This can be a win/win all around.

As an experiment, would you like to try excluding these files from the globs in your workspace and see if you get the caching you want? If that works for you we can absolutely explore a more proper change.

[gkossakowski]

thanks greg! is there a way to exclude these files declaratively from my workspace without forking bazel and modifying it there? The relevant filegroup that needs modification is this: https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD#L104

[gregestren]

Good point. Can you clone the file before making your changes, then redirect Blaze to that JDK with --javabase? (this is the default)

[gkossakowski]

will setting javabase rewire everything in bazel that depends on @bazel_tools//tools/jdk:jdk? I tried to find out with baze query but it doesn't recognize the javabase option.
I'm worried that some parts will see my own jdk and some others will still refer to the default one. How to find out?

[gkossakowski]

I just tried setting the --javabase but it seems like not all actions are picking it up. Is there a way to determine which actions are still dependent on local_jdk?

[gregestren]

We can examine on a per-action basis, but I'd still like to see if we can work through the query approach, since there are less targets than actions.

I forgot above to include --host_javabase, which is the same thing, but for building host tools (tools that run as part of the build). When I include both --javabase and --host_javabase I see no more dependencies on local_jdk.

About setting these options with query, if you're using the latest Bazel you can use the new cquery command: it's the same as query but accepts all build options (it looks like it's not in the documentation yet - I'll ping the creator about that).

So here's what I see:

$ cat java/testapp/BUILD
java_library(
    name = "MyLib",
    srcs = ["MyLib.java"])

java_runtime_suite(
    name = "jdk",
    default = ":myjdk",
    visibility = ["//visibility:public"])

java_runtime(
    name = "myjdk",
    srcs = [], # gibberish proof of concept
    java_home = "/usr/bin")

$ bazel version
Build label: 0.11.1-homebrew    # Yup, includes cquery!

$ bazel cquery '(deps(//java/testapp:MyLib)' --javabase=//java/testapp:jdk \
    | grep local_jdk
@local_jdk//:jdk (HOST)
...

$ bazel cquery '(deps(//java/testapp:MyLib)' --javabase=//java/testapp:jdk \
    --host_javabase=//java/testapp:jdk | grep local_jdk

[gkossakowski]

First of all, neat! cquery is what i wished to have for a long time.

Do I understand correctly that by specifying srcs to be empty, we're side-stepping calculating digests of files in jdk?

I reproduced your configuration in https://github.com/gkossakowski/bazel-scala-example and tried it on two linux machines configured exactly the same. I continue to see a disagreement of digestKeys:

Action cache (7 records):

0, bazel-out/host/internal/_middlemen/external_Sscala_Uexample_Stools_Sgkk_Ujdktools_Sgkk_Ujdk_Cmyjdk:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 00000000000000000000000000000000

      packed_len = 40

1, bazel-out/stable-status.txt:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = a2b5ce947a4b532057d60656a790b7e5

      packed_len = 40

2, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params:
      actionKey = 5acebb6ca936bcb66253ea9a3e49d774
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = b826002096247f3a387f422573208934

      packed_len = 72

3, bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath-impl.jar:
      actionKey = e1bb033a30ca88b50dfd5573137e59f4
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = adf6f8ae416bd36d6a89856d51cefff9

      packed_len = 72

4, bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath.jar:
      actionKey = 1138dbbdb68551d9ef21550dea65839a
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 4afc86eafae3a5a1428a1b669111cadd

      packed_len = 72

5, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar:
      actionKey = 0b0d2ddc7c0f528a1892786634a34e21
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 858883d734533267f574dfb611eb5d29

      packed_len = 72

vs

ction cache (7 records):

0, bazel-out/host/internal/_middlemen/external_Sscala_Uexample_Stools_Sgkk_Ujdktools_Sgkk_Ujdk_Cmyjdk:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 00000000000000000000000000000000

      packed_len = 40

1, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params:
      actionKey = 5acebb6ca936bcb66253ea9a3e49d774
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = b826002096247f3a387f422573208934

      packed_len = 72

2, bazel-out/stable-status.txt:
      actionKey =
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = da093b315c2986d2d7799ffa7c1b51d3

      packed_len = 40

3, bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath-impl.jar:
      actionKey = e1bb033a30ca88b50dfd5573137e59f4
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 71d068833a72c402b6846168d5222a56

      packed_len = 72

4, bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath.jar:
      actionKey = 1138dbbdb68551d9ef21550dea65839a
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = dae7d832d7cb1fb72b3c8bc346f70c02

      packed_len = 72

5, bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar:
      actionKey = 0b0d2ddc7c0f528a1892786634a34e21
      usedClientEnvKey = 00000000000000000000000000000000
      digestKey = 7d8685ccba31f52a961164b4ff3091e4

      packed_len = 72

[gkossakowski]

I recalled from my notes that I did debug the problem with platformclasspath digest keys disagreeing. At the time, it was due to differences in $PATH that were mitigated by --experimental_strict_action_env. I gave this a whirl but no success in making digests match.

[gregestren]

I emptied out srcs just as a proof of concept for query / cquery. Since that should also empty out javac, I wouldn't expect that to build at all (when I bazel build with the same options as my last comment I get /usr/bin/bin/javac: No such file or directory).

What command did you run exactly? Do you see any diffs in libexample-java.jar-2.params, or are the diffs just in its inputs?

I'd expect bazel-out/stable-status.txt to be different, since that includes the host name. platformclass shouldn't matter since that's intended just for bootstrapping Bazel.

As for libexample - the meat of this - I wish we had better tooling for dumping the full input list. This exists inside Google, but it's part of Google's remote executor, so it's not here in Bazel. @meisterT recently started committing code for bazel dump --action_graph, which should help a lot when ready.

[gkossakowski]

Sorry got sidetracked by other work. I just tried bazel build //example-java --javabase=//example-java:jdk --host_javabase=//example-java:jdk and it builds successfully for me.
I pushed my branch with an example to https://github.com/gkossakowski/bazel-scala-example/tree/gkk-reproducibility

I'll check the .params file across machines and report back.

[gkossakowski]

The contents of bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params is the same on both machines but libexample-java.jar have different digest key.

[jmillikin-stripe]

gkossakowski and I have been looking a bit at the minimal hello world, using javabase flags. When I run bazel build -s in a clean repo it shows all the commands being invoked, and when I shasum the files involved the only difference is in platformclasspath-impl.jar and platformclasspath.jar. I know gregestren said a few comments ago that these two jars shouldn't matter for compiling normal targets, but they are literally the only files that have different checksums between the two machines.

These jars look the same when extracted, but I notice that zipinfo reports their contents in a different order. Looking at Bazel's tools/jdk/DumpPlatformClassPath.java, I see it's walking some directory tree but _not_ sorting the resulting file list. Is it possible that this non-determinism is simply because Bazel's Java bootstrap isn't producing deterministic classpath jars here?

Also, I accidentally built the test program with Bazel 0.6.1 and the cache keys matched on both systems. So I strongly suspect this non-determinism was introduced between 0.6.1 and 0.11.1.

[jmillikin-stripe]

When compiling libexample-java.jar, the params file bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params contains --bootclasspath bazel-out/host/genfiles/external/bazel_tools/tools/jdk/platformclasspath.jar. So this file must be available within the sandbox, and therefore part of the action digest.

[jmillikin-stripe]

edit: never mind, this mismatch was due to a stray newline in the source file on one of the machines.

[jmillikin-stripe]

I've verified that the above change to platformclasspath.jar generation causes both machines to generate the same action cache digest key.

[pauldraper]

Re: Bazel in Docker
https://github.com/nadirizr/dazel

[rdnetto4]

Bazel in Docker

Be aware that building large projects in Docker on OSX resulted in a ~3x slowdown the last time I looked into it. IMO, it is a completely inadequate solution to the problem.

[pauldraper]

@rdnetto4 disk I/O is premium for any virtualization (and Docker on Mac is in VM).

Did you happen to

1. Use the delegated mode for mounting the workspace.
2. Ensure Bazel output directory is not on the shared host mount.
3. Further than (2), put the Bazel ouput directory on a separate Docker volume, which bypasses Docker's layered container filesystem and uses the hosts/VM's storage directly.

[rdnetto4]

It was about a year ago, so I can't remember the exact details. I have a
vague recollection I didn't spend much time fine tuning it, as for our
application even a 20% slowdown would have been too much.

On Mon, Sep 10, 2018 at 11:11 AM Paul Draper [e-mail]
wrote:

@rdnetto4 https://github.com/rdnetto4 disk I/O is premium for any
virtualization (and Docker on Mac is in VM).

Did you happen to

1. Use the delegated
   https://docs.docker.com/docker-for-mac/osxfs-caching/ mode for
   mounting the workspace.
2. Ensure Bazel output directory is not a shared mount with host.
3. Put the Bazel ouput directory on a separate Docker volume (which I
   believe will us a more efficient type than Docker's layered container file
   system).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4714#issuecomment-419760233,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AikcoRZbs9yKasi0ypJBLhO6Fdx5Nb7Yks5uZbw9gaJpZM4SUKiz
.

[pauldraper]

The various consistency modes for bind mounts arrived in Docker last April,
so you may have been using that feature.

It's pretty much a requirement for any Docker dev on MacOS. Sped up example
builds 2-3x
https://blog.docker.com/2017/05/user-guided-caching-in-docker-for-mac/

On Sun, Sep 9, 2018 at 8:57 PM rdnetto4 [e-mail] wrote:

It was about a year ago, so I can't remember the exact details. I have a
vague recollection I didn't spend much time fine tuning it, as for our
application even a 20% slowdown would have been too much.

On Mon, Sep 10, 2018 at 11:11 AM Paul Draper [e-mail]
wrote:

@rdnetto4 https://github.com/rdnetto4 disk I/O is premium for any
virtualization (and Docker on Mac is in VM).

Did you happen to

1. Use the delegated
   https://docs.docker.com/docker-for-mac/osxfs-caching/ mode for
   mounting the workspace.
2. Ensure Bazel output directory is not a shared mount with host.
3. Put the Bazel ouput directory on a separate Docker volume (which I
   believe will us a more efficient type than Docker's layered container
   file
   system).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/bazelbuild/bazel/issues/4714#issuecomment-419760233
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AikcoRZbs9yKasi0ypJBLhO6Fdx5Nb7Yks5uZbw9gaJpZM4SUKiz

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4714#issuecomment-419773431,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVph1KO76jdsbmN1krsbbalf_ZdHcyvks5uZdUVgaJpZM4SUKiz
.

[talya]

@pauldraper and @rdnetto4 - Note that the consistency mode delegated is not actually implemented and currently is identical to cached mode.
This very long thread describes that d4m plan to implement this "in order to speed up write-heavy workloads", but no updates for a long time now.

We have been experimenting for some time now with developing with bazel inside docker, and we use a local nfs mount instead of a docker volume/mount. This has been working quite well for as so far.

If I understand what you meant by (3) correctly, a "separate Docker volume, which bypasses Docker's layered container filesystem" - this will indeed allow bazel to happily run and output everything with native speed, however you will still have to wait (a lot) in order to actually see and use these outputs from your IDE which is running locally on your mac (similar to what happens when using docker-sync project as a workaround for this exact bottleneck)

[pauldraper]

however you will still have to wait (a lot) in order to actually see and use these outputs from your IDE

Hm, is that often an issue? Do IDEs need any of the outputs? (Assuming you are building, testing, running, etc. in the container.)

We have been experimenting for some time now with developing with bazel inside docker, and we use a local nfs mount instead of a docker volume/mount. This has been working quite well for as so far.

Interesting.

And then do you do file watch (ibazel) on the host?

[talya]

IDEs need the outputs - the intellij-info.txt files which are how bazel sync works, all the logs (for running tests and seeing their log and more importantly the test.xml file for the plugin to display test results/count), and probably the generated binaries for some cases.

re file watch - no need to, the nfs mount is from the mac's host native filesystem, and is available to the docker container via a type:nfs docker volume.
So when the container writes to it's output it's actually writing to yhe native filesystem.
(I didn't know about ibazel, interesting, do you use it for this docker dev use case?)

[thelgevold]

I have also had some issues with high cache miss rates when the build clients are not "identical".

So far I've had the most luck from running in docker to "control" the environment. Configured my containers to build a few large example apps using BuildFarm for remote execution + caching.

Still looking into how I can relax the identical computer requirement.

Here is my docker setup in case anyone is interested/has suggestions for how to improve it: https://github.com/thelgevold/remote-builder

[gregestren]

Been doing some experiments in this space recently: see https://github.com/bazelbuild/bazel/issues/6431#issuecomment-444261808. I'm personally not sure how Docker affects the story and this thread has gotten pretty big since I last visited it.

What's the most actionable request we're looking for on this issue now? Is this all contingent on docker now?

[guw]

@gregestren I don't think it should be about Docker.

The requirement for us is:

• sharing Java outputs in Bazel remote cache across platforms (Linux/Mac)

Basically, we want to populate a remote cache from a set of blessed/trusted environments for all developers. These trusted builds run on Linux. Some developers use Macs. It's mostly Java output I'm concerned about (class files and/or jars).

I would like to add:

• don't require building inside Docker

[pauldraper]

Sure, that's a good goal.

On Wed, Mar 6, 2019 at 11:56 AM Gunnar Wagenknecht [e-mail]
wrote:

@gregestren https://github.com/gregestren I don't think it should be
about Docker.

The requirement for us is:

• sharing Java outputs in Bazel remote cache across platforms
  (Linux/Mac)

I would like to add:

• don't require building inside Docker

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/4714#issuecomment-470230715,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVph4S4t0AYDD7zqeEIlkzJkvGteRiUks5vUA9VgaJpZM4SUKiz
.

[gregestren]

Hmm, the comments in https://github.com/bazelbuild/bazel/issues/4714#issuecomment-374795250 and below look promising. So it seems like some people are getting caching and others aren't?

For those who are, are you targeting the same CPU on both machines? i.e. bazel-out/k8-fastbuild/bin/example-java/libexample-java.jar-2.params implies the target CPU is always k8. Is that ever not the case (especially when building on Mac vs. Linux)?

@guw - are you finding this problem on any java builds (even trivial example ones)? Can you share more details of what you're seeing?

[cgruber]

Seconding @guw's comment above. Populating a build cache with platform-agnostic artifacts (e.g. java/kotlin/android artifacts) from blessed machines (which are nearly certainly Linux machines) and consuming cache content from developer machines (which are nearly certainly Mac machines).

[gregestren]

FYI Experimental Content-Based Output Paths is a proposal to address the output path side of the problem.

I'd like to implement it as an --experimental feature specifically for Java outputs for a first pass (assuming the design review gets the thumbs up). And I'd love to collaborate with some of these setups to see how far this takes you.

[gregestren]

I'm going to mark this a dupe of https://github.com/bazelbuild/bazel/issues/8339. Technically I believe the issue is broader. But that one should better track our focused efforts, and naturally leads here anyway.

[johnynek]

@gregestren you said it was a dupe of itself. Wild concept. If there is another issue it is a dupe of can you edit your comment?

[gregestren]

Apologies - updated the link.