Bazel: Obtaining output paths of the build artifacts

[vitarb]

There seem to be no easy way of getting location of the artifacts built by bazel, which results in a lot of unnecessary complexity and potentially faulty assumptions in tooling that needs to access build artifacts.

Most difficulties arise from the fact that bazel adds various sub-folders in the output such as "external", "$platform_$arch_[stripped|opt]", "$target_name.runfiles", etc. which requires any tooling that needs to be able to find the location know information about the build target, environment and all bazel layout conventions.

Buck solved this problem by introducing buck targets --show-output command that prints location of each build target in the build output folder.

For example:

buck targets --show-output //java/com/myproject:binary
> //java/com/myproject:binary buck-out/gen/java/com/myproject/binary.apk

The only potential work-around that I was able to find was using bazel aquery as described here but it's not easy to use.

It would be nice to have equivalent in bazel, is there anything that prevents us from exposing this information as part of bazel query command?

Can we have bazel query mytarget --show-output or other simple equivalent?

[shs96c]

I'm not sure why this was closed, because it's a great feature. Digging into the bazel source code, I think what needs to be returned are the "important artifacts" for each target. This loop seems to have the logic needed.

[laurentlb]

@meisterT This could be an output option for aquery?

[meisterT]

It already outputs this information. Example:

$ bazel aquery //src:bazel-bin
INFO: Analyzed target //src:bazel-bin (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
action 'Executing genrule //src:bazel-bin'
  Mnemonic: Genrule
  Target: //src:bazel-bin
  Configuration: k8-fastbuild
  ActionKey: a8904630b97056992474b857be1a11b9
  Inputs: [bazel-out/k8-fastbuild/bin/src/bazel_jdk_minimal, external/bazel_tools/tools/genrule/genrule-setup.sh]
  Outputs: [bazel-out/k8-fastbuild/bin/src/bazel]
  Command Line: (exec /bin/bash \
    -c \
    'source external/bazel_tools/tools/genrule/genrule-setup.sh; rm -f bazel-out/k8-fastbuild/bin/src/bazel; cp bazel-out/k8-fastbuild/bin/src/bazel_jdk_minimal bazel-out/k8-fastbuild/bin/src/bazel')

More information on aquery is here:
https://docs.bazel.build/versions/master/aquery.html

Another source for the information for a particular build is the BEP, e.g. running:

$ bazel build --build_event_text_file=/tmp/bep.txt //src:bazel-bin
...
$ cat /tmp/bep.txt | grep ^completed -A15
completed {
  success: true
  output_group {
    name: "default"
    file_sets {
      id: "0"
    }
  }
  important_output {
    name: "src/bazel"
    uri: "file:///home/twerth/.cache/bazel/_bazel_twerth/d8eb7a85967b22409442664d380222c0/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/bazel"
    path_prefix: "bazel-out"
    path_prefix: "k8-fastbuild"
    path_prefix: "bin"
  }
}

[shs96c]

@meisterT, I'm not sure how to get the important_output without running the build, so I believe that this issue has been closed too soon.

The output information doesn't appear to be in the aquery output, or easy to identify. The BEP is only there after executing the build.

One solution would be to expose the important_output from aquery.

buck targets --show-output provided a simple mechanism to integrate buck into other systems using standard UNIX commands. Bazel has no functional equivalent yet.

[meisterT]

Simon, note that aquery exposes all outputs, but doesn't distinguish between important and other outputs.

Joe, please have a look if we can easily expose the "important outputs" from aquery.

[Globegitter]

I also agree it would be nice to have an easier way to get the output for a target. I have had the question sometimes from developers new to bazel on how to easily figure out where a built target ends up. And so far my answer has been: Look at the end of the logs of the bazel build command and that has sometimes lead to follow-up questions. But it would be nice if I could send them one simple command which would give them the answer in an easily parse-able way.

[meisterT]

The caveat with a simple command is that the output location might change depending on the flags you pass to the build command. I have ideas to change that but they haven't left brainstorming stage yet (and are rather invasive and incompatible).

[shs96c]

@meisterT, it looks like aquery respects the same flags that are passed to the build command. If that's the case, then exposing the important_outputs would be enough to get this off the ground. It wouldn't be a "one liner", but it would be possible to extract the information we need, and that's the key thing :)

@joeleba, good luck!

[linzhp]

For an runnable target, we found a workaround using bazel run --run_under "echo " //path/to/binary. But for general cases, an easily parsable output location is still desired.

[shs96c]

Sadly, most of the rules I want the output to are either custom rules, or no runnable. That is a clever hack, though.

[illicitonion]

I ended up putting together this script which uses aquery to get the output path for a particular mnemonic:

#!/bin/bash -eu

# Prints the path to the (single) output file of the action with the given mnemonic, for the given target.

if [[ $# != 2 ]]; then
  echo >&2 "Usage: $0 target mnemonic"
  echo >&2 "e.g. $0 //some:target JavaSourceJar"
  exit 1
fi
target="$1"
mnemonic="$2" # e.g. GoLink or PythonZipper

out="$(mktemp)"
trap "rm -f ${out}" EXIT
bazel aquery --output=jsonproto "${target}" > "${out}"

outputs_id_query_base=".actions[] | select(.[\"mnemonic\"] == \"${mnemonic}\")| .outputIds"
output_count="$(jq "${outputs_id_query_base} | length" "${out}")"
if [[ "${output_count}" -ne 1 ]]; then
  echo >&2 "Saw ${output_count} outputs for the mnemonic and target but expected 1"
  exit 1
fi
output_id="$(jq "${outputs_id_query_base}[0]" "${out}")"
jq -r ".artifacts[] | select(.[\"id\"] == ${output_id}) | .execPath" "${out}"

[mexisme]

I ended up putting together this script which uses aquery to get the output path for a particular mnemonic:

#!/bin/bash -eu

# Prints the path to the (single) output file of the action with the given mnemonic, for the given target.
[...]

This is pretty nice, just what I've been looking for, and _very_ tempting to turn into a little Rust or Go tool (for fewer runtime deps...)
It doesn't look like there's a consistent mnemonic to look-up, however?

[illicitonion]

It doesn't look like there's a consistent mnemonic to look-up, however?

There isn't, each language will have its own primary output Mnemonic. tbh there are probably few enough that you could hard-code them: Javac, GoLink, etc.

You could potentially write an aspect to look up the DefaultInfo Provider of the target, and either use that as-is or merge it with the above...

[mexisme]

There isn't, each language will have its own primary output Mnemonic.

A pity, but understandable.

tbh there are probably few enough that you could hard-code them: Javac, GoLink, etc.

That was my fall-back thought.

You could potentially write an aspect to look up the DefaultInfo Provider of the target, and either use that as-is or merge it with the above...

Thanks, that sounds worth looking into.