Bazel: BEP's named set of files is incomplete if an action from any output group of a set of groups fails

[jin]

Let T be a target, foo-outputs and bar-outputs be output groups, foo-action and bar-action be actions in T, and foo.out and bar.out be outputs generated by foo-action and bar-action respectively.

• If we build T and request for foo-outputs, we expect foo.out to be in a NamedSetOfFiles in the BEP output. This is WAI.
• If we build T and request for foo-outputs and bar-outputs, we expect foo.out and bar.out to be in in their own NamedSetOfFiles, since they were generated by different actions foo-action and bar-action. This is WAI.
• If we build T and request for foo-outputs and bar-outputs, and bar-action fails, we expect that foo.out to still be listed in BEP. However, this is not the case and foo.out is missing, even with --keep_going.
• Additionally, if there is a target U that depends on T, U's foo-outputs files will also be missing from the BEP.

This has caused issues with IJwB's Java integration. See this issue for the diagnosis and a repro project. https://github.com/bazelbuild/intellij/issues/1167#issuecomment-533693212

The intuitive BEP state here is that the foo-outputs files should still be listed, regardless of failing actions requested by other output groups.

[jin]

See simplified repro here: https://github.com/jin/repro/tree/master/9413

[jin]

@ulfjack @aehlig does the following intuition look right to you?

The intuitive BEP state here is that the foo-outputs files should still be listed, regardless of failing actions requested by other output groups.

[brendandouglas]

Quick mention that this is very important for the IntelliJ plugin, both for Blaze and Bazel users. The workaround of running separate builds (which we don't do, and which would require significant work) causes serious performance regressions when running remotely, so isn't a good option.

[ulfjack]

The semantics you describe seem reasonable. Unless it's overly invasive, it seems reasonable to change the implementation to match. I haven't looked at the code.

[jin]

Moving this over to the Bazel Core team.

[jin]

(adgar@ is looking at this)

[ittaiz]

Any updates?

[jin]

Yes, see https://bazel-review.googlesource.com/c/bazel/+/120613

[jin]

cc @michaeledgar

[michaeledgar]

Sorry for the delay with fixing this particular issue.

The first attempt at a fix was known to misreport some action outputs when building with --nokeep_going. My latest attempt addresses that concern. The new approach also handles the case where some (but not all) actions fail while producing outputs for the same output_group.

[jin]

thanks, @michaeledgar!

[michaeledgar]

No problem, thanks for the super-detailed report!

[ittaiz]

Thanks! This will get in 2.0 right?

[jin]

2.0's release candidates have been cut, so this will come in 2.1.

[michaeledgar]

Reopening this issue while the fix submitted as 0552bd7 is reworked to avoid producing overlarge NamedSetOfFiles messages. That fix has already been rolled back (d7d441b).