Bazel: Windows `repository_ctx.execute(command)` sometimes returns incomplete result

[meteorcloudy]

This is causing https://github.com/bazelbuild/bazel/issues/2434 and the internal Kokoro Windows Flakiness: missing "LIB"

Extracting Bazel installation...
................................
____Loading package: src
____Loading package: @bazel_tools//tools/cpp
____Loading package: @bazel_tools//tools/jdk
____Loading package: @local_config_xcode//
____Loading package: @local_jdk//
ERROR: in target '//external:cc_toolchain': no such package '@local_config_cc//': key "LIB" not found in dictionary.

[ulfjack]

Any progress? This is killing my presubmits.

[meteorcloudy]

Still haven't found the root cause, but looks like there is already a workaround for the Kokoro presubmit?(cl/150360798) I'll keep digging...

[meteorcloudy]

This issue might have something to do with https://github.com/bazelbuild/bazel/issues/2774
Yet, I could never reproduce this on my local machine.
//cc @laszlocsomor

[meteorcloudy]

@laszlocsomor Can you check if there is similar problem in SkylarkExecutionResult.java ?

[dslomov]

Dupe for #2774 ?

[laszlocsomor]

@dslomov : I'm not convinced of that yet. Are you sure?

[dslomov]

Missed a question mark :)

[laszlocsomor]

Ah :) I'm trying to repro it now.

[laszlocsomor]

@meteorcloudy : re https://github.com/bazelbuild/bazel/issues/2675#issuecomment-291195407: I don't think so, I believe we may be losing the stdout/stderr somehow. Since you were unable to repro this locally, I thought maybe because you didn't redirect the output to a file, whereas on CI it is redirected I believe.

Anyway, I set up an experiment with repo rules: I was building two of them in parallel, a fast and a slow one. It seems repo rules' actions (repository_ctx.execute) cannot be interrupted -- neither with Ctrl+C nor with one of them calling fail().

@dslomov : because of the reasons above, I doubt this bug is a dupe of #2774.

[hlopko]

Is this still expected to be fixed in 0.5? We still have some release blockers and we will not cut the release sooner than in 2 weeks, but we should be getting ready to.

[dslomov]

I do not think we have a repro, so we do not know if it is fixed or not. I do not think we need to treat it as a release blocker.

[laszlocsomor]

I've added stricter error checks and more logging to cc_configure.bzl in hopes of catching this bug the next time it appears. We don't have a repro, sadly.

[hlopko]

Ok thanks, not treating as release blocker. Good luck! :)

[damienmg]

Let's move it to 0.6

[snnn]

I can reproduce it on 688dbf7a71d2c2c2a092ab77ddaa2e2bfdebef8a in a CI system. As @meteorcloudy , I could never reproduce this on my local machine.

[meteorcloudy]

I finally figure out the root cause, it's a one-liner fix. I'll send it soon.

[snnn]

Hi @meteorcloudy , How is it going? Is it solved?

[meteorcloudy]

Well, it turned out to be much more complicated that I thought. But yes! I have a fix under review, it will be fixed soon. https://bazel-review.googlesource.com/#/c/bazel/+/18790/