Bazel: Release 0.29 - August 2019 (stable)

[dslomov]

Target RC date - August 1st, 2019.
This release will have no breaking changes.

See the blog post for some details

[or-shachar]

Is there a release candidate available?

[jin]

@or-shachar the target RC date is in August.

[katre]

@or-shachar I plan to cut the release on Thursday morning (NYC time).

[lberki]

https://github.com/bazelbuild/bazel/issues/6790, deemed a requirement for Bazel 1.0, requires an incompatible flag that has not been submitted yet.

@scentini @katre please make sure the release is cut after the flag lands.

[lberki]

https://github.com/bazelbuild/bazel/issues/9006, deemed a requirement for Bazel 1.0, requires an incompatible flag that has not been submitted yet (but is under review!)

@brandjon @katre please make sure the release is cut after the flag lands.

[dslomov]

Let's postpone the cut until we have clarity on platform situation.

[dslomov]

Re platforms: we decided to make no changes to platforms that need to get in 0.29
What is the status of #6790? /cc @scentini @lberki
What is the status of #9006? /cc @brandjon @lberki

[brandjon]

9006 is blocked on #9019, which is progressing well. It looks like it just needs to be broken up into multiple PRs (for third_party/ workflow reasons). If there aren't any surprises it'll be submitted today.

[katre]

At this point waiting for tomorrow (August 2) before cutting the release.

@scentini and @dslomov, please let me know whether I need to wait for #6790, or just cherrypick the fix later.

[brandjon]

The third_party/ submit dance for #9006 has concluded and the incompatible flag has been added. I've marked that issue migration-ready.

[lberki]

As per our discussion the day before yesterday, #6790 won't make it into 1.0 (right, @scentini ?)

[scentini]

Yes, #6790 won't make it into 1.0.

[katre]

Evaluating 2d5b9f35a859bbeec4df66fc6c2256db299dd56a as a possible baseline: I've started a test run to validate it while I start checking over issues.

[katre]

Test run with 2d5b9f35a859bbeec4df66fc6c2256db299dd56a as baseline is good: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1117

I am starting the release.

$ scripts/release/release.sh create --force_rc=1 0.29.0 2d5b9f35a859bbeec4df66fc6c2256db299dd56a

[katre]

And the build has failed with python errors building the deb: https://buildkite.com/bazel-trusted/bazel-release/builds/86#5b8ae185-9cb4-46e6-b67e-223b5ea67ce3

Also, Windows has failed with errors creating the MSI: https://buildkite.com/bazel-trusted/bazel-release/builds/86#661817f8-ae61-485b-8869-e06f278d71f7

[katre]

I have fixes ready, waiting on https://github.com/bazelbuild/continuous-integration/pull/783 and https://github.com/bazelbuild/continuous-integration/pull/785

[katre]

Trying again. Since 2d5b9f35a859bbeec4df66fc6c2256db299dd56a is being reverted internally, baseline is now 6c5ef5369a3ffceb8a65cc159a2fff1401242810.

$ scripts/release/release.sh create --force_rc=2 0.29.0 6c5ef5369a3ffceb8a65cc159a2fff1401242810

[katre]

Draft release notes are available for editing.

[katre]

RC2 is available: https://releases.bazel.build/0.29.0/rc2/index.html

[beasleyr-vmw]

I feel 0.29 absolutely needs https://github.com/bazelbuild/bazel/pull/9008 (or some other fix for the same underlying issue). Since 0.26, a single HTTP connection timeout when fetching artifacts terminates the build. (Bazel stopped retrying fetch attempts.)

[artem-zinnatullin]

To add to @beasleyr-vmw's comment, #9015 is equally if not more important as it actually makes Bazel to use alternative url(s) as fallback(s).

[buchgr]

Agreed, #9008 is a regression and needs to be fixed in the 0.29 release. @laurentlb can you review this change please?

[katre]

@buchgr @aehlig @beasleyr-vmw @artem-zinnatullin Please propose specific commits to be cherrypicked and the issues that they resolve.

[artem-zinnatullin]

@katre Proposing following two commits addressing #8974 to be cherry-picked

• https://github.com/bazelbuild/bazel/commit/338829f2633e91ae0492ee4169446465e10b5994 "Fix retrying of SocketTimeoutExceptions in HttpConnector", allows HTTP downloads to recover from SocketTimeoutExceptions HTTP connection/headers phase
• https://github.com/bazelbuild/bazel/commit/14651cd86b6fc1d48f56a208a9b5278b3e2dcf75 "Fallback to next urls if download fails in HttpDownloader", allows HTTP downloads to recover from IOExceptions during HTTP download (any phase) by falling back to secondary url(s)

[buchgr]

~Release Blocker: https://github.com/bazelbuild/bazel/issues/9032~

[laszlocsomor]

~Release blocker: https://github.com/bazelbuild/bazel/issues/9089~

Update: this is not a blocker.

[buchgr]

cherry pick: b7d300c6be3e130dec0e62a4f19493105f595d57 ... this fixes https://github.com/bazelbuild/bazel/issues/9072

[lberki]

Update: #9032 is not a release blocker. It's only that gRPC needs to be updated (for some reason, the breaking change did not break any downstream projects).

[laszlocsomor]

Update: https://github.com/bazelbuild/bazel/issues/9089 is NOT a release blocker. The affected file is not embedded into Bazel.

[aehlig]

@buchgr @aehlig @beasleyr-vmw @artem-zinnatullin Please propose specific commits to be cherrypicked and the issues that they resolve.

• #9008 is committed as 338829f2633e91ae0492ee4169446465e10b5994
• #9015 is committed as 14651cd86b6fc1d48f56a208a9b5278b3e2dcf75

Both together fix #8974. Your call if this regression fix needs to be cherry-picked.

[katre]

Creating RC3 with cherrypicks:
```
$ scripts/release/release.sh create --force_rc=3 0.29.0 6c5ef5369a3ffceb8a65cc159a2fff1401242810 338829f2633e91ae0492ee4169446465e10b5994 14651cd86b6fc1d48f56a208a9b5278b3e2dcf75 b7d300c6be3e130dec0e62a4f19493105f595d57
````

Waiting on tests now.

[katre]

RC3 is available at https://releases.bazel.build/0.29.0/rc3/index.html

Tests are underway: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1127
 

[katre]

RC3 shows errors in rules_docker and rules_nodejs.

[meteorcloudy]

I did a rerun for RC3 at https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1129

The docker failure went away, I believe it's some infra change that has been fixed. I saw the same error with Bazel 0.28.1 and later it's gone.

For the rules_nodejs failure, I noticed it's not failing at HEAD. So I did a bisect, found out 960217631abdcab0a7ed95e2ab10acd55f636639 can fix the problem. We probably need to cherry-pick it into 0.29.0. /cc @lberki can confirm that.

[lberki]

I'm just the merciless sheriff who rolls back breaking changes. /cc @buchgr

[meteorcloudy]

I can confirm the test fails at 0f0a0d58725603cf2f1c175963360b525718a195 but not at its parent commit. So this is also a breaking change in Bazel.

The reason rules_nodejs wasn't failing with RC2 is because the test was only introduced after RC2 was created.

[meteorcloudy]

So please cherry-pick 9602176 to fix the rules_nodejs breakage.

[katre]

@meteorcloudy Thanks very much for debugging this! I'll create rc4 with that cherrypick today.

[katre]

Creating RC4:

$ scripts/release/release.sh create --force_rc=4 0.29.0 6c5ef5369a3ffceb8a65cc159a2fff1401242810 338829f2633e91ae0492ee4169446465e10b5994 14651cd86b6fc1d48f56a208a9b5278b3e2dcf75 b7d300c6be3e130dec0e62a4f19493105f595d57 960217631abdcab0a7ed95e2ab10acd55f636639

Waiting for tests and release stages now.

[katre]

RC4 is available at https://releases.bazel.build/0.29.0/rc4/index.html.

Downstream tests are running: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1130

[katre]

All downstream tests pass.

[cgruber]

Not sure if #9116 is a blocker, but it breaks bazel query in some situations, and I found this behavior from 0.29.0rc4 all the way back to 0.27.0.

[laszlocsomor]

Blocker: https://github.com/bazelbuild/bazel/issues/9106

Though not a regression, this bug prevents enabling Bash-less "bazel run" in Bazel 1.0, see https://github.com/bazelbuild/bazel/issues/8240. There's a fix under review, and if possible I'd like to get it cherry-picked.

[johnedmonds]

https://github.com/bazelbuild/bazel/issues/9115 might also block. We can't build google-cloud-cpp with 0.29.0rc4.

[katre]

@laszlocsomor I agree we should cherrypick the fix for #9106. Let me know when there's a commit.

@johnedmonds #9115 definitely looks bad. I'll wait on @lberki to triage it, however.

[laszlocsomor]

@katre: Thanks for waiting! Please cherry-pick https://github.com/bazelbuild/bazel/commit/da557f96c697102ad787e57bbf7db2460f6a60a8 to fix #9106 .

[katre]

9115 appears to be fixable in google-cloud-cpp, not in Bazel. I'll prepare rc5 on Monday.

[katre]

Creating RC5 (required an edit for the cherrypick on da557f9):

$ scripts/release/release.sh create --force_rc=5 0.29.0 6c5ef5369a3ffceb8a65cc159a2fff1401242810 338829f2633e91ae0492ee4169446465e10b5994 14651cd86b6fc1d48f56a208a9b5278b3e2dcf75 b7d300c6be3e130dec0e62a4f19493105f595d57 960217631abdcab0a7ed95e2ab10acd55f636639 da557f96c697102ad787e57bbf7db2460f6a60a8

Waiting for https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1139

[katre]

All downstream tests pass: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1139

[katre]

Created rc6:

$ scripts/release/release.sh create --force_rc=6 0.29.0 28f7d8f2d2aa46ec2fa981e6a6c90bf48111747f ef8b6f68cc8ffd2e6523a894034ae383e87ec74c

Waiting for tests: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1143

[steeve]

Kindly asking for the fix to https://github.com/bazelbuild/bazel/issues/9078 to be included in the release when it's solved, if possible

[katre]

@steeve: I don't think #9078 is a release blocker, since we aren't yet telling people to enable --incompatible_enable_cc_toolchain_resolution. One of the blockers is, specifically, iOS toolchain selection, which is the heart of the issue.

Your workaround is to continue using the legacy cc toolchain selection mechanism.

[katre]

Given the lack of new bug reports since I sent out RC6, I plan to release Bazel 0.29.0 tomorrow, August 19th. Please update this issue if you know of a release blocker.

[zegl]

What is the expectations of "no breaking changes"? Is it that all projects that builds with 0.28.1 also should build with 0.29.0?

If that's the case, please take a look at https://github.com/zegl/bazel-0-29-incompat-repro. It contains a project that does build with 0.28.1, but not with rc6.

---

Update: I've narrowed the incompatibility down to io_grpc_grpc_java at v1.20.0, it does build with 0.28.1, but not rc6.

[katre]

For the "no breaking changes" promise, we specifically mean "no incompatible flags will have their default value changed". If there are further regressions we will try to find and mitigate them. I will look into the issue with io_grpc_grpc_java.

[Yannic]

This is the same error that happened in https://github.com/bazelbuild/bazel/issues/9032#issuecomment-518631754 with gRPC.

[katre]

@Yannic Thanks for pointing this out.

@zegl Given that there is a patch in grpc (https://github.com/grpc/grpc/pull/19860), I don't think there's anything further we need to do.

[steeve]

The protobuf issues are because of https://github.com/bazelbuild/bazel/issues/7157. Most, but not all repos have been updated.
For instance, I just opened a PR for rules_swift: https://github.com/bazelbuild/rules_swift/pull/288

Nothing to do on the bazel side.

[Yannic]

0.29 removes --incompatible_disallow_load_labels_to_cross_package_boundaries which was available in 0.28 (defaulted to true). Is this allowed under the incompatibility policy?

https://github.com/bazelbuild/bazel/commit/4efb6c2563b7d96bfbd9b30175f1ab9b969708da

[katre]

@dslomov, please clarify the policy for removing incompatible flags.

[dslomov]

0.29 removes --incompatible_disallow_load_labels_to_cross_package_boundaries which was available in 0.28 (defaulted to true). Is this allowed under the incompatibility policy?

Yes. The --incompatible_ flags themselves are not subject to the policy. Once they have been flipped to on, they can be immediately removed. That is why our guidance is to never rely on --incompactible_* flags for production builds

[laurentlb]

What is the expectations of "no breaking changes"?

Projects that build with 0.28 should build with 0.29, assuming you don't pass any incompatible_ or experimental_ flag.

By this definition, 0.29 is not compatible with 0.28, as shown by https://github.com/zegl/bazel-0-29-incompat-repro

$ USE_BAZEL_VERSION=0.28.1 bazelisk build ...
(works)

$ USE_BAZEL_VERSION=0.29.0rc6 bazelisk build ...
ERROR: /tmp/repro/bazel-0-29-incompat-repro/src/BUILD:22:1: Action src/service_java_grpc-proto-gensrc.jar failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-grpc-java=bazel-out/host/bin/external/io_grpc_grpc_java/compiler/grpc_java_plugin' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
google/protobuf/timestamp.proto: File not found.

I'll defer to @dslomov or @katre to decide whether it needs a cherrypick.

[dslomov]

What is the expectations of "no breaking changes"?

Projects that build with 0.28 should build with 0.29, assuming you don't pass any incompatible_ or experimental_ flag.

By this definition, 0.29 is not compatible with 0.28, as shown by https://github.com/zegl/bazel-0-29-incompat-repro

$ USE_BAZEL_VERSION=0.28.1 bazelisk build ...
(works)

$ USE_BAZEL_VERSION=0.29.0rc6 bazelisk build ...
ERROR: /tmp/repro/bazel-0-29-incompat-repro/src/BUILD:22:1: Action src/service_java_grpc-proto-gensrc.jar failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-grpc-java=bazel-out/host/bin/external/io_grpc_grpc_java/compiler/grpc_java_plugin' ... (remaining 4 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
google/protobuf/timestamp.proto: File not found.

I'll defer to @dslomov or @katre to decide whether it needs a cherrypick.

Thanks, it does look bad indeed. @lberki or @hlopko, do you know what is going on there?

[steeve]

@dslomov most likely because of https://github.com/bazelbuild/bazel/issues/7157, which needed patching rules for proto libraries

[katre]

Given this discussion, I am not going to release 0.29.0 today, while we figure out what needs to be updated.

[thomasvl]

Looks like a few PR are being filed against things for protos for different languages, given the original statement of _This release will have no breaking changes._, this doesn't appear to be the case.

Where are the proto changes documented so rule authors can update things? The PRs with references to _virtual_imports_ is all something I can't say I've run into before, and I don't see anything in the release notes to find out more.

[brandjon]

Updated the draft release notes doc to no longer say the rules python flag is being flipped in 1.0. (Not sure if changes are still being pulled in from that doc.)

[katre]

@brandjon Yes, please continue to update the release notes until I change the status from "DRAFT" to "PUBLISHED".

[lberki]

@thomasvl , I am the culprit! I reshuffled a bit proto_library works and was confident in my judgement that it's not a user-visible change. Then it turned out that gRPC relied on the exact old source tree layout.

The fix is https://github.com/grpc/grpc/commit/e2ba3aa07009292617c3cabe734e8e44099b22ac and I'm ambivalent about it: we didn't change the spec but what if someone depends on implementation details? If we change implementation details, is it a breaking change?

[lberki]

...actually, the fix is https://github.com/grpc/grpc-java/commit/b22017851560197a41015acd90f443f7b9519984 . I'm looking at why https://github.com/zegl/bazel-0-29-incompat-repro does not pull that in.

[lberki]

Got it. It appears that the above repository pulls the version of grpc-java specified here:

https://github.com/stackb/rules_proto/blob/master/deps.bzl#L327

which is https://github.com/grpc/grpc-java/commit/3c24dc6fe1b8f3e5c89b919c38a4eefe216397d3, a version from 2019 March 9. So I'm afraid the only answer I have is either to patch https://github.com/grpc/grpc-java/commit/b22017851560197a41015acd90f443f7b9519984 or update your grpc-java and apologize for the unintended breakage :(

[dslomov]

So I'm afraid the only answer I have is either to patch grpc/grpc-java@b220178 or update your grpc-java and apologize for the unintended breakage :(

I don't think that a good enough answer. If a fix for #7157 is a breaking change, it should have followed the policy. Is there a way to make it so? (If there is, I am willing to accept a cherry-pick to 0.29)

[lberki]

Yeah, thanks for keeping me honest :)

I think you have the wrong link (https://github.com/bazelbuild/bazel/pull/7175 is a random pull request), but I'll cook something up. Fortunately, it'll be easily cherry-pickable if the gods don't frown upon me today.

[dslomov]

I meant #7157 (updated)

[lberki]

Well, that'll be collateral damage :( I will roll forward the fix as soon as it lands so that the time frame for regressing on that is as small as possible.

[katre]

Once @lberki submits the rollback of behavior, I'll cut a new RC and we can test that out.

Thanks, everyone!

[lberki]

Proto rollback here: https://github.com/bazelbuild/bazel/commit/209175ff8ffeb05628ed8a187dd414a3d2935c55

[katre]

Created rc7:

$ scripts/release/release.sh create --force_rc=7 0.29.0 ad9f7608230dade9d00f694cfad903684dbcc627 209175ff8ffeb05628ed8a187dd414a3d2935c55

Waiting for release jobs and tests.

[thomasvl]

So we should close https://github.com/bazelbuild/rules_swift/pull/288 as not need thanks to the rollback?

Is this change going to happen again? Any docs on what it will be some rule teams can read up/understand it in advance?

[thomasvl]

ps - should changes to the grpc repos also get rolled back then?

[katre]

Downstream testing for rc7: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1155

[katre]

rules_go is failing with an error related to proto import paths: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1155#3154f4a3-c377-4ed1-a9cc-0fdb27008fb1

@lberki is this due to your commit, and if so, does this mean we need to roll back changes in rules_go or elsewhere?

[katre]

Confirmed: the rules_go failure is due to proto_library failing when "--incompatible_generated_protos_in_virtual_imports" is disabled.

[katre]

I've filed https://github.com/bazelbuild/bazel/issues/9219 as a release blocker while I investigate the problem.

[katre]

@lberki is finding a fix for #9219: once that is ready I'll create rc8.

[meteorcloudy]

Please waiting for a fix for https://github.com/bazelbuild/bazel/issues/9222

[meteorcloudy]

@katre Please cherry-pick https://github.com/bazelbuild/bazel/commit/644060b7a4bc98384b66e3d2343b950b875b5e35 to fix #9222 , thanks!

[aehlig]

Another relevant fix is pending as https://bazel-review.googlesource.com/c/bazel/+/111374/

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[dslomov]

So we should close bazelbuild/rules_swift#288 as not need thanks to the rollback?

Is this change going to happen again? Any docs on what it will be some rule teams can read up/understand it in advance?

The change will now be guarded under --incompatible_generated_protos_in_virtual_imports that we expect to flip in the next release. Please follow up on #9215 if migration steps are not clear.

[katre]

Waiting for https://bazel-review.googlesource.com/c/bazel/+/111374/ before I create RC8.

[steeve]

@thomasvl the PR should be safe to apply wether or not the flag is flipped, and I can confirm it's working (currently using rc6 atm, can't test on rc7 because of rules_go breakage)

[aehlig]

Waiting for https://bazel-review.googlesource.com/c/bazel/+/111374/ before I create RC8.

https://bazel-review.googlesource.com/c/bazel/+/111374/ has been submitted as 76ed014e77d7b862f6eb2894600ae525ea570f11.

[katre]

Created RC8:

$ scripts/release/release.sh create --force_rc=8 0.29.0 a4e7a7a5235f415eea5515b49121ec79a95b78ad 644060b7a4bc98384b66e3d2343b950b875b5e35 067040d7bcb3b24a88432e210a96adacee3f37b4 76ed014e77d7b862f6eb2894600ae525ea570f11

Waiting for tests to complete the release process.

Assuming no new issues, this needs to bake until next Tuesday (August 27) before it can be released.

[katre]

RC8 is available for testing: https://releases.bazel.build/0.29.0/rc8/index.html

Downstream tests are running: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1159

[katre]

Downstream tests all pass.

[katre]

Barring any last minute release blockers, I plan to finish the release of 0.29.0 later today.

[katre]

Still working to finalize the release announcement: at this point the release is probably tomorrow.

[katre]

Starting the release.

[katre]

The release has now been pushed to https://github.com/bazelbuild/bazel/releases/tag/0.29.0

@vbatts, @petemounce, and @excitoon, can you please update your builds?

[katre]

The blog post is available, and docs should be propagating shortly.

[vbatts]

Fedora & Centos build:
https://copr.fedorainfracloud.org/coprs/build/1023211

[jmillikin-stripe]

While upgrading a build from 0.27 to 0.29, I discovered there was a regression in v0.28 for git_repository pulling at a tagged commit. The fix is merged in https://github.com/bazelbuild/bazel/pull/9059, would it be possible to cherry-pick this into v0.29.1 ?

[petemounce]

Pushed to chocolatey.

[katre]

@jmillikin-stripe Please open an issue and assign it to me so I can track fixing the regression.

[katre]

Filed #9293 to track the 0.29.1 point release: since 0.29.0 is finished I am closing this issue.

[laszlocsomor]

Thanks @petemounce !