Bazel: Release 3.7 - October 2020

[philwo]

Status of Bazel 3.7

• Target baseline: 2020-10-01
• Expected release date: 2020-10-15
• List of release blockers

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:

• [x] Update GitHub issues for incompatible changes
• [x] Pick release baseline: a991db7c2f66a354666388d888dcef9b0d0f70c0
• [x] Create release candidate: https://releases.bazel.build/3.7.0/rc1/index.html
• [x] Check downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1688
• [x] [Create draft release announcement](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit)
• [x] Send for review the release announcement PR: https://github.com/bazelbuild/bazel-blog/pull/240
• [x] Push the release, notify package maintainers:
• [x] Update the documentation
• [x] Push the blog post
• [x] Update the release page

[philwo]

We need these commits in the release to fix an error regarding invalid authentication credentials when using remote execution:

• https://github.com/bazelbuild/bazel/pull/12106
• https://github.com/bazelbuild/bazel/pull/12156

[katre]

Due to https://github.com/bazelbuild/bazelisk/issues/170, we should not have multiple releases in flight at once. I will delay cutting the first RC for 3.7 until 3.6.0 finishes release, or until Bazelisk is fixed and released.

[katre]

3.6.0 is now released, so I will start preparing 3.7.0 today.

[katre]

Last night's downstream run was green (https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1687), using a991db7c2f66a354666388d888dcef9b0d0f70c0 as the baseline.

[katre]

Creating RC1:

$ scripts/release/release.sh create --force_rc=1 3.7.0 a991db7c2f66a354666388d888dcef9b0d0f70c0
From github.com:bazelbuild/bazel
 * branch                  HEAD       -> FETCH_HEAD
Creating new release branch release-3.7.0rc1 for release 3.7.0
Switched to a new branch 'release-3.7.0rc1'
Applying cherry-picks
Created 3.7.0rc1 on branch release-3.7.0rc1.

[katre]

RC1 is available: https://releases.bazel.build/3.7.0/rc1/index.html

[philwo]

arm64 binaries have been uploaded for Bazel 3.7.0rc1!

[katre]

Downstream tests are starting: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1688

[katre]

Target release date is currently October 20, 2020, assuming no release blockers in the meantime.

[katre]

Downstream tests all pass.

[katre]

Finalizing the release today.

[davido]

@katre there is no way we can cherry-pick https://github.com/bazelbuild/bazel/commit/ffd088f29a59aa2fbc291c09157427bf677b79e3 any more in this release?

(It's not a regression, but upgrade to new java_tools release that includes java toolchain with Java 15 support.)

[katre]

Release has been pushed

[katre]

@vbatts @petemounce @excitoon: Can you update your packages? Release page is https://github.com/bazelbuild/bazel/releases/tag/3.7.0

@philwo: Can you create the arm64 binaries?

[katre]

@davido: You managed to ask this right after I started the release process, so it was already too late for a cherrypick, regardless of whether we want to or not.

Is this something that we can't wait for the next release in November for? We could in theory create a 3.7.1 patch, but typically we only do that for regressions.

[katre]

Blog post is live: https://blog.bazel.build/2020/10/20/bazel-3-7.html

[philwo]

arm64 binaries will follow in a few minutes.

[vbatts]

fedora and centos build-in-progress:
https://copr.fedorainfracloud.org/coprs/build/1714006

On Tue, Oct 20, 2020 at 10:39 AM Philipp Wollermann <
[email protected]> wrote:

arm64 binaries will follow in a few minutes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/12188#issuecomment-712901810,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAQL2P6UQDZMIGQMNGLFM3SLWOJXANCNFSM4R5RXMGA
.

[vbatts]

all passed, even arm64!

[philwo]

Linux arm64 binaries uploaded!

[beasleyr-vmw]

Will there need to be a 3.7.1?

• error-prone bug in ComparableType check #12270 -- already fixed on master
• cannot build Bazel 3.7.0 from source: ./src/main/tools/logging.h:44:27: error: expected ')' before 'PRId64' #12327 -- I pasted a patch but can generate a PR

[wrowe]

If there is a 3.7.1+, please consider that LLVM 11.0.0 breaks bazel clang-cl builds on windows due to malformed include and library paths, caused by an error parsing the output of the of the clang-cl -v command as of 11.0.0 (10.0.0 -v output does not exhibit this issue). These two PR's https://github.com/bazelbuild/bazel/commit/ea763e53fdf850b4290b09870adf42b254c73935 and https://github.com/bazelbuild/bazel/commit/e6751ed25615e0ce6e5ecee68dd6010b2de2225c in 4.0.0 fixed this defect.

[katre]

@wrowe Is there an issue open for this? We can decide whether it's worth a point release.

[wrowe]

@katre asks;

@wrowe Is there an issue open for this?

https://github.com/bazelbuild/bazel/pull/12458 is the PR behind the request.
https://github.com/bazelbuild/rules_cc/pull/88 is the underlying defect, I hadn't opened an issue, per se.

If Bazel project prefers Issue -> PR workflows, we can certainly do that. Wasn't clear if this would be a bazel or rules_cc ticket.
I'm experimenting with a workaround of bumping only rules_cc within our use case at envoy to see if the defect is present when we simply build the toolchain, or can be mitigated at build time.

[wrowe]

@katre asks;

@wrowe Is there an issue open for this?

12458 is the PR behind the request.

bazelbuild/rules_cc#88 is the underlying defect, I hadn't opened an issue, per se.

We can decide whether it's worth a point release.

I believe you might wish to do so. Whether using msvc cl.exe or llvm clang-cl.exe, when bazel heuristically detects llvm, it generates the corrupted LIB and INCLUDE environment strings, illustrated here;
https://github.com/envoyproxy/envoy/pull/13688#issuecomment-729231947

This renders bazel inoperable on any Windows PC with llvm 11.0.0 installed.

[katre]

Thanks, I'll try and cut a patch release today.

[katre]

Current cherrypick candidates:

• 8cec289ace6de2d54dfd78100ee2012d6028d8b1: fix for errorprone check
• ea763e53fdf850b4290b09870adf42b254c73935: update rules_cc reference
• e6751ed25615e0ce6e5ecee68dd6010b2de2225c: rules_cc supports clang llvm 11 on windows

There was a request for https://github.com/bazelbuild/bazel/issues/12327, but it hasn't yet been merged.

[katre]

Creating RC1:

$ scripts/release/release.sh create --force_rc=1 3.7.1 3.7.0 8cec289 ea763e5 e6751ed

Both 8cec289ace6de2d54dfd78100ee2012d6028d8b1 and e6751ed have conflicts which I merged.

[katre]

And my fix for conflict re-added a file, which caused srcs_test to fail, so now there is an RC2 with a commit to re-remove the file.

[katre]

Artifacts are deployed to https://releases.bazel.build/3.7.1/rc2/index.html
Downstream tests are at https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1771

[wrowe]

@katre, I do appreciate the credit, but please note I only co-authored the underlying fix with Sunjay Bhatia (@sunjayBhatia), if you are going to credit the final point release.

I've run rc2 through the paces for the envoy windows msvc-cl and clang-cl builds with llvm 11.0.0 in place, and everything is looking good here (the project was previously anchored at bazel 3.6, so this is very good news!)

Thanks for the rapid response!

[katre]

Downstream tests look good, but I am looking into the Envoy failures.

[katre]

Looks like they have the wrong checksum, so it's unrelated.

[philwo]

FYI, another cherry-pick request for a regression in Bazel 3.7.0 via bazel-discuss for https://github.com/bazelbuild/bazel/commit/02838a1b2aa2f6d03980536ab2ac6840c3c98e84.

[katre]

Creating RC3:

$ scripts/release/release.sh create --force_rc=3 3.7.1 e055b433efdccb28b9c21082e72d8e79d9b34e0f 02838a1b2aa2f6d03980536ab2ac6840c3c98e84

[katre]

RC3 is available: https://releases.bazel.build/3.7.1/rc3/index.html

Downstream tests are running: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1776

[katre]

Downstream tests are good except for bazelisk failures, which I am investigating.

[katre]

The bazelisk error seems to be a flake in the binary publishing pipeline, and unrelated to the release.

[katre]

Downstream tests are now green.

I plan to finalize 3.7.1 on Tuesday if nothing else comes up.

[lizan]

https://github.com/bazelbuild/bazel/issues/12545 is a regression that we are experiencing in Envoy and a blocker for us to upgrade to 3.7.x. cc @wrowe

[katre]

@lizan I don't think #12545 can be fixed in time for a 3.7 release. I'll try to get it to the right people's attention and evaluate whether it's a release blocker and the fix can be cherrypicked into 4.0, however.

[katre]

I'm going to finalize 3.7.1 today, but please make sure to ping the issue for 4.0 when #12545 is fixed.

[katre]

Release 3.7.1 is ready for use: https://github.com/bazelbuild/bazel/releases/tag/3.7.1

@vbatts @petemounce @excitoon, can you update your packages?
@philwo, can you add the arm binaries?

Thanks everyone.

[wrowe]

Was there something special to be done, a PR or whatnot, to tickle rbe-toolchains-robot to refresh https://github.com/bazelbuild/bazel-toolchains/releases/ post-3.7.1 release?

[katre]

@smukherj1, are you still involved with rbe-toolchains-robot? Do you know who is?

[vbatts]

Fedora and Centos builds are kicked off
https://copr.fedorainfracloud.org/coprs/build/1792784

[smukherj1]

@smukherj1, are you still involved with rbe-toolchains-robot? Do you know who is?

@DaveGay is looking into publishing configs

[katre]

Thanks!

[smukherj1]

Note that toolchain configs already exist for Bazel 3.7.0 and in theory Bazel 3.7.1 should be compatible with configs generated for 3.7.0.

While we work to release 3.7.1 configs (ETA 1 day), @wrowe can you try using the 3.7.0 release of the toolchain configs repo with Bazel 3.7.1?

[wrowe]

While we work to release 3.7.1 configs (ETA 1 day), @wrowe can you try using the 3.7.0 release of the toolchain configs repo with Bazel 3.7.1?

For posterity, we were in an unusual situation here, it wasn't possible to use the 3.7.0 or even 3.6.0 toolchain due to early env overrides inflicted by the llvm detection bug in the clang-cl rules_cc tooling. And, it wasn't hacked to follow usual bazel conventions, so loading any toolchain prior to 3.7.1 against llvm 11.0.0 was an absolute failure. With the 3.7.1 toolchain in place, it's now possible for us to proceed, typical changes to the toolchain don't cause us grief in the envoy bazel build.

[wrowe]

As @lizan notes in https://github.com/bazelbuild/bazel/issues/12188#issuecomment-732813900 the 3.7.0 and 3.7.1 releases remain broken and unusable by the envoy project, and a fix exists. We were operating on the assumption that 4.0.0 would happen this year, but that seems highly unlikely. Would a 3.7.2 regression-fix release be realistic in the coming days?

[philwo]

@wrowe Can you please link to the commit that you need as a regression fix for 3.7.2?

[smukherj1]

While we work to release 3.7.1 configs (ETA 1 day), @wrowe can you try using the 3.7.0 release of the toolchain configs repo with Bazel 3.7.1?

For posterity, we were in an unusual situation here, it wasn't possible to use the 3.7.0 or even 3.6.0 toolchain due to early env overrides inflicted by the llvm detection bug in the clang-cl rules_cc tooling. And, it wasn't hacked to follow usual bazel conventions, so loading any toolchain prior to 3.7.1 against llvm 11.0.0 was an absolute failure. With the 3.7.1 toolchain in place, it's now possible for us to proceed, typical changes to the toolchain don't cause us grief in the envoy bazel build.

@wrowe This part is unclear to me. My understanding was Envoy only uses the config generation capabilities of bazel-toolchains, not the pre-generated configs hosted by the repository for the Ubuntu 16.04 based clang + JDK container. The config generation capabilities are implemented by Bazel rules and these don't change when we "release" a new version of the bazel-toolchains repository for a new Bazel version. The new release only adds pre-generated configs for the new Bazel version targeting the Ubuntu 16.04 based container. My understanding was that Envoy doesn't use the RBE maintained Ubuntu 16.04 container and thus the pre-generated configs aren't used.

TL; DR My understanding is Envoy shouldn't need to wait for a new release of the bazel-toolchains repo when upgrading the Bazel version used in the Envoy build.

so loading any toolchain prior to 3.7.1 against llvm 11.0.0 was an absolute failure

Could you provide details on what error you saw here?

[sunjayBhatia]

The issue was with the rules_cc packaged with Bazel itself that is used to generate windows tool chain configs, not bazel toolchains, possibly overloaded uses of “toolchain” here

[wrowe]

My understanding is Envoy shouldn't need to wait for a new release of the bazel-toolchains repo when upgrading the Bazel version used in the Envoy build.

We did test against the combination of the corrected bazel rules_cc at master's current tag with the then-released (3.7.0) and came up empty, the env table is corrupted by the invocation of bazel itself ahead of processing rules.cc, unless both a new bazel build (I'd tested and confirmed 3.7.1rc2 ahead of release) combined with the current ruleset. Even with that combination, a refreshed bazel-toolchains appeared to be necessary for our tooling to generate our resulting envoy bazel toolchains package against llvm 11.0.0, due to the formatting assumption in old rules_cc. We can't get as far as evaluating our checkout of the preferred bazel toolchains, owing to this bug in rules_cc embedded in bazel.exe. Once we have a good flavor of bazel.exe, we must then checkout the right rev containing the fix to rules_cc in our repository_locations.bzl, or again, environment becomes corrupted before processing our BUILD rules.

Again, this is a very odd edge case, generally rules will transform the environment table later in the process, generally it will not be outright corrupted, generally the last combination of bazel/rules_cc/envoy's spec can put together envoy's next toolchain with no issues whatsoever. Sorry this was an irritating total-failure exception case when llvm 11.0.0 was detected by bazel, and I'm already verifying that bazel 4.0.0 w/ toolchains 3.7.1 you generated works out just fine for the next bump, without the urgency of tagging again.

[wrowe]

@philwo https://github.com/bazelbuild/bazel/commit/a689d673abadf80f1efaf8ddaeee92d56fc2847b fixes the 3.7.x regression, but tagging @lizan and @gyias to ask them to verify there wasn't a second required fix-of-fix to stabilize 3.7.2, as sometimes happens. This is already on 4.0.0-rc.

Regression at issue; https://github.com/bazelbuild/bazel/issues/12545

[katre]

I can cut a 3.7.2 and do a short RC period next week, but we are getting close to our holiday freeze, so I may need to check what's appropriate there.

[gyias]

@wrowe Not that I'm aware of. I checked the fix with the repro example @lizan provided and also some more complicated ones I created myself, and everything ran fine.

[wrowe]

I can cut a 3.7.2 and do a short RC period next week, but we are getting close to our holiday freeze, so I may need to check what's appropriate there.

@katre ... understood and thanks.

I'm only thinking about the scope of roadblocks to adopting 3.7 from 3.6 (perhaps earlier) for those projects such as envoy with very early January releases on the calendar. 4.0 will get here in January, but that will be too late for some projects to catch up to current bazel.

[katre]

@philwo Will a 3.7.2rc1 conflict with your RCs for 4.0.0?

[philwo]

@katre No, I think that should be fine.

[katre]

Created 3.7.2rc1:

$ scripts/release/release.sh create --force_rc=1 3.7.2 3.7.1 a689d673abadf80f1efaf8ddaeee92d56fc2847b
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Total 9 (delta 8), reused 9 (delta 8), pack-reused 0
Unpacking objects: 100% (9/9), 868 bytes | 124.00 KiB/s, done.
From github.com:bazelbuild/bazel
 * branch                  HEAD       -> FETCH_HEAD
Creating new release branch release-3.7.2rc1 for release 3.7.2
Switched to a new branch 'release-3.7.2rc1'
Applying cherry-picks
  Cherry-picking a689d673abadf80f1efaf8ddaeee92d56fc2847b
Created 3.7.2rc1 on branch release-3.7.2rc1.

Unfortunately, the release process failed due to a problem generating the arm64 binaries (https://github.com/bazelbuild/continuous-integration/issues/1067), so the actual RC will be out on Monday.

[katre]

We have an RC: https://releases.bazel.build/3.7.2/rc1/index.html

[katre]

Downstream tests finished: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1815

The Tulsi failure is the same as at head, but I'll investigate the rules_apple failures.

[hrfuller]

Is it possible to cherry pick https://github.com/bazelbuild/bazel/commit/362dbeb9f15731837040ad4160233fc61eafb3d0 into a future release candidate on 3.7. This is a critical change for many including myself, and it would be nice to not have to block on upgrading to bazel 4.0 to get this change.

[katre]

@hrfuller Unfortunately, 362dbeb does not fix a regression, so it's not a candidate for a patch release.

[katre]

Local testing (thanks @susinmotion!) can't reproduce the rules_apple failures. I will go ahead an finalize the 3.7.2 patch release Thursday unless any release blockers appear.

[wrowe]

@katre, testing against 3.7.2rc1 at Envoy shows no new regressions, issues known to us are all resolved. TYVM.

[katre]

Release 3.7.2 is out now: https://github.com/bazelbuild/bazel/releases/tag/3.7.2

@vbatts @petemounce @excitoon, can you update your packages?

[katre]

Homebrew update is in progress: https://github.com/Homebrew/homebrew-core/pull/67060