Bazel: Release 0.26 - May 2019

[laurentlb]

Target RC date: May 2nd

[dslomov]

We would like to make sure that at least experimenal implementation of #8201 is in this release.

[aehlig]

We would like to make sure that at least experimenal implementation of #8201 is in this release.

When do you expect this implementation to be committed to the bazel code base?

[brandjon]

I'd like to flip --incompatible_use_python_toolchains in this release, but I ended up being blocked on #5104, which would mean that flipping this flag will break users of --build_python_zip who are not currently overriding the default Python runtime. Not sure how I'll resolve this yet, will update here...

[sergiocampama]

I'd like to flip --incompatible_disable_objc_provider_resources, and I'm trying to submit this change before the May 2nd deadline, would it be fine to cherrypick if I don't make it by the 2nd?

EDIT: This has been submitted on May 1st.

[brandjon]

Submitted the flag flip for Python toolchains, so if the baseline is cut after today then I'm good.

[dslomov]

We would like to make sure that at least experimenal implementation of #8201 is in this release.

When do you expect this implementation to be committed to the bazel code base?

In the coming days.

[aehlig]

Current state of our downstream projects.

• The last fully green nightly is from Apr 17, 2019
• The nightly from May 1st is pretty good, with only "BUILD file generator" failing, "rules_gwt" broken, and "TensorFlow" failing to clone.

[brandjon]

Apparently my flag flip broken a mac test in postsubmit. Working on a simple forward fix to disable the flip for that test now...

[aehlig]

Apparently my flag flip broken a mac test in postsubmit. Working on a simple forward fix to disable the flip for that test now...

Looking at today's nightly, a lot of breakage seems python related. So, rolling back seems more appropriate, unless you verify that your forward fix fixes the python-related downstream breakages as well. With that large number of downstream breakages, we're stuck with the nightly from May 1st (i.e., commit baefeabefc64bb30c293a3dfa9e45a0017b7696b) anyway.

[brandjon]

Sadly yes. I'll roll back my flag flip and work on downstream fixes.

[brandjon]

Rolled back by 4837e11bdd5f243d2b10f6a2b99f2edaba8c1131, so you can cherrypick that or cut the baseline after.

[aehlig]

The current nightly has all the breakages of the one from May 1st and, on top of that, the ones tracked by #8227. So the the canonical base line would be baefeabefc64bb30c293a3dfa9e45a0017b7696b which we should take according to our policy. However, I was ordered to delay the release to wait for #8201.

[dslomov]

For context #8201 is critical for the https://github.com/angular/angular/issues/19058 - Angular team is blocked on this for enabling bazel in their 8.0 release (which is alerady in RC and is going GA late May/early June). I understand that this is somewhat frustrating, but @irengrig and other folks on the team are hard at work getting the last bits in.

[irengrig]

8201 is submitted!!! :)

[aehlig]

To fix the non-determinism uncovered (but not caused) by 29f1932b18aac34a557f42e41638741cbffa67b5, will target as baseline commit https://bazel-review.googlesource.com/c/bazel/+/98615, once in.

[aehlig]

The latest nightly seems alright, at least not worse than the one of May 1st. "BUILD file generator" failing, "rules_gwt" failing to build, and "Tensorflow" failing to clone; the latter is a known issue.

So, let's take daa8ae565ab2023e49134f0aad233b0a8bd7a5d0 as a baseline, and cherry-pick https://bazel-review.googlesource.com/c/bazel/+/98615 once in. Depending on whether there will be a patch release for 0.25.0 (#7498), we might need additional cherry-picks.

[aehlig]

Creating RC1 failed.

ERROR: /workdir/scripts/packages/debian/BUILD:69:1: in _pkg_deb rule //scripts/packages/debian:bazel-debian:
Traceback (most recent call last):
        File "/workdir/scripts/packages/debian/BUILD", line 69
                _pkg_deb(name = 'bazel-debian')
        File "/workdir/tools/build_defs/pkg/pkg.bzl", line 128, in _pkg_deb_impl
                ctx.outputs.deb.path
object of type 'NoneType' has no field 'path'
ERROR: Analysis of target '//scripts/packages/debian:bazel-debian.deb' failed; build aborted: Analysis of target '//scripts/packages/debian:bazel-debian' failed; build aborted

Investigating.

[aehlig]

This is caused by commit bbe47a1564a832e1a175206f2dfbc92af94c120b. I verified that reverting it makes the build of //scripts/packages/debian:bazel-debian.deb succeed. Will try to create a roll back.

[aehlig]

Rollback submitted as c2001a4569483596d9dc74ba9cabcbe4b6f1887f

Between the previous baseline and this commit, there is another rollback which we should include anyway, as releases should include all rollbacks of commits in their baseline, and a commit adding a single test case, which shouldn't do much harm. So I'll bump the baseline to c2001a4569483596d9dc74ba9cabcbe4b6f1887f for my attempt to create RC2.

[aehlig]

RC2 is available at https://releases.bazel.build/0.26.0/rc2/index.html

The release announcement is off though; I'll have to investigate where our release script took the wrong baseline for generating the release notes. Nevertheless, please test the release candidate.

[aehlig]

Will try to create RC3 with same baseline and cherry-picking e67c961905792cd63950c6f6efc33275ad243c49 as this is the fix for a non-determinism problem we observed while searching for a good baseline.

[aehlig]

RC3 is available at https://releases.bazel.build/0.26.0/rc3/index.html

The release notes are still off. It is really bazeline c2001a4569483596d9dc74ba9cabcbe4b6f1887f with cherry-pick e67c961905792cd63950c6f6efc33275ad243c49.

[aehlig]

Tests for downstream projects

• RC2: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/951
• RC3: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/952

[aehlig]

The downstream runs for RC3 look reasonable. Please test RC3 and report regressions before May 14.

[dkelmer]

https://github.com/bazelbuild/bazel/issues/8236 The fix to that will need to be cherry picked into the patch release for 0.25 and the 0.26 release

[petemounce]

rc3 pushed to chocolatey

[aehlig]

Creating RC4 including the fix for #8236.

• Baseline c2001a4569483596d9dc74ba9cabcbe4b6f1887f
• Cherry picks: e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a

[aehlig]

RC4 available at https://releases.bazel.build/0.26.0/rc4/index.html

[aehlig]

The downstream projects with 0.26.0rc4 pass. Please test RC4 and report any issues found.

[laszlocsomor]

RC4 cherry-pick list is broken: https://releases.bazel.build/0.26.0/rc4/index.html

Release notes formatting is also inconsistent:

Incompatible changes:
...
--incompatible_windows_escape_jvm_flags is enabled by default, and the flag no longer exists
--incompatible_no_output_attr_default is enabled by default.

[aehlig]

RC4 cherry-pick list is broken: https://releases.bazel.build/0.26.0/rc4/index.html

Same as for the other RCs. That's why explicitly mentioned base-line and cherry-picks here. I still have not yet found the bug in our release scripts, but that shouldn't stop the release.

[Globegitter]

https://github.com/bazelbuild/bazel/pull/7512 was introduced in bazel 0.25 which contains a bug (https://github.com/bazelbuild/bazel/issues/8052) which just got fixed in master. Does this kind of bug warrant a cherry-pick or will this have to wait for 0.27?

Edit: https://github.com/bazelbuild/bazel/commit/597e289b097d3bfed8eea1cb0924bbeb04877e42 is the commit that fixes the reported issue btw.

[aehlig]

7512 was introduced in bazel 0.25 which contains a bug (#8052) which just got fixed in master. Does this kind of bug warrant a cherry-pick or will this have to wait for 0.27?

That certainly is a regression and warrants a cherry-pick into bazel 0.26. I will create a new release candidate soon; I'm just waiting for the fix for #8254 which is currently under review to be submitted as well.

[aehlig]

./scripts/release/release.sh create --force_rc=5 0.26.0 c2001a4569483596d9dc74ba9cabcbe4b6f1887f e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a 597e289b097d3bfed8eea1cb0924bbeb04877e42 942f7cf6a0da0a4ecc804615424f039e50963933

I.e., RC5, with baseline c2001a4569483596d9dc74ba9cabcbe4b6f1887f and cherry-picks e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a 597e289b097d3bfed8eea1cb0924bbeb04877e42 942f7cf6a0da0a4ecc804615424f039e50963933

[aehlig]

RC5 available here https://releases.bazel.build/0.26.0/rc5/index.html

[jin]

The Android emulator team recently pushed a new stable emulator release that removed some files that Bazel's Android SDK BUILD template looks for (emulator64-x86 and emulator64-arm). Because of this, all Bazel versions will fail when trying to test android_instrumentation_test targets or run android_device rules. The problem is that there is no way for users to download an older version of the emulator from Android Studio or the sdkmanager command line, so users who update their Android emulators will see this breakage with no workaround.

https://github.com/bazelbuild/bazel/issues/8280

I understand that this isn't a regression introduced in core Bazel, but a core native Android rule is broken for all of our users with no straightforward workaround. I wonder if our cherrypick policy can also include situations like this?

This problem affects the rules_jvm_external - examples, Android Testing and nightly downstream pipelines.

The fix is here: https://github.com/bazelbuild/bazel/commit/85a5a2bd569a5274950fc7327a044c395248c024

cc @ahumesky

[aehlig]

I wonder if our cherrypick policy can also include situations like this?

I guess, our policy would say, the Android emulator team should not have done such a thing; but I guess this outside our control...

Anyway, the issue...

The Android emulator team recently pushed a new stable emulator release that removed some files that Bazel's Android SDK BUILD template looks for (emulator64-x86 and emulator64-arm). Because of this, _all_ Bazel versions will fail when trying to test android_instrumentation_test targets or run android_device rules. The problem is that there is no way for users to download an older version of the emulator from Android Studio or the sdkmanager command line, so users who update their Android emulators will see this breakage with no workaround.

...seems really important, and, more importantly, ...

The fix is here: 85a5a2b

...the fix is well-defined in scope, so I'm pretty sure, it will not do harm to uses of bazel that are not android related. And as this release has started by not following policies, we can as well continue this way. I will cherry-pick the fix. But we probably should consider it as a warning to not bundle too specific knowledge in the bazel release binary.

[aehlig]

Creating RC6 with baseline c2001a4569483596d9dc74ba9cabcbe4b6f1887f and cherry-picks e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a 597e289b097d3bfed8eea1cb0924bbeb04877e42 942f7cf6a0da0a4ecc804615424f039e50963933 85a5a2bd569a5274950fc7327a044c395248c024

[aehlig]

RC6 is available here https://releases.bazel.build/0.26.0/rc6/index.html

[meteorcloudy]

Unfortunately we need to cherry-pick another commit: https://github.com/bazelbuild/bazel/commit/9835cb4135503768cdf1161746b95d7969ccb938 to fix supporting multiple package path.
This bug is reported internally and fixed by rolling back the culprit. (b/132327507)
Since Bazel is also affected, we need this cherry-pick.

[aehlig]

Creating RC7 with baseline c2001a4569483596d9dc74ba9cabcbe4b6f1887f and cherry picks e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a 597e289b097d3bfed8eea1cb0924bbeb04877e42 942f7cf6a0da0a4ecc804615424f039e50963933 85a5a2bd569a5274950fc7327a044c395248c024 9835cb4135503768cdf1161746b95d7969ccb938

[aehlig]

RC 7 available here https://releases.bazel.build/0.26.0/rc7/index.html

[aehlig]

Runs of the downstream projects

• for RC6 https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/961
• for RC7 https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/962

[laszlocsomor]

I found a problem with --incompatible_windows_escape_python_args, fix is being submitted. Can I please have its fix cherry-picked (once it's submitted)?

[laszlocsomor]

I found a problem with --incompatible_windows_escape_python_args, fix is being submitted. Can I please have its fix cherry-picked (once it's submitted)?

Submitted. I realize it's not a regression, because 0.26.0 is where we add --incompatible_windows_escape_python_args, but it'd be nice to add it bug-free. Ultimately it's your decision @aehlig .

If you're fine with cherry-picking, then please add https://github.com/bazelbuild/bazel/commit/c963ba21073b514961946b8b4b45b091f08fdaa1.

[aehlig]

c963ba21073b514961946b8b4b45b091f08fdaa1 looks sufficiently limited in scope, so that it seems safe to cherry-pick. Again, as this release has started by not following policies, we can as well continue this way. I will cherry-pick the fix later today.

[aehlig]

Creating RC8, with additional cherry-pick c963ba21073b514961946b8b4b45b091f08fdaa1

[aehlig]

RC8 available here https://releases.bazel.build/0.26.0/rc8/index.html

[aehlig]

Run of downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/970

[irengrig]

I just fixed a bug #8337 for managed directories feature (related to #8201).
Can I please have its fix cherry-picked to 0.26?

[keith]

We're experiencing this crash with rc8 https://github.com/bazelbuild/bazel/issues/8364

[aehlig]

I just fixed a bug #8337 for managed directories feature (related to #8201).
Can I please have its fix cherry-picked to 0.26?

Those things are the very reason, policies don't allow delaying of the release base line to wait for features to come in and instead encourage the addition of the new features to happen on master just after the release branch is created; also, technically, it is not a regression as the feature never worked in an earlier release. But since I was told to treat some features more equal than others, will create a new release candidate with a1ea487e0a9e180a36fa4aab57f7c746ddcf367a as additional cherry-pick.

[irengrig]

Thank you!

[hlopko]

In case there will be another RC, I kindly ask for cherrypicking the fix for https://github.com/bazelbuild/bazel/issues/8330 in https://github.com/bazelbuild/bazel/commit/7dc78cdd04eedf2f4373b170053ba5fc2a990929. This is a bugfix, but not for a recent regression. Having this in 0.26 will allow us to flip toolchains/platforms for C++ rules (https://github.com/bazelbuild/bazel/issues/7260) in 0.27.

Please ignore this if there are no more RCs planned.

[aehlig]

In case there will be another RC [...] Please ignore this if there are no more RCs planned.

No problem. I've only just created RC9 and have to wait for the new CI VM image to be rolled out before I can start the downstream tests; so it doesn't delay anything, to create RC10 right away.

[aehlig]

RC10 is available here https://releases.bazel.build/0.26.0/rc10/index.html but I'm still waiting on news from #8364.

[aehlig]

Run of downstream projects looks good.

[aehlig]

Commit dd9ac13f7e3b71bdf2eca717bc7681bdd12389a2 that was fixing #8364 looks sufficiently limited to be save to cherry-pick. So I'll create rc11 today and aim for a release on Wednesday.

[aehlig]

RC11 available here https://releases.bazel.build/0.26.0/rc11/index.html

[aehlig]

The run of the downstream projects for RC11 looks good. So please test carefully. If no further release blockers are reported till Wednesday (May 22, 2019), this will be the release.

[or-shachar]

@aehlig - I'm having a difficulty installing the RC11 on linux (from the packed installer available at https://releases.bazel.build/0.26.0/rc11/bazel-0.26.0rc11-installer-linux-x86_64.sh)
I get:

/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

Is it documented that I need to install some newer prerequisites?

(I did not change the base image for a while now).

---

$ cat /proc/version

Linux version 4.9.0-0.bpo.4-amd64 ([email protected]) (gcc version 4.9.2 (Debian 4.9.2-10) ) #1 SMP Debian 4.9.51-1~bpo8+1 (2017-10-17)

$ cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
builduser@ip-10-42-156-29-docker00:/tmp$

[gertvdijk]

@or-shachar Yeah, it seems that the libstdc++ requirement has been lifted and it does not run on Debian oldstable anymore (probably also older CentOS affected?).
Moreover, the Debian package does not include the version requirement in the dependencies as well, so it installs, but fails at runtime. Very bad...

I believe the requirement on newer stdlibc++ is unexpected for users and a regression, because the release notes do not mention anything regarding a change on this.

I think we're looking at something similar to #8294.

Release blocker or amending the release notes to exclude older OSs from supported list?

^{Also consider updating to Debian stable, Stretch. Oldstable is old.}

[davido]

@gertvdijk Good catch and it was a good idea to kill support for Debian Jessie and Java 7 on Gerrit Code Review's GerritForge CI one week ago and move entirely to Debian Stretch and Java 8 in: [1].

//CC @lucamilanesio.

• [1] https://gerrit-review.googlesource.com/c/gerrit-ci-scripts/+/207554

[lucamilanesio]

@gertvdijk Good catch and it was a good idea to kill support for Debian Jessie and Java 7 on Gerrit Code Review's GerritForge CI one week ago and move entirely to Debian Stretch and Java 8 in: [1].

//CC @lucamilanesio.

• [1] https://gerrit-review.googlesource.com/c/gerrit-ci-scripts/+/207554

Indeed :-)

[or-shachar]

@gertvdijk - so I must upgrade my base image to stretch?

[gertvdijk]

@gertvdijk - so I must upgrade my base image to stretch?

I was saying two things:

• It seems like a bug to me that Bazel suddenly requires a newer stdlibc++. I don't want to judge, but it's either a release blocker, or it should be communicated well that this new requirement is intentional, but then it's a Debian packaging bug.
• I think everyone still on Jesse should upgrade, unless you have a compelling reason to be using old software? Jessie is old. But that's totally in general, agnostic to what software you're using on Debian.

[aehlig]

Concerning the "Linux binaries", our documentation _never_ claimed they would work on debian, especially not on oldstable. It over-optimistically claims they would work on Ubuntu 14.04, 16.04, and 18.04, even though, in reality they're (still, that will change with the next release) built on Ubuntu 14.04 amd64 and as binaries never tested anywhere else. So they'll work, if your system is sufficiently similar to Ubuntu 14.04 amd64, but for everything else the only recommended way is to compile from source.

[gertvdijk]

Thanks for clarifying @aehlig, but this doesn't really explain why what/this has changed with 0.26. Wouldn't you agree that one may expect to find an entry in the release notes that a runtime dependency has changed for Bazel?

built on Ubuntu 14.04 amd64 and as binaries never tested anywhere else

Also, I've got news for you, it doesn't work on Ubuntu 14.04 anymore as well:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.6 LTS
Release:        14.04
Codename:       trusty
$ bazel version
/usr/bin/bazel: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /usr/bin/bazel)
/usr/bin/bazel: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/bin/bazel)
/usr/bin/bazel: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /usr/bin/bazel)
/usr/bin/bazel: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /usr/bin/bazel)

(This was on a clean ubuntu:trusty Docker image, ubuntu@sha256:2f7c79927b346e436cc14c92bd4e5bd778c3bd7037f35bc639ac1589a7acfa90 at the time of writing, using https://releases.bazel.build/0.26.0/rc11/bazel_0.26.0rc11-linux-x86_64.deb.)

[aehlig]

built on Ubuntu 14.04 amd64 and as binaries never tested anywhere else

Also, I've got news for you, it doesn't work on Ubuntu 14.04 anymore as well:

That is indeed a breakage, caused by an accidental change of our build infrastructure. @philwo already has a pending fix for this. Once this is in, I will build RC12 from the same sources as RC11 to get hopefully correct release binaries.

[gertvdijk]

Hey @philwo, thanks for the commit and info in there (mail on bazel-discuss)! Just wanted to say...

• Looks good and sane to me to drop Ubuntu 14.04! (I am in favour of moving forward, as long as it's communicated.)
• Your _proposal_ did not receive any follow-up.
• The change did not end up in the release notes.

To me this sounds reasonable to cut this in the next release.

[philwo]

@gertvdijk Yes, the current flip to building on Ubuntu 18.04 was just an accident, due to copy & pasting the Docker configuration for our pipeline steps. Bazel 0.26 will be built on Ubuntu 14.04 and thus have the same support as releases before. :)

The mail thread didn't receive any negative feedback, so my plan was to just go ahead with it. I wasn't able to meet the original timeline and didn't want to rush things, so this will take effect a month later with Bazel 0.27. I'll send an e-mail on that thread just to make sure people are aware! Thanks for reminding me.

Bazel 0.27 will then also have an entry in the release notes, that's a good idea.

[aehlig]

RC12 is available at https://releases.bazel.build/0.26.0/rc12/index.html

Note, code-wise this is the same as RC11, however, this time, the linux binaries should be built on Ubuntu 14.04 as it should be the case for this release.

[aehlig]

The downstream projects look good as well. Please test carefully. If no further release blockers will be reported till Thursday (May 23, 2019), this will be the release.

[buchgr]

The fix for this will require a new release candidate, as the bug effectively causes an an infinite loop for remote execution: https://github.com/bazelbuild/bazel/issues/8320

[keith]

If 0.26.0 is supposed to be the migration window for https://github.com/bazelbuild/bazel/issues/7899 (based on the current tags on that issue) I think something will have to be cherry picked to fix https://github.com/bazelbuild/bazel/issues/8414 (here's my pass at this https://github.com/bazelbuild/bazel/pull/8415) cc @brandjon

[brandjon]

Note that #8414 prevents the use of the default Python toolchain (which searches for a Python interpreter on PATH) on Mac. However, you can still define your own custom Python toolchain that knows the path to the Python 2 and Python 3 interpreters on your specific system.

[davido]

As reported in https://github.com/bazelbuild/bazel/issues/7347#issuecomment-494522677 this option --incompatible_disallow_struct_provider_syntax (that is part of --all_incompatible_changes) seems to report breakage in Bazel itself. I've tested on RC12.

To reproduce, run bazel build --incompatible_disallow_struct_provider_syntax //... on Gerrit Code Review.

[keith]

I'm not sure this is an important enough issue but there was a regression related to being able to negate log path flags https://github.com/bazelbuild/bazel/issues/8393 that I have just submitted a fix for https://github.com/bazelbuild/bazel/pull/8428

[or-shachar]

Just checked with rc12 and got the error:

no such attribute 'exec_compatible_with' in 'scala_junit_test' rule

Isn't exec_compatible_with in common attributes?

[aehlig]

Will cherry-pick 0ff19c6d0adf3c0df94fff59ca3bd13cbcf99897 for #8320. For the others who want "something" to be cherry-picked, please specify

• which commit precisely should be cherry picked, and
• what the justification is.

Note that, in general, migration windows can be moved, if the feature is not yet ready for migration.
Thanks.

[or-shachar]

@aehlig - did you notice my issue?

[aehlig]

@aehlig - did you notice my issue?

Well, I've seen your comment. However, in which way is it actionable? I have no idea which code you're looking at, no idea how scala_junit_test is defined. Are you claiming this is a regression with respect to 0.25, or with respect to an earlier rc?

[laurentlb]

@or-shachar Can you file a separate issue? We can look at it.

[aehlig]

RC 13 is available at https://releases.bazel.build/0.26.0/rc13/index.html

[aehlig]

Looking at the downstream projects,

• Kythe is broken by their version-number check; will ignore those kind of errors, and
• upb is broken in the same way as in the nightly; will have to find out whether this caused by bazel or the project.

[aehlig]

Actually, looking at the upb error again, it is still a broken version number check. Will ignore all those kind of errors during releases.

[brandjon]

To recap the issue with #8414: It makes migration for Python toolchains (#7899) more difficult on mac than it needs to be, but doesn't block migration entirely. It's fixed by 7f495315749478e75a3424726cc273a535b7c3b8, but the fix modifies the PATH environment relative to what Bazel sets up, and that's fixed in ddce7235ef29a0aba727c265eae865d15af4ed09.

So cherrypicking both commits (in that order) would be helpful for migration. Either way, the flag will be flipped in 0.27. Without the cherrypicks, mac users have the option of 1) opting out of the flag flip in 0.27 until they can migrate, or 2) defining their own platform toolchain instead of relying on the default autodetecting one.

[keith]

I think if that flag is going to be flipped in 0.27.0 it needs to function in 0.26.0 so that mac users don't have to ship with an incompatible flag, or delay updating to 0.27.0 while they migrate.

[brandjon]

Escalating to @dslomov for @keith's concern on migration window timing. I don't want to push this flag flip past 0.27 because we're looking at a longer stability window starting at that release. Flipping the flag helps us fix #4815 (except on Windows, which still needs an autodetecting toolchain implementation), which has caused confusion around Python versions for a long time.

I think requiring mac users to define a toolchain, or else rely on the 0.27 release for migration (via flag opt out) is acceptable. On the other hand, if we cherrypick those two commits this may be moot.

[dslomov]

@brandjon can you summarize why pushing the flag flip beyond 0.27 is detrimental? Basically all the new features should already be available, so people can already use them even if the old behavior still exists, or am I missing something?

[aehlig]

[...] It's fixed by 7f49531, but the fix modifies the PATH environment relative to what Bazel sets up, and that's fixed in ddce723.

So cherrypicking both commits (in that order) would be helpful for migration. [...]

Thanks for the summary. Looking at the patches, they seem to be limited enough in scope to be safely cherry-picked.

[aehlig]

./scripts/release/release.sh create --force_rc=15 0.26.0 c2001a4569483596d9dc74ba9cabcbe4b6f1887f e67c961905792cd63950c6f6efc33275ad243c49 81aefe7ee01cc73646a53f9c72ed40ead09f9f5a 597e289b097d3bfed8eea1cb0924bbeb04877e42 942f7cf6a0da0a4ecc804615424f039e50963933 85a5a2bd569a5274950fc7327a044c395248c024 9835cb4135503768cdf1161746b95d7969ccb938 c963ba21073b514961946b8b4b45b091f08fdaa1 a1ea487e0a9e180a36fa4aab57f7c746ddcf367a 7dc78cdd04eedf2f4373b170053ba5fc2a990929 dd9ac13f7e3b71bdf2eca717bc7681bdd12389a2 0ff19c6d0adf3c0df94fff59ca3bd13cbcf99897 7f495315749478e75a3424726cc273a535b7c3b8 ddce7235ef29a0aba727c265eae865d15af4ed09

RC 15 is available under https://releases.bazel.build/0.26.0/rc15/index.html

[aehlig]

The run of the downstream projects looks good (again upb is broken because of their version-number check).

[aehlig]

Please test RC15 carefully! If no blockers are reported here till Monday, May 27, 2019, morning Munich time, I will promote RC15 to release.

[brandjon]

Thanks for the cherry pick. To respond to Dmitry: Without the flag flip, the default behavior of Bazel will be that #4815 is still the reality for users out-of-the-box on all platforms. With the flag flip, that bug goes away on linux and mac (but not yet windows). #4815 leads to a pretty bad experience with how Bazel manages PY2 vs PY3, and it's not immediately obvious to new users why it would happen when they explicitly declared the right version at analysis time.

[dslomov]

Thanks for the context @brandjon, cherry-pick lgtm.

[katre]

Please cherrypick 35dd05a059fa7fddfdd888cfc69102994e3c04dc for #8451.

[aehlig]

RC 16 available at https://releases.bazel.build/0.26.0/rc16/index.html

[Edit: fixed URL]

[aehlig]

The run of the downstream projects looks good. All failures are just version checks (effectively saying, it is OK to not test those projects).

Please test RC16 carefully and report any problems here. If nothing else gets reported, I will promote RC16 to release on Tuesday (May 28, 2019) morning Munich time.

[gertvdijk]

RC 16 available at https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/skylark/SkylarkRepositoryFunction.java

I believe that link should have been https://releases.bazel.build/0.26.0/rc16/index.html :smile:

All looks good to me from here. Also really nice to see the binary download size has come down again with 0.26, amazing work!

• bazel_0.22.0-linux-x86_64.deb 161 MB
• bazel_0.23.2-linux-x86_64.deb 138 MB
• bazel_0.24.1-linux-x86_64.deb 91.2 MB
• bazel_0.25.3-linux-x86_64.deb 71.7 MB
• bazel_0.26.0rc16-linux-x86_64.deb 46 MB

[aehlig]

RC 16 available at https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/skylark/SkylarkRepositoryFunction.java

I believe that link should have been https://releases.bazel.build/0.26.0/rc16/index.html smile

Oh yes, indeed. Thanks! Edited the comment.

[aehlig]

Will promote RC16 to release.

[aehlig]

The release 0.26.0 is available at https://releases.bazel.build/0.26.0/release/index.html

[aehlig]

The release 0.26.0 is available at https://releases.bazel.build/0.26.0/release/index.html

CC @vbatts @petemounce @excitoon

[hlopko]

And we have the first regression: https://github.com/bazelbuild/bazel/issues/8475. Fix is being implemented.

[aehlig]

And we have the first regression: #8475. Fix is being implemented.

So we will have a patch release, once this is fixed.

[irengrig]

The fix for #8475 is submitted.

[steeve]

Submitting https://github.com/bazelbuild/bazel/pull/8522 for cherry-pick, which fixes https://github.com/bazelbuild/bazel/issues/8520.
This fixes rules_go on iOS and Android.

[katre]

c3d2aa74ccd23dfb8a8173c2b3e2955f0c5892cb is submitted and fixes #8520.

[aehlig]

Creating rc1 for the patch release 0.26.1.

scripts/release/release.sh create 0.26.1 cb82ed84d44db0169a8fbf15f9cee434b77002bb d1c0d205945f5a765efb0a48593b1cd82699ce32 c3d2aa74ccd23dfb8a8173c2b3e2955f0c5892cb

• Taking as baseline the release commit for 0.26.0, i.e. cb82ed84d44db0169a8fbf15f9cee434b77002bb
• cherry-picking d1c0d205945f5a765efb0a48593b1cd82699ce32 for #8475
• cherry-picking c3d2aa74ccd23dfb8a8173c2b3e2955f0c5892cb for #8520

It is available at https://releases.bazel.build/0.26.1/rc1/index.html
(Please don't get confused by the list of cherry-picks mentioned there; our release script got them wrong again; we only have the two cherry-picks mentioned here.)

[aehlig]

The downstream projects look good (the "Bazel Integration testing" failure is due to changed RBE configurations).

So please test 0.26.1rc1. I plan to release it on Wednesday, June 5, 2019, in the morning, Munich time.

[jin]

Please cherrypick https://github.com/bazelbuild/bazel/commit/b0403a7004976cb959a51500d7a162f37e9bfed1 for https://github.com/bazelbuild/intellij/issues/845. Fixes Bazel with IntelliJ when building annotation processor targets.

[aehlig]

Please cherrypick b0403a7 for bazelbuild/intellij#845. Fixes Bazel with IntelliJ when building annotation processor targets.

Is this really a _regression_, i.e., something not working on 0.26.0 that _was_ working on 0.25.3? Also note that cherry-picks are supposed to be small, well-understood changes where we can be reasonably sure that they do not break anything else; a bump of an opaque archive usually does not qualify as such.

[philwo]

@aehlig Absolutely a regression, I can’t even build Bazel itself anymore with IntelliJ inside a Google and thought I’m doing something wrong... I'm getting the Genclass error that’s mentioned in the linked commit. This always worked before 0.26.0.

[aehlig]

This is the diffstat(1) between the contents of the old and the bumped linux archive.

 BUILD                                                                           |   31 
 README.md                                                                       |    6 
 java_tools/JacocoCoverage_jarjar_deploy.jar                                     |binary
 java_tools/JavaBuilder_deploy.jar                                               |binary
 java_tools/VanillaJavaBuilder_deploy.jar                                        |binary
 java_tools/ijar/ijar                                                            |binary
 java_tools/ijar/zipper                                                          |binary
 java_tools/src/main/cpp/util/errors_windows.cc                                  |    5 
 java_tools/src/main/cpp/util/file_windows.cc                                    |    7 
 java_tools/src/main/cpp/util/path_platform.h                                    |    3 
 java_tools/src/main/cpp/util/path_windows.cc                                    |  154 --
 java_tools/src/main/cpp/util/profiler_windows.cc                                |    4 
 java_tools/src/main/cpp/util/strings.cc                                         |    7 
 java_tools/src/main/native/windows/file.cc                                      |  120 +-
 java_tools/src/main/native/windows/file.h                                       |   51 
 java_tools/src/main/native/windows/util.cc                                      |   25 
 java_tools/src/main/native/windows/util.h                                       |    6 
 java_tools/src/tools/singlejar/diag.h                                           |    6 
 java_tools/third_party/java/jacoco/LICENSE                                      |  544 ++--------
 java_tools/third_party/java/jacoco/jacocoagent-0.8.3.jar                        |binary
 java_tools/third_party/java/jacoco/jacocoagent.jar                              |binary
 java_tools/third_party/java/jacoco/org.jacoco.agent-0.7.5.201505241946-src.jar  |binary
 java_tools/third_party/java/jacoco/org.jacoco.agent-0.7.5.201505241946.jar      |binary
 java_tools/third_party/java/jacoco/org.jacoco.agent-0.8.3-sources.jar           |binary
 java_tools/third_party/java/jacoco/org.jacoco.agent-0.8.3.jar                   |binary
 java_tools/third_party/java/jacoco/org.jacoco.core-0.7.5.201505241946-src.jar   |binary
 java_tools/third_party/java/jacoco/org.jacoco.core-0.7.5.201505241946.jar       |binary
 java_tools/third_party/java/jacoco/org.jacoco.core-0.8.3-sources.jar            |binary
 java_tools/third_party/java/jacoco/org.jacoco.core-0.8.3.jar                    |binary
 java_tools/third_party/java/jacoco/org.jacoco.report-0.7.5.201505241946-src.jar |binary
 java_tools/third_party/java/jacoco/org.jacoco.report-0.7.5.201505241946.jar     |binary
 java_tools/third_party/java/jacoco/org.jacoco.report-0.8.3-sources.jar          |binary
 java_tools/third_party/java/jacoco/org.jacoco.report-0.8.3.jar                  |binary
 33 files changed, 406 insertions(+), 563 deletions(-)

This is more than a small, well-understood, change. Isn't there a way to just fix the regression without changing two binaries, 16 java archives, and the license?

[aehlig]

@aehlig Absolutely a regression, I can’t even build Bazel itself anymore with IntelliJ [...]

So how was that regression introduced into the code base. I doubt it was introduced by a downgrade of the java tools...

[philwo]

I don’t know, I’m just a user in this case who can no longer work until this is fixed.

[aehlig]

I don’t know, I’m just a user in this case who can no longer work until this is fixed.

OK, then we'll bisect to find the offending commit and try to cherry-pick a revert of it (the revert needn't go on master, it can stay on a side branch); but such a big library change to fix forward is too large for a patch release, in my opinion.

[iirina]

While b0403a7 fixes bazelbuild/intellij#845, I'm not sure that was the culprit. The typo fixed by the new java_tools release was there from the first java_tools version, so downgrading the java_tools version will not work. @aehlig's proposal seems the right way forward.

[philwo]

I'm bisecting this now.

[philwo]

This commit is the culprit: 888716990fd1fe0cc3370e5a7e1d5ec2c75dec92

Verified by checking out 0.26.0 and doing a git revert. I can then no longer repro the issue.

[aehlig]

This commit is the culprit: 8887169

Verified by checking out 0.26.0 and doing a git revert. I can then no longer repro the issue.

Thanks for the bisect. Now we're in trouble here, as the culprit is also a bump of an embedded library. Will have to have a look at both changes to make a call, what is less likely to break things. Any technical information about those two library changes would be greatly appreciated.

[aehlig]

Now we're in trouble here, as the culprit is also a bump of an embedded library.

Fortunately, the changes in the libraries hidden in the culript commit are comparatively small; 192 lines of BUILD file (and a README file). So, let's try cherry-picking the revert of the culprit.

[aehlig]

@iirina created a patch-release for the java tools with only the minimal changes needed to fix https://github.com/bazelbuild/intellij/issues/845. Thank you! So we can get away with cherry-pick a much smaller change, 55e42052a22a60b68d88a89932b2a068311b1a95.

[aehlig]

scripts/release/release.sh create --force_rc=3 0.26.1 cb82ed84d44db0169a8fbf15f9cee434b77002bb d1c0d205945f5a765efb0a48593b1cd82699ce32 c3d2aa74ccd23dfb8a8173c2b3e2955f0c5892cb 55e42052a22a60b68d88a89932b2a068311b1a95
scripts/release/release.sh push

(There will be no rc2, as I forgot to fetch the branches on first attempt and thus decided it's safer to drop the rc number.)

[aehlig]

8556 tracks our efforts of avoiding such regressions in the future.

[aehlig]

0.26.1rc3 is available at https://releases.bazel.build/0.26.1/rc3/index.html

[aehlig]

The downstream projects still look good, again with "Bazel Integration Testing" broken due to changed RBE configurations.

Please test carefully. I plan to release 0.26.1 on Thursday, Jun 6, 2019, early afternoon Munich time.

[aehlig]

Promoting 0.26.1rc3 to release...

[aehlig]

0.26.1 is available at https://releases.bazel.build/0.26.1/release/index.html