Bazel: Release 0.27 - June 2019

[dslomov]

Target RC date: June 3rd

[Ragora]

I would like https://github.com/bazelbuild/bazel/commit/84c5dc9ad7cc9c00572c7e175b09b3979c5052a8#diff-73cc3e84377e7c63ef4406039e060016 (or an equivalent thereof) to be in this release.

For my employer I started this fairly large migration to bazel seeing from documentation it did everything we needed, however, it seems pkg_deb does not have a templates parameter which we use quite heavily in our existing software meaning the resulting packages in the current state are kindof useless.

I'm at a point of having to make a bazel build from master just for this one feature.

Amazing tool otherwise, though! It's just this one thing is kindof a stickling point at the moment.

[laurentlb]

It will definitely be included. Any change submitted in May will be included. Changes submitted in June are unlikely to part of 0.27.

Thanks for the feedback!

[Ragora]

Cool, was checking because I got confused about the 0.26 release still not having it, guess I just unlucky about the way times line up then. In the meantime I'll use a custom build, but at least it's not long to wait.

[laurentlb]

Status of Bazel 0.27

I plan to keep this message up to date during the release process and provide the important information.

• Target baseline: June 3rd
• Expected release date: Monday, June 17th
• List of release blockers

To report a release-blocking bug: file a bug, use the Release blocker label, and cc @laurentlb. Release blockers are regressions in Bazel relative to Bazel 0.25 or Bazel 0.26.

Task list:

• [x] Update GitHub issues for incompatible changes
• [x] Pick release baseline: 8c3b3fba3f68833bd97d3df2db9c48f0539efc3b
• [x] Create release candidate: https://releases.bazel.build/0.27.0/rc5/index.html
• [x] Check downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1034 (a few failures, likely due to the projects themselves)
• [x] [Create draft release announcement](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit#heading=h.ibnqhsyprlp8)
• [x] Send for review the release announcement PR: https://github.com/bazelbuild/bazel-blog/pull/178
• [x] Push the release, notify package maintainers: https://github.com/bazelbuild/bazel/releases/tag/0.27.0
• [x] Update the documentation
• [x] Publish the announcement (blog & mailing-list)

[brandjon]

I'm about to flip --incompatible_use_python_toolchains. This was introduced in 0.25, flipped at head for 0.26, and immediately rolled back when it broke downstream. So it's now had two months of migration window.

All known downstream breakages are summarized in this comment. Most breakages are because targets had been declaring themselves (perhaps by default) to be Python 3, even though they were really only compatible with Python 2. Previously they sometimes got ran under Python 2 anyway due to #4815, but this flag fixes that bug on every platform besides Windows. Migration usually consists of adding python_version = "PY2" to Python targets, and --host_force_python=PY2 if any of them are used in the host configuration.

+cc Green Team sheriffs @vladmos and @laszlocsomor.

[laurentlb]

Creating 0.27.0rc1 with:

scripts/release/release.sh create 0.27.0 5935259724bebd1c4bdebc90e159d0f655c8c219

[jin]

0.26.0 / 0.27.0 is broken in IntelliJ when building java_plugin targets - https://github.com/bazelbuild/intellij/issues/845

Please cherry-pick https://github.com/bazelbuild/bazel/commit/b0403a7004976cb959a51500d7a162f37e9bfed1 (if it's after baseline)

[brandjon]

Created blockers #8547 and #8549 to track my effort to add better diagnostics for users whose Python targets fail due to toolchains.

[philwo]

Can we ensure that https://github.com/bazelbuild/bazel/commit/8c3b3fba3f68833bd97d3df2db9c48f0539efc3b gets into the release? It resolves a major issue with —keep_going that causes failing builds to hide the error and exit with code 0 (https://github.com/bazelbuild/bazel/issues/7674). This in turn causes Bazel’s CI to Mark pipelines as green, even though the projects are broken.

[laurentlb]

Created 0.27rc2 with:

scripts/release/release.sh create --force_rc=2 0.27.0 8c3b3fba3f68833bd97d3df2db9c48f0539efc3b

(this includes both b0403a7 and 8c3b3fb)

You can download the release candidate here: https://releases.bazel.build/0.27.0/rc2/index.html

[brandjon]

Of the two release blockers I filed, one is fixed and the other has a pending CL. Both concern the ease of migrating to toolchains but do not affect CI.

The already submitted one is 123c68daed17b19927372e4df7f7a2256db6b80e and can be cherrypicked now. Will post when the other is submitted.

[brandjon]

And the other one is 052167e907373ac7ea43238c3049739f6e94a9d1.

[meisterT]

Laurent, this might require a cherry pick: #8539

[davido]

Laurent, this might require a cherry pick: #8539

As pointed out in discussion in the above issue, https://github.com/bazelbuild/bazel/commit/bad5a2beb9f8441d97b88d5c4323055e710f72e8 introduced very serious regression (and is included in 0.27rc2):

The previous javac version had a non-standard modification to default to
the Java 8 language level, which wasn't carried forward to 11.

As the consequence, after the upgrade from 0.26 to 0.27rc2 wrong byte code is generated, when building with default_java_toolchain. Even though, Java 8 is used, byte code with major version 55 is produced that corresponds to Java 11.

  $ bazel version
  Build label: 0.27.0rc2

  $ $JAVA_HOME/bin/java -version
  openjdk version "1.8.0_212"

  $ bazel build java/com/google/gerrit/common:server 
  [...]
  Target //java/com/google/gerrit/common:server up-to-date:
  bazel-bin/java/com/google/gerrit/common/libserver.jar
INFO: Elapsed time: 46.555s, Critical Path: 30.31s
INFO: 36 processes: 27 linux-sandbox, 9 worker.
INFO: Build completed successfully, 38 total actions

  $ javap -verbose -cp bazel-bin/java/com/google/gerrit/common/libserver.jar com.google.gerrit.common.data.SubscribeSection | grep major 
  major version: 55

On Bazel 0.26 the last command would correctly print Java 8 byte code major version:

  $ javap -verbose -cp bazel-bin/java/com/google/gerrit/common/libserver.jar com.google.gerrit.common.data.SubscribeSection | grep major 
  major version: 52

As the consequence, after building Gerrit Code Review project with 0.27.rc2, the produced artifact cannot be run any more on Java 8 (all is fine on 0.26):

  $ $JAVA_HOME/bin/java -version
  openjdk version "1.8.0_212"

  $ bazel build release
  [...]

  $ $JAVA_HOME/bin/java -jar bazel-bin/release.war init -d ../test_site_xxx_yyy_zzz
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: Main has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)

Can this be fixed, or at very least, can this disruptive change be documented in the release documentation?

[philwo]

Please note that we finished the migration to Ubuntu 16.04 for release builds and all other pipelines on Bazel CI now. This means that 0.27.0rc3 and following versions will be built on Ubuntu 16.04 and will no longer compatible with Ubuntu 14.04.

We will mention this in the release notes, too.

[iirina]

@laurentlb 6ef6d879ab69225d54ecab3db847fb4eff33bbeb fixes #8539 and should be cherry picked for backwards compatibility.

[laurentlb]

rc3 created with:

scripts/release/release.sh create --force_rc=3 0.27.0 8c3b3fb 123c68daed17b19927372e4df7f7a2256db6b80e 052167e907373ac7ea43238c3049739f6e94a9d1 6ef6d879ab69225d54ecab3db847fb4eff33bbeb

Available here: https://releases.bazel.build/0.27.0/rc3/index.html

[brandjon]

Filed #8576 for bugs I just noticed in 052167e907373ac7ea43238c3049739f6e94a9d1. (Sorry.) Working on it now.

[brandjon]

Fixed #8576 with 50fa3ec27efdd95771c70faa38a4543d4fed44f2, please cherrypick.

[irengrig]

Please cherry-pick https://github.com/bazelbuild/bazel/commit/6efc5b787ad3164cc2fb779c73377695032b4524, a fix for #8564.

[jin]

Please cherry pick https://github.com/bazelbuild/bazel/commit/e2a626c4f61fe4ceb79a5675d09a6f25ca7b5b22 for https://github.com/bazelbuild/rules_jvm_external/issues/165 and https://github.com/googlesamples/android-testing/issues/279

Thank you!

[laurentlb]

rc4 created with:

scripts/release/release.sh create --force_rc=4 0.27.0 8c3b3fb 123c68d 052167e 6ef6d87 50fa3ec27efdd95771c70faa38a4543d4fed44f2 e2a626c4f61fe4ceb79a5675d09a6f25ca7b5b22

https://github.com/bazelbuild/bazel/pull/8564#issuecomment-499796063 was not included.

Download rc4: https://releases.bazel.build/0.27.0/rc4/index.html
Status of downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1027

[brandjon]

Update on toolchain downstream breakages here.

[brandjon]

8536 is now release-blocking as I decide how to address the gerrit breakage. The error mode only occurs when

1. using --incompatible_strict_action_env in combination with
2. the default Python toolchain
3. on a mac, and
4. in a build that includes a genrule-within-a-genrule (or genrule underneath another host-configured target)

This combination should be pretty rare.

[brandjon]

8536 is fixed by a pending commit (blocked on transient infrastructure failures).

Laurent mentions that since this bug only manifests itself with --incompatible_strict_action_env, we may be allowed to break this use case and fix it in the next release with no cherrypick. However, some users have come to rely on this flag. It arguably functions more like an ordinary feature flag than an incompatible change flag since it has been around for a long time and apparently isn't going to be flipped.

I'll defer to @philwo, @aehlig, and @buchgr on whether it should be considered blocking.

[juliexxia]

Request for a cherrypick for #8610

There's a change currently inside the release candidate that flips --experimental_build_setting_api to true. I've just found a pretty major incremental correctness concern with build settings so I think we should patch a flip back. Apologies for the added complexity!

[brandjon]

8536's fix is in as 3a4be3c93813987a27a97dade3f9ebbc5770e349. Please cherrypick if creating a new RC.

[laurentlb]

I've created rc5 with 3 new cherry-picks. Unless I missed something, all cherry-pick requests from the thread have been included.

scripts/release/release.sh create --force_rc=5 0.27.0 c5f93b88d13631a01530fb78213dfb575486d675 6efc5b787ad3164cc2fb779c73377695032b4524 3a4be3c93813987a27a97dade3f9ebbc5770e349 5c1005c0947b010ee36ca851b8ba07c9479cf682

[laurentlb]

Test of downstream projects: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1034
Expected release date: next Monday
Try the new candidate: https://releases.bazel.build/0.27.0/rc5/index.html

[juliexxia]

Woof, Is it possible to add another cherrypick for #8633 ? 49862c5fa66f65deb513ee938d3d052b9e4cfbbe is the commit. Right now bazel can crash because of the issue. Sorry to delay the release timeline again!

[davido]

I have a call to document in the release notes for 0.27, that Python 3 is now required for Bazel.

I am able to build Gerrit Code Review on Mac OS on Bazel 0.26 without having Python3 installed, see this comment: [1].

With 0.27.0rc5 I am not able to build Gerrit Code Review any more. I have to install Python3. Looking into release notes on https://releases.bazel.build/0.27.0/rc5/index.html the only relevant section:

Python rules now determine the Python runtime using toolchains rather than --python_top and 
--python_path, which are deprecated.

doesn't clearly telling me, that Python3 is now de-facto pre-requisite for Bazel 0.27 to work.

• [1] https://github.com/bazelbuild/bazel/issues/8536#issuecomment-501974143

[laurentlb]

For the release announcement, see:
https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit#heading=h.ibnqhsyprlp8
or:
https://github.com/bazelbuild/bazel-blog/pull/178

[davido]

@laurentlb Thanks, that document clarifies the issue.

[brandjon]

@davido, glad the (updated) release notes helped.

FTR, Bazel 0.27 shouldn't require Python 3 in order to function. But it may cause Bazel to try to use Python 3 by default where it didn't before. This can be worked around to restore the pre-existing behavior as per #7899.

[alexeagle]

@laurentlb the rc5 breaks rules_nodejs. This isn't visible in the BuildKite build because it doesn't build all the tests in nested Workspaces. A failure is visible on Ubuntu in https://circleci.com/gh/bazelbuild/rules_nodejs/20685#tests/containers/2 - this is from a PR where we tried to update the rules_nodejs monorepo to 0.27 rc5

Failure is in compiling ts_library rules.
Adding --strategy=TypeScriptCompile=standalone makes them build successfully. I tested it in earlier RCs, the problem is present in rc1 but not in 0.26.1.

EDIT: I tracked it down to --incompatible_list_based_execution_strategy_selection which is in the release notes. This has the subtle side effect of enabling worker mode for actions that support it, with no action from users. Maybe add this to the release notes?

[laurentlb]

Bazel 0.27 is released.
https://github.com/bazelbuild/bazel/releases/tag/0.27.0

Announcement will be published soon.

cc @vbatts @petemounce @excitoon

[excitoon]

Cool! ScoopInstaller/Main@2a62d2949a0e4d32204b64be42ff4068fa0379ba

[vbatts]

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/936671/

[vbatts]

Is there are hard dependence on python3 now?

[vbatts]

fedora 30 and epel (centos7) failed due to python2.7

[brandjon]

@vbatts There is no hard dependency, it's just a default that is now correctly being followed. Buried in the log output is an error message about using --host_force_python and/or the non-strict Python toolchain. See #7899.

[petemounce]

Published to chocolatey.

[vbatts]

i'm still having issues getting this built for fedora and centos.
Centos python 2 or 3 seems to be a clunky now.
And for fedora, it is failing on:

ERROR: /builddir/build/BUILD/bazel-0.27.0/src/BUILD:311:2: Executing genrule //src:embedded_tools_nojdk failed (Exit 127): bash failed: error executing command 
  (cd /tmp/bazel_P4UK5kXs/out/execroot/io_bazel && \
  exec env - \
    PATH=/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/src/create_embedded_tools "bazel-out/k8-opt/bin/src/embedded_tools_nojdk.zip" bazel-out/k8-opt/bin/src/embedded_tools_nojdk.params')
Execution platform: @bazel_tools//platforms:host_platform

[brandjon]

@vbatts Can you open a new issue and we'll debug there? Did you pass --host_force_python=PY2? Please include the full output (in case there was anything emitted by the failing target aside from that error log line), invocation line, and platform.

[vbatts]

python detection issue: https://github.com/bazelbuild/bazel/issues/8665

[buchgr]

If there will be a patch release, please cherry-pick 484ffae78ff592fa44c2da9b131c1437800c8283 and d75b6cf85c211fd284c302941bc5efc1b361b83a

[vbatts]

fedora, centos7 and rhel8 rpms pushed.
https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/941771/

[iirina]

@laurentlb is there going to be a patch release? d4589630428c1c1bc08a9baf36257e636fe9d746 should be cherry picked (fixes #8670 which is a regression in bazel 0.27).

[philwo]

I did all required changes to fix https://github.com/bazelbuild/bazel/issues/8652 now. If a patch release for 0.27.0 is created, it will automatically be built on CentOS 7, which will fix the linked issue and restore compatibility with Ubuntu 14.04, CentOS 7 and RHEL 7.

[vbatts]

Cool cool to see official rpms. We'll see how well they work for Fedora. I'm glad to offload that from my copr.
As for transitioning the thousands of subscribers to my builds:

• Are you going to make a yum repo for these centos rpms?
• Is this a beta build for now?
• Is there going to be an announcement? (With transition steps for users)

-------- Original Message --------
On Jun 25, 2019, 10:08, Philipp Wollermann wrote:

I did all required changes to fix #8652 now. If a patch release for 0.27.0 is created, it will automatically be built on CentOS 7, which will fix the linked issue and restore compatibility with Ubuntu 14.04, CentOS 7 and RHEL 7.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[philwo]

@vbatts For this change, I just did the work on Bazel to pass its test suite on CentOS 7 (required a few small patches due to e.g. an older Git version in CentOS 7) and create a Docker container for our CI that contains what we need to build and test Bazel.

Then I added CentOS 7 as a supported platform to our CI and changed our scripts to build the "default" Linux binaries on it. I verified that when using these binaries built on CentOS 7, our downstream pipeline still passes (even when you use these binaries on e.g. Ubuntu 18.04). I didn't expect any issues here, but wanted to make sure.

That means, all official releases of Bazel will now have their Linux binaries built on CentOS 7 (instead of Ubuntu 16.04). This restores compatibility with Ubuntu 14.04, CentOS 7, while also staying compatible with newer Linux versions. As they should otherwise be functionally identical and pass all of our downstream tests, I'd consider them production ready.

It would be nice to create official RPMs and setup a yum mirror, but that's a next step - as much as I personally like working with CentOS and Fedora, I really don't know anything about how to build good packages for it. :) I'm happy to look at your COPR packages and give it a try though and get your feedback to make sure that it looks good.

[vbatts]

@philwo ah okay. so the *.zip of binaries is good to go for those folks. Then I will keep rolling with the copr builds for now.

If you were to make rpms and a yum(dnf) repo, copr is about as handy as anything. It does purge older builds, but if you're tagging releases, then folks can rebuild something about as equivalent. There is some automation to automatically trigger builds, but I haven't taken the time to iron that out. For now I mainly bump the version in the bazel.spec file and run make copr.

Once bazel gets aarch64 build support, I can flip the switch to enable build chroots for all fedora's and centos/rhel (7 and 8).

[jayconrod]

@laurentlb Gentle ping on @iirina's comment above: will there be a patch release?

rules_go coverage tests are failing in CI with 0.27.0, and d458963 should fix it. We don't have a workaround for this issue at the moment.

[laurentlb]

Bazel 0.27.1rc1 is available here: https://releases.bazel.build/0.27.1/rc1/index.html

It was created with:
scripts/release/release.sh create 0.27.1 c82eb4878c7dc829455caeb915affe36c89df06f d4589630428c1c1bc08a9baf36257e636fe9d746

So it's a cherry-pick of Irina's commit.
Please give it a try.

[philwo]

Dear users, please also verify that Bazel 0.27.1rc1 restores compatibility with Ubuntu 14.04 and CentOS 7.

[jayconrod]

I've verified 0.27.1rc1 fixes #8670 (bazelbuild/rules_go#2100), both locally and on CI (log). Thanks!

[laurentlb]

0.27.1 was released.
https://github.com/bazelbuild/bazel/releases/tag/0.27.1

[laszlocsomor]

Regression: https://github.com/bazelbuild/bazel/issues/8723
Fix is pending review: https://github.com/bazelbuild/bazel/pull/8784

I think we'll need a 0.27.2 patch release.

[laszlocsomor]

Please create patch release 0.27.2 with https://github.com/bazelbuild/bazel/commit/6d0b14b95a71175362030b4811ca74512b00a890 to fix regression https://github.com/bazelbuild/bazel/issues/8723.

[laurentlb]

Created 0.27.2rc1 using
scripts/release/release.sh create 0.27.2 c82eb4878c7dc829455caeb915affe36c89df06f d458963 6d0b14b

The binary is available here: https://releases.bazel.build/0.27.2/rc1/index.html

[laurentlb]

Released: https://github.com/bazelbuild/bazel/releases/tag/0.27.2