Bazel: Release 3.5 - August 2020

[meisterT]

Tracking bug for the Bazel 3.5 release. Instructions are at https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md

[aiuto]

Status update: Branch cut slightly delayed because of power outages after storms in NYC area. Probably will cut on morning of 2020-08-07

[davido]

Probably will cut on morning of 2020-08-07

This regression seems to be a release blocker: https://github.com/bazelbuild/bazel/issues/11837?

[aiuto]

Status of Bazel 3.5

• Target baseline: 2020-08-03
• Expected release date: 2020-08-26
• List of release blockers

To report a release-blocking bug, please file a bug using the Release blocker label, and cc me.

Task list:

• [x] Update GitHub issues for incompatible changes
• [x] Pick release baseline: 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e
• [x] obsolete Create release candidate: release-3.5.0rc4
• [x] Create release candidate: release-3.5.1rc1
• [x] Check downstream projects:
  
  
  
  • rules_python appears broken but is not. It is pinned to 3.3.1. It passes with 3.5 if run manually
  
  
  • rules_typescript appear broken but is not. It is pinned to 2.1, but passes with 3.5.
  
  
• [x] [Create draft release announcement](https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit)
• [x] Send for review the release announcement PR: https://github.com/bazelbuild/bazel-blog/pull/232
• [x] Push the release, notify package maintainers:
• [x] Update the documentation
• [x] Push the blog post
• [ ] Update the release page

[aiuto]

Currently broken and investgating:
flogger: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1594#eda26e49-1c79-4ac6-b9a9-284715b51ac6
https://github.com/google/flogger/issues/179

The two PRs which most likely caused the problems are correctness fixes, so I am not counting this as a blocking change yet.

[or-shachar]

Hey hey,
How can we consume rc1? https://releases.bazel.build/3.5.0/rc1/index.html is not available (yet?)

I want to test it against rules_scala and to see that our issue is indeed fixed.

[aiuto]

It will be available later today. If you want it earlier, you can clone the
repo, check out the release-3.5.0rc1 branch, and build that.

On Wed, Aug 12, 2020 at 5:29 AM Or Shachar [e-mail] wrote:

Hey hey,
How can we consume rc1? https://releases.bazel.build/3.5.0/rc1/index.html
is not available (yet?)

I want to test it against rules_scala and to see that our issue is indeed
fixed.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/11885#issuecomment-672763972,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHEXTAKK4WCED7LJUKDSAJOHRANCNFSM4PTA3RRQ
.

[philwo]

FYI, arm64 users: The arm64 build process for this release is still partly manual, so please give me a few minutes after @aiuto pushes rc1, so that I can kick-off the build and upload of the arm64 binaries ;)

[aiuto]

It is now available: https://releases.bazel.build/3.5.0/rc1/index.html

[davido]

@aiuto

So the fix in #11838 will be included.

The fix from #11838 wasn't included, unfortunately.

Now Bazel-3.5.0rc1 is reporting ca. 100 errors, when trying to build gerrit, that are actually ignored: [1]. You don't see this in Bazel CI, because the errors are actually ignored. Probably other projects are also affected, like JGit and Gitiles.

The only available workaround to silent those errors is to migrate external workspaces from foo-bar to foo_bar. But given that this is a regression, I would rather cherry-pick the PR that is linked to #11838.

Note that all is fine in Bazel 3.4.0 with external workspaces with hyphen char in their names, like foo-bar.

[1] http://paste.openstack.org/show/796786

[aiuto]

Oh whoops. Why did I not see that.
I can cherrpick it in to rc2

On Wed, Aug 12, 2020 at 1:51 PM David Ostrovsky [e-mail]
wrote:

@aiuto https://github.com/aiuto

So the fix in #11838 https://github.com/bazelbuild/bazel/pull/11838
will be included.

The fix from #11838 https://github.com/bazelbuild/bazel/pull/11838
wasn't included, unfortunately.

Now bazel is reporting ca 100 errors, when trying to build gerrit, that
are actually ignored: [1].

The only available workaround to silent those errors is to migrate
external workspaces from foo-bar to foo_bar. But given that this is a
regression, I would rather cherry-pick the PR that is linked to #11838
https://github.com/bazelbuild/bazel/pull/11838.

[1] http://paste.openstack.org/show/796786

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/11885#issuecomment-673020436,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHEH3TRF3DWPY3CKIYDSALJDHANCNFSM4PTA3RRQ
.

[konste]

Release notes advertise the new feature cc_common.compile support for include_prefix/strip_include_prefix, but it does not seem to be available in 3.5.0RC1
The fix went in just a couple days ago. Is it even going to make it to 3.5.0 release?

[aiuto]

I have no answer for why you don't see it. That PR was before the 3.5 baseline, so it should be there.
I would ask @oquenchil about it. If it turns out the release notes got ahead of the functionality, we can fix the release notes more easily then a cherrypick.

[konste]

The feature cc_common.compile support for include_prefix/strip_include_prefix does work with 3.5.0RC1
Really sorry for the false alarm! Bazelisk got me confused.

[konste]

Seems to be a regression in 3.5.0RC1: https://github.com/bazelbuild/bazel/issues/11936

[aiuto]

3.5.0rc2 is ready.

[davido]

3.5.0rc2 is ready.

Thanks, confirmed that the regression #11838 is fixed in 3.5.0rc2.

[or-shachar]

Just checked it on Wix repo - seems like there's a new restriction on external repository names?
we used to have external repositories for docker images with version in their names like busybox_1.25.0 and now we get the error:

10:44:25  Error in workspace: busybox_1.25.0 is not a legal workspace name

I didn't see this in the release notes

[meisterT]

@or-shachar see https://github.com/bazelbuild/bazel/issues/11936

[philwo]

Bazel 3.5.0rc2 Linux arm64 binaries have been successfully built and uploaded to https://releases.bazel.build/3.5.0/rc2/index.html.

[davido]

@meisterT @or-shachar

10:44:25 Error in workspace: busybox_1.25.0 is not a legal workspace name

If it is your primary WORKSPASE file, then I think this is rather the same regression as in #11837. I only added hyphen char in my fix, because Gerrit project doesn't use dot char in workspace repository names.

See also this https://github.com/bazelbuild/bazel/pull/11838#issuecomment-668435204 where @Gromba asked to allow dot char in repository names as well.

[or-shachar]

@davido @meisterT - it's not the primary workspace name but a generated workspace from the rule container_pull. We had dots in the external repository name which worked fine until this version

[aiuto]

I'll do some hunting to figure out why we added the restriction in the first place. That will help me decide if we do a patch in the minor releases until the next major and break then, or revert totally.

[aiuto]

I have a change in the pipeline to allow dot in workspace names. That will roll into RC3

[laurentlb]

The issue is discussed on https://github.com/bazelbuild/bazel/issues/11837

[aiuto]

We can resolve the prelude processing problem #11940 by picking a baseline from Aug. 7. 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e

Now the question is if the new error reporting is a blocker or not.

• are there reasonable workarounds?
• is there a short fix forward to suppress the better error reporting?

And also, do we revert the change allowing hyphens in names first. Given today's discussions, that is probably called for first.
I'll think through all this by Friday EOD and come up with a plan.

[aiuto]

Background on the flogger issue: https://github.com/bazelbuild/continuous-integration/issues/1006
The proposal was to provide the needed Android SDK in the CI environment.

[aiuto]

I think I have a good handle on the unneeded processing of WORKSPACE files in external repos. It should be easier to fix forward than roll back the improved error handling, so I'm going to try that approach. In the worst case, we can't do that and then we can have the debate about if this is an actual blocker or one people can work around by ignoring the log spam.

I will attempt that patch to mainline on Monday. If it goes well, I can cherrypick a new RC against the earlier baseline on Tuesday and send that out. This pushes back the expected release date to August 26 at the earliest.

[ulfjack]

FYI: the hyphen issue (#11837) also affects TensorFlow, at least in some configurations.

[aiuto]

Status update:

• I believe I have the WORKSPACE processing issues resolved. The change is going through review.
• I have another change out to allow dot in repo names.
• When both are merged, I can cut a new release candidate from the Aug 7 baseline, with those cherrypicks.
• I expect that can happen on 2020-08-18.

[konste]

Thank you @aiuto for your diligent work on these problems! Appreciate that a lot!

[aiuto]

You're welcome. Did you see the PR to update your repro? I played a lot with that today to make sure the fix would not silently break other things.

[konste]

Yes, the risk of regressions is the primary concern at this point. I am going to jump on the next RC when it is available and run it through the different scenarios on all platforms to make sure it is stable.

[aiuto]

I should have mentioned this, but rc3 went out yesterday, on August 18.
https://releases.bazel.build/3.5.0/rc3/index.html

[konste]

Thank you for the notice! We'll jump right on it!

[pariscraigm]

I am testing for regressions in rc3 for @konste and I am getting this error: Error in compile: compile() got unexpected keyword argument 'include_prefix'

Was include_prefix for cc_common.compile remove in rc3?

[konste]

I can confirm that the feature cc_common.compile support for include_prefix/strip_include_prefix was indeed present in RC1 and RC2, but disappeared from RC3. This regression is a Release stopper. We already took a dependency on this feature. @aiuto please take a look.

[aiuto]

That will not be in rc3, because we moved the baseline back to August 7.
How is this a release stopper if it is a new feature?

[konste]

Darn... When we found it in release notes and in RC1 we took it for granted. Why the baseline had to be moved back? I already regret that I raised that issue. We'd rather get include_prefix feature and work around WORKSPACE issues than the other way around. Don't anybody else expect include_prefix to appear in 3.5.0?

[aiuto]

We had to move it back because the prelude loading change was very incompatible for some users. Fixing it forward was too large a job.

[konste]

I see. Well, at least it was not me who caused it. I will remove the dependency on include_prefix feature (too bad, too bad - it was so handy) and we re-test RC3 once again.

[steeve]

Hi,

Would it be possible to cherry-pick https://github.com/bazelbuild/bazel/commit/a7a0d48fbeb059ee60e77580e5d05baeefdd5699 ?

That would unlock https://github.com/bazelbuild/rules_go/pull/2451.

Thanks!

[aiuto]

Not at this late in the process. That is only from 3 days ago.

[aiuto]

I just looked a little deeper and don't get the timeline. What is the relation between the bug mentioned in bazelbuild/rules_go#2445 and that commit? 2445 says it that the 3.5 release would have the fix, but I am not seeing the connection. Then there is an interesting flip on priorities for the underlying bug https://github.com/bazelbuild/bazel/issues/11196, so maybe I am not aware of some urgency.

[aiuto]

Just for kicks, I am trying a tentative rc4. It includes:

• the features which were in rc1, because it was disappointing to see them there and then have them vanish
• plus a7a0d48 because that looks like a fix to an even earlier fairly serious regression

https://releases.bazel.build/3.5.0/rc4/index.html

[konste]

Oh, wow! Feels a bit like a rollercoaster, but I enjoy it! @aiuto I appreciate the voluntary effort!

[steeve]

@aiuto sounds fair, thank you for the feedback

As for bazelbuild/rules_go#2445, the fix was for https://github.com/bazelbuild/bazel/issues/11167, which was merged in https://github.com/bazelbuild/bazel/commit/a90c4084315a490c4fecc864d0b3435cc7b2bd64 which _looks_ like it's before the cut.

[aiuto]

Running the downstream tests right now. https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1608
@konste I have kids. I know the look on their faces when you tell them something is going to happen and then it doesn't.
@steeve The no-op transition problem from a7a0d48 was still a fairly significant bug.

[aiuto]

@steeve rules_go fails on windows with rc4. Is this a real problem? https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1608#30eaebca-0f8d-4255-9653-1896d1cc0b50

[steeve]

Apparently these are link errors because of duplicated symbols in msvcrt, I'd say it may be a regression in bazel, cc @jayconrod for confirmation

[konste]

I am not familiar with the Buildkite, but still looking at the rules_go log out of curiosity. Is Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '3.3.1' a red herring? Just making sure it is noticed.

[aiuto]

Thanks for catching that. It looks to me like a test configuration problem.

[jayconrod]

Yep looks like a real regression. Something related to linking with msys2?

            ERROR: C:/b/xxofdm5f/bazel_testing/bazel_go_test/main/BUILD.bazel:33:8: GoLink racy_test_race_mode_/racy_test_race_mode.exe failed (Exit 1)
            c:/tools/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/libmsvcrt.a(/2203): duplicate symbol reference: _unlock_file in both libgcc(.text) and libgcc(.data)
            libgcc(.text): relocation target ___chkstk_ms not defined
            libgcc(.text): relocation target atexit not defined
            link: error running the following subcommand: exit status 2

[philwo]

@meteorcloudy 🚨 ;)

[meteorcloudy]

Interesting, let me try to reproduce this

[meteorcloudy]

Wait, this is not a regression, the same failure happens with Bazel 3.4.1:
https://buildkite.com/bazel/rules-go-golang/builds/2327#145a71ba-dee2-4eb7-b7ab-ff045ded70de

Possibly caused by https://github.com/bazelbuild/rules_go/commit/631e26be08da010780f1240f10d96ef09bf3dfb1

[meteorcloudy]

I updated the Windows image for CI machines yesterday, that might changed the MSYS version..

[jayconrod]

Hmm, that PR was green on presubmit. I'll try reverting it and see if the test still fails.

It should have been unrelated to MSYS though, so if the MSYS version was changed at the same time, that seems like a more likely cause.

[meteorcloudy]

@jayconrod You don't have to revert to verify. The CI already checked the last green commit, it is failing now.
https://buildkite.com/bazel/rules-go-golang/builds/2329

The Bazel Auto Sheriff also reported rules_go was broken due to infra change:
https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/252

[jayconrod]

Ah, ok. I'll let the revert PR finish testing (https://buildkite.com/bazel/rules-go-golang/builds/2330) just to confirm and will close it afterward.

[meteorcloudy]

I'll have to look into what actually changed on CI. But for now, I think we can consider there is no regression in rc5 rc4.

[konste]

@meteorcloudy we can consider there is no regression in rc5 RC4 you mean, right? There is no RC5 (yet?).

[meteorcloudy]

@konste Oh, yes. I meant the latest rc, which should be rc4

[aiuto]

Since this has seemed to settle out, it looks like I can turn RC4 into the
real release on Wednesday or Thursday.

On Mon, Aug 24, 2020 at 4:27 AM Yun Peng [e-mail] wrote:

@konste https://github.com/konste Oh, yes. I meant the latest rc, which
should be rc4

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/11885#issuecomment-678985525,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHCBM3JUEYUQU4VBL6LSCIP6XANCNFSM4PTA3RRQ
.

[aiuto]

Status update:
Blog post is ready: https://github.com/bazelbuild/bazel-blog/pull/236
I should be able to push on 2020-08-26

[asuffield]

ping?

[aiuto]

I am trying to push it right now. Having some ACL problems. Sigh.
They should be resolved today.

On Tue, Sep 1, 2020 at 9:35 AM Andrew Suffield [e-mail]
wrote:

ping?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/11885#issuecomment-684857816,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHDATOOXH7ERRZ764C3SDT2BZANCNFSM4PTA3RRQ
.

[aiuto]

@vbatts @petemounce @excitoon Bazel 3.5.0 is available at https://github.com/bazelbuild/bazel/releases

[philwo]

Linux arm64 binaries have been built and uploaded. 🦾

[or-shachar]

Yey 3.5.0! Thanks @aiuto for managing this!

It's probably just matter of propagation time - but note that l.gcr.io/google/bazel:3.5.0 is still not available.

[Yannic]

Bazelisk + Incompatible flags on Buildkite is broken with Bazel 3.5.0: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/615#1f97d2d1-4d06-454b-af61-deff7963d347/95-104

[meisterT]

Filed https://github.com/bazelbuild/bazel/issues/12058

[petemounce]

Chocolatey has been published.

[philwo]

Thank you @petemounce 😊

[vbatts]

I'm late to the party.
It looks like c++ error on g++ 10.2.1 (on fedora 32)?
This is from a local build, and i've kicked off a copr build for all platforms: https://copr.fedorainfracloud.org/coprs/build/1654639

ERROR: /var/tmp/bazel_QlyAxJ5T/out/external/upb/BUILD:57:11: C++ compilation of rule '@upb//:upb' failed (Exit 1): gcc failed: error executing command
  (cd /var/tmp/bazel_QlyAxJ5T/out/execroot/io_bazel && \
  exec env - \
    PATH=/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/bin-hack:/home/vbatts/workspace/bin:/home/vbatts/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/sbin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.d '-frandom-seed=bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.o' -iquote external/upb -iquote bazel-out/k8-opt/bin/external/upb -Werror -Wno-long-long -pedantic -Wstrict-prototypes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/upb/upb/upb.c -o bazel-out/k8-opt/bin/external/upb/_objs/upb/upb.o)
Execution platform: //:default_host_platform
In file included from /usr/include/string.h:495,
                 from external/upb/upb/upb.h:16,
                 from external/upb/upb/upb.c:2:
In function 'strncpy',
    inlined from 'upb_status_seterrmsg' at external/upb/upb/upb.c:40:3:
/usr/include/bits/string_fortified.h:106:10: error: '__builtin_strncpy' specified bound 127 equals destination size [-Werror=stringop-truncation]
  106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 20.600s, Critical Path: 2.91s
INFO: 60 processes: 60 local.
FAILED: Build did NOT complete successfully

$ g++ --version
g++ (GCC) 10.2.1 20200723 (Red Hat 10.2.1-1)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[aiuto]

This is for the Fedora bless package of Bazel 3.5, so it is mostly about building Bazel. Right?
Even so, does the 3.5 binary (un-repackaged) work with user projects using C++? Or, are the rule broken as well with g++ 10.2.1

[aiuto]

It looks like it fails for all platforms in \@upb, which gRPC uses. My hunch is that this is related to a 3.4 problem with gRPC that caused some rollback/forward and a 3.4.1 release.
https://github.com/bazelbuild/bazel/pull/11792
c3ea5a1253c6cfad46fffa25d9842bce1fdc69c2
fa7299c4e446ad76f86ebc3d34cbfd756eabc644

@meteorcloudy Do you have any ideas?

[vbatts]

This is for the Fedora bless package of Bazel 3.5, so it is mostly about building Bazel. Right?
Even so, does the 3.5 binary (un-repackaged) work with user projects using C++? Or, are the rule broken as well with g++ 10.2.1

this is in building bazel itself.
https://github.com/vbatts/copr-build-bazel/tree/v3.5.0-1
run: make rpm from a fedora 32 host.

[meteorcloudy]

Looks like it's the same issue as https://github.com/bazelbuild/bazel/issues/12056

upb has fixed the problem, we should update the upb version in Bazel.

[meteorcloudy]

PR sent: https://github.com/bazelbuild/bazel/pull/12077

[aiuto]

@meteorcloudy Is it time to try a 3.5.1 with the cherrypick of https://github.com/bazelbuild/bazel/commit/b3ac8f60973ba60d578ae6a653cdd993a2d206d7
or is there more to do?

[aiuto]

Actually, @vbatts, perhaps you are in the best situation to test, as you have the right compiler environment.
If you patch that commit into the 3.5.0 sources does your build then work?

[vbatts]

tested b3ac8f6, and it seems to rely on other files than are included in
3.5.0:

         File "/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/tools/build_defs/repo/http.bzl", line 121, column 10, in _http_archive_impl
                 patch(ctx)
         File "/home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/tools/build_defs/repo/utils.bzl", line 135, column 22, in patch
                 ctx.patch(patchfile, strip)
Error in patch: Not a regular file: /home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/third_party/grpc/upb_gcc10_fix.patch
ERROR: no such package '@upb//bazel': Not a regular file: /home/vbatts/workspace/src/github.com/vbatts/copr-build-bazel/bazel-3.5.0/third_party/grpc/upb_gcc10_fix.patch

[vbatts]

applied 26cbf776e0cdd8fbb9a40aca4c1be6f5313b6eb4 also
and local build is making it further, so i've kicked off https://copr.fedorainfracloud.org/coprs/build/1658045

[vbatts]

Those two commits did the trick.
(The two failures are the couple of aarch64 that have consistently been failing)

[aiuto]

Thanks.

I started a 3.5.1 run with those two cherrypicks + correctness bug fix from https://github.com/bazelbuild/bazel/commit/b9bb10252a004047f9a997165aa1ab7d6c6fa2e7.

scripts/release/release.sh create --force_rc=1 3.5.1 \
    889bc0b523b47eeb38a72bf9bb6858ee525a7c7e \
    a7a0d48fbeb059ee60e77580e5d05baeefdd5699 \
    b37c51c7085f0aefe04034dd451acb847605ddb5 \
    f6ad35fcde93f92c591778ed7db38d167f5bbc03 \
    39bc97eab295bddb35b38bfc4a2ff3d2b15d034e \
    b9706675a7abf6ceebb250f0b3dfa4087a0c35f6 \
    e574d558da17cfd0f818e7a937a07926aa270069 \
    9993785fa0c4fa4172aa31d306f3abea76833abf \
    b3ac8f60973ba60d578ae6a653cdd993a2d206d7 \
    26cbf776e0cdd8fbb9a40aca4c1be6f5313b6eb4 \
    b9bb10252a004047f9a997165aa1ab7d6c6fa2e7

Alas, it seems to fail on an unrelated issue.

src/tools/android/java/com/android/tools/r8/CompatDxSupport.java:16: error: cannot find symbol
--
  | import com.android.tools.r8.utils.AndroidApp;

https://buildkite.com/bazel/bazel-bazel/builds?branch=release-3.5.1rc1.

I'll have to dig into that.

[philwo]

@aiuto This looks like an incompatibility between Bazel 3.5.x and a too new Android SDK. IIRC we added a newer version of the Android build-tools on the CI machines in the meantime due to a request from the Android folks. Because Bazel's WORKSPACE build does not pin the tools to a fixed version, Bazel 3.5.x now picks up that latest version, but is not yet compatible with it.

[aiuto]

On Sat, Sep 12, 2020 at 2:51 AM Philipp Wollermann [e-mail]
wrote:

@aiuto https://github.com/aiuto This looks like an incompatibility
between Bazel 3.5.x and a too new Android SDK.

It does look like that.

IIRC we added a newer version of the Android build-tools on the CI
machines in the meantime due to a request from the Android folks. Because
Bazel's WORKSPACE build does not pin the tools to a fixed version, Bazel
3.5.x now picks up that latest version, but is not yet compatible with it.

Wouldn't Bazel at head fail in CI if we did not have a corresponding change
at head to account for the new SDK?
Last night I started looking through all the commits between Aug 7 and now
but did not find evidence of that.
Perhaps it was just too long a day.

Are there notes about how we put the SDK on the CI machines? Those would be
handy for setting up personal machines to do full testing locally.

[aiuto]

@philwo The obvious question is you say "we added a newer version. If you really mean added, rather than replaced, this should only require pinning the version in the WORKSPACE.

[philwo]

@aiuto Yes, we added Android build-tools 30.0.1 and removed unused versions 28.0.3 and 29.0.0: https://github.com/bazelbuild/continuous-integration/commit/050153de261cd0cca54d59760b2dc0ca95a49e0a. This means, we have build-tools 28.0.2, 29.0.2, 29.0.3 and 30.0.1 available on the CI machines.

Before this change, Bazel would have used build-tools 29.0.3, after this change it would use 30.0.1.

Bazel at HEAD and Bazel 3.6 are compatible with build-tools 30.0.1, but Bazel 3.5 is not yet compatible.

I believe the commits that added the compatibility are these:

• https://github.com/bazelbuild/bazel/commit/6b591a7e20f05224898b31ad5b03ba514eff6118
• https://github.com/bazelbuild/bazel/commit/7a11752a8ae7689d2bd482e23d466cb44a3261a1

You can check Google bug b/158484558 for some context.

I think as a way forward we have two options:

• Pin Android build-tools to 29.0.3 in the WORKSPACE of Bazel 3.5.x. This is the lowest risk, but a bit annoying as we do not have this as a commit we can cherry-pick, so we would somehow have to get it into master first and then cherry-pick it and then change master again to be pinned to build-tools 30.0.1.
• Or we can cherry-pick the above commits.

[aiuto]

I am trying the cherrypicks. I'll know if this works in about 50 minutes.

scripts/release/release.sh create --force_rc=1 \
    3.5.1 889bc0b523b47eeb38a72bf9bb6858ee525a7c7e \
    a7a0d48fbeb059ee60e77580e5d05baeefdd5699 \
    b37c51c7085f0aefe04034dd451acb847605ddb5 \
    f6ad35fcde93f92c591778ed7db38d167f5bbc03 \
    39bc97eab295bddb35b38bfc4a2ff3d2b15d034e \
    b9706675a7abf6ceebb250f0b3dfa4087a0c35f6 \
    e574d558da17cfd0f818e7a937a07926aa270069 \
    9993785fa0c4fa4172aa31d306f3abea76833abf \
    b3ac8f60973ba60d578ae6a653cdd993a2d206d7 \
    26cbf776e0cdd8fbb9a40aca4c1be6f5313b6eb4 \
    b9bb10252a004047f9a997165aa1ab7d6c6fa2e7 \
    6b591a7e20f05224898b31ad5b03ba514eff6118 \
    7a11752a8ae7689d2bd482e23d466cb44a3261a1

[aiuto]

Pushed release: 3.5.1rc1

• https://releases.bazel.build/3.5.1/rc1/index.html
• https://github.com/bazelbuild/bazel/tree/release-3.5.1rc1

@vbatts @petemounce @excitoon : Could you please test that this can be repackaged in your respective systems.

[aiuto]

Since yesterday was a holiday, I will let this age one more business day before dropping the "rc1" and upgrading to 3.5.1

[philwo]

Can we release Bazel 3.5.1? Due to a bug in Bazelisk (https://github.com/bazelbuild/bazelisk/issues/170), certain builds around the world are failing until we do so.

[aiuto]

I will tonight. I was hoping to get a thumbs up from the downstream
packagers, but I don't want to keep you blocked.

On Wed, Sep 30, 2020, 5:35 PM Philipp Wollermann [e-mail]
wrote:

Can we release Bazel 3.5.1? Due to a bug in Bazelisk (
bazelbuild/bazelisk#170
https://github.com/bazelbuild/bazelisk/issues/170), certain builds
around the world are failing until we do so.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/11885#issuecomment-701659498,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHDX2IFUPEH53574L5TSIOQCXANCNFSM4PTA3RRQ
.

[aiuto]

I just pushed release: 3.5.1

https://github.com/bazelbuild/bazel/releases/tag/3.5.1

@vbatts @petemounce @excitoon : Could you please repackage this for your respective systems.

[vbatts]

Fedora and Centos builds are kicked off: https://copr.fedorainfracloud.org/coprs/build/1691954

[vbatts]

I forgot to remove the cherry-picked patches, so that build failed.
Kicked off a new build: https://copr.fedorainfracloud.org/coprs/build/1692951

[philwo]

Bazel 3.5.1 arm64 binaries uploaded to GitHub and https://releases.bazel.build/3.5.1/release/.