Bazel: Release 0.24 - March 2019

[laurentlb]

Target RC date: March 1st

[katre]

Looks like we have a good baseline for the release in 235e76b0e756d05599a6cbe1663ff8e13df84a86. Test run is here: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/831. @meteorcloudy has investigated the two failing projects and they are not release-related.

[meteorcloudy]

@oquenchil 235e76b won't contain your flip of --incompatible_disable_legacy_cc_provider

[katre]

The bazel-at-head-plus-downstream tests that include the flag flip for
incompatible_disable_legacy_cc_provider show many failures of downstream projects related to the missing "cc" provider (see https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/834). I think this feature needs more time to migrate downstream projects.

[meteorcloudy]

@katre Yes, @oquenchil and I just agreed to rollback the flag flip

[katre]

Creating rc1:

scripts/release/release.sh create 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86

Tests are running now.

[katre]

Release is blocked by #7605: I cannot actually get the release binaries to build.

[meteorcloudy]

@katre Thanks for adding migration-0.24 for the incompatible flags didn't make into 0.24.0.
But we should not remove migration-0.22 and migration-0.23 for them, because if we are using an old version of Bazel on CI we won't be able to fetch them for testing. (For example, we currently still uses 0.22.0 because 0.23.0 has a bug and 0.23.1 is not yet released)
https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/22#98b39372-548a-43be-827f-b096f70ab30e only got 7 incompatible flags, but there are more that should be tested.

I'm adding the migration labels back for 0.22 and 0.23

[philwo]

Thank you @meteorcloudy!

[katre]

@meteorcloudy My mistake. We should update the docs at https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md and clarify that (I think I see where I went wrong, so expect a PR from me today).

[katre]

Creating rc2:

scripts/release/release.sh create --force_rc=2 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8

[brandjon]

Requesting a cherrypick for #7615. (CL submitting as we speak.)

[katre]

RC2 downstream tests pass (modulo TF setup): https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/839#_

[katre]

@brandjon Please comment in this thread with the specific commit id that needs to be cherrypicked for #7615. If there is no further work on the issue, it should be closed, otherwise the release-blocker label should be removed.

[brandjon]

Commit is 33e571939085dd158422e1b3503cfc738e0a3165.

[brandjon]

If there is no further work on the issue, it should be closed, otherwise the release-blocker label should be removed.

Not sure how to parse this -- Do you mean if all commits associated with the issue are in (it's just the one commit), the issue should be closed now, rather than after the cherrypick is merged into the RC? And that if there were still pending commits, it by definition would not block the release?

[katre]

That's correct: if the commit fixes the issue it should be closed. If the issue is still open and requires more commits to fix, then we need to figure out whether those also need to be cherrypicked.

[brandjon]

I see, thanks. In that case #7615 is closed.

[meteorcloudy]

https://github.com/bazelbuild/bazel/issues/7661 is a new blocker for 0.24

[meteorcloudy]

@katre Please cherry-pick https://github.com/bazelbuild/bazel/commit/56366ee3a73e2c92b2fa36a9840478202b9618ca to fix #7661

[katre]

Preparing RC3 with 33e571939085dd158422e1b3503cfc738e0a3165 and 56366ee3a73e2c92b2fa36a9840478202b9618ca.

scripts/release/release.sh create --force_rc=3 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8 33e571939085dd158422e1b3503cfc738e0a3165 56366ee3a73e2c92b2fa36a9840478202b9618ca

[katre]

@meteorcloudy and @brandjon: RC3 fails tests on Windows, specifically //src/test/py:test_wrapper_test: https://buildkite.com/bazel/bazel-bazel/builds/7108#d390d900-6d21-4b70-8cb9-ea37f91605fd

Is this due to either cherrypick?

[katre]

Adding 22b3fbf4800113df51d603d943bd9eb9517ef904 to fix the test_wrapper_test issue on windows.

Creating rc4

scripts/release/release.sh create --force_rc=4 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8 33e571939085dd158422e1b3503cfc738e0a3165 56366ee3a73e2c92b2fa36a9840478202b9618ca 22b3fbf4800113df51d603d943bd9eb9517ef904

[katre]

Tests pass for rc4: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/846

[juliexxia]

requesting cherrypick for https://github.com/bazelbuild/bazel/pull/7535. CL is submitted.

This very simple change is unfortunately in f14d447 d99bc47 and 3529ad7. Is there a way I can combine those to make this process easier?

[meteorcloudy]

https://releases.bazel.build/0.24.0/rc4/index.html
Why does it show so many cherry-picks there?

[philwo]

@meteorcloudy I think that's the same bug as what happened in 0.23.0:

https://groups.google.com/forum/#!msg/bazel-discuss/EOCwSvhn8l4/9T8IH9XsEQAJ

[meteorcloudy]

Oh, ok, thanks!

[sergiocampama]

Here to support @juliexxia's request. rules_apple has been waiting for this change for a long time, almost all the pieces are already there, we're just missing this last piece. This will unblock so many migrations and cleanups on our codebase, and at this point there's not a lot more work we can do on rules_apple without this change.

[katre]

@juliexxia No way to merge the commits once they are submitted: we only want to cherrypick exactly what was committed to master.

Preparing RC5 now:

scripts/release/release.sh create --force_rc=5 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8 33e571939085dd158422e1b3503cfc738e0a3165 56366ee3a73e2c92b2fa36a9840478202b9618ca 22b3fbf4800113df51d603d943bd9eb9517ef904 f14d447cb56aee563f6e686b8f5b086a3bb55d47 d99bc478db1f3414b4f6cd3dc14ca70aacf6b375 3529ad7ccf0c26dfb20a9d67b9d96de15f309f8b

[katre]

@juliexxia Please make sure that the draft release notes contain an appropriate description of this feature.

[nlopezgi]

Spoke with @katre on chat about issues with rbe_ubuntu config.
These issues are because CI is running now with this PR (https://github.com/bazelbuild/continuous-integration/pull/551). If you build with a version of bazel before this PR (https://github.com/bazelbuild/bazel/commit/de0612ad3ef7cc8c44069261befdeb0d15b97c10) the RBE config will fail to run. This should not impact other configs. @katre stated that as long as this only impacts the RBE config it should be fine.
Apologies for the issue and let me know if this is problematic to leave as is and would rather I revert https://github.com/bazelbuild/continuous-integration/pull/551

[juliexxia]

The whitelist is for starlark build configurations and the work for that is still experimental/in development. rules_apple is a special case since we've already been in talks with them but for other users, I'd like to have some semblance of documentation/on-boarding procedure in place before we open the floodgates (or announce in the release). If it's alright with you, @katre, I won't add anything to the notes for this release.

[katre]

0.24.0rc5 is now available.

[katre]

Will cherrypick 3e660ad178926648e8e10e2ee7a1a30b12f9b3d1 today to fix MacOS performance regressions.

[katre]

Creating 0.24.0rc6:

scripts/release/release.sh create --force_rc=6 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8 33e571939085dd158422e1b3503cfc738e0a3165 56366ee3a73e2c92b2fa36a9840478202b9618ca 22b3fbf4800113df51d603d943bd9eb9517ef904 f14d447cb56aee563f6e686b8f5b086a3bb55d47 d99bc478db1f3414b4f6cd3dc14ca70aacf6b375 3529ad7ccf0c26dfb20a9d67b9d96de15f309f8b de0612ad3ef7cc8c44069261befdeb0d15b97c10 3e660ad178926648e8e10e2ee7a1a30b12f9b3d1

[katre]

Downstream tests for 0.24.0rc6 passed, except for TensorFlow (expected, due to their version pin), and gmaven_rules on Windows (investigating).

[steeve]

Hi,

Would it be possible to cherry-pick this commit? This is causing major pain throughout the team (had always been, though).
https://github.com/bazelbuild/bazel/commit/d03dcfb450a8f3717d81f9ecde5ac24ec53e042a

Thanks!

[steeve]

Also, I just tried 0.24rc6 and this NPE is still there: https://github.com/bazelbuild/bazel/issues/7721, it used to work on 0.21.

[katre]

@steeve Commit d03dcfb450a8f3717d81f9ecde5ac24ec53e042a appears to have been rolled back (in c6c9b58f14ec828cc050f0e6d201316bd986cf3f) and not re-applied, so I do not want to cherrypick it into the release.

Is there a fix for #7721? Is it a release blocker? Asking @aragos to triage.

[steeve]

@katre sorry I did not see it was rolled back. in which case it's best to leave it out indeed, thank you

[keith]

@steeve this issue is tracking resubmitting that fix https://github.com/bazelbuild/bazel/issues/7592

[katre]

Creating rc7:

scripts/release/release.sh create --force_rc=7 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8 33e571939085dd158422e1b3503cfc738e0a3165 56366ee3a73e2c92b2fa36a9840478202b9618ca 22b3fbf4800113df51d603d943bd9eb9517ef904 f14d447cb56aee563f6e686b8f5b086a3bb55d47 d99bc478db1f3414b4f6cd3dc14ca70aacf6b375 3529ad7ccf0c26dfb20a9d67b9d96de15f309f8b de0612ad3ef7cc8c44069261befdeb0d15b97c10 3e660ad178926648e8e10e2ee7a1a30b12f9b3d1 314cf1f9e4b332955c4800b2451db4e926c3e092 fc586a86b614667a21e5a01aea3544ac0338de78 ea1703b30f9ac43a0c3262f5729c34658ed8d473

[katre]

All tests pass! https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/867

[or-shachar]

Do we need to change anything in our toolchains / crosstool ?
We started getting this error:

12:07:26  ERROR: /home/builduser/.cache/bazel/_bazel_builduser/d95f300509f1e228190b211abeba17c4/external/bazel_toolchains/configs/ubuntu16_04_clang/1.1/bazel_0.20.0/default/BUILD:57:1: @bazel_toolchains//configs/ubuntu16_04_clang/1.1/bazel_0.20.0/default:cc-compiler-k8: no such attribute 'dynamic_runtime_libs' in 'cc_toolchain' rule

and I don't see any comment about dynamic_runtime_libs in the release doc...
Note that we use toolchain from revision 31b5dc8c4e9c7fd3f5f4d04c6714f2ce87b126c1

[buchgr]

We unfortunately discovered a release blocker when trying to upgrade tensorflow to 0.23.2. The issue is https://github.com/bazelbuild/bazel/issues/7794. The issue also exists in 0.23.2, but given that 0.24 is so close I thin it makes sense to only cherry pick this fix in 0.24? I am currently working on the fix.

[katre]

@buchgr Thanks for the heads-up, let me know when the fix is ready.

[katre]

@or-shachar We did make some fixes for dynamic libraries, but only for objc targets, in ea1703b30f9ac43a0c3262f5729c34658ed8d473. Does this seem relevant? Please open an issue and reference it here so we can investigate. Reproduction instructions would be helpful.

[iirina]

7767 is a regression in 0.23.0, can we cherry pick the fixes 109810a8a7d6b23a8a2f3001306af1f6e2bf6c3e and ba3ebcb21da47c487fd132f0a1f3ad0a93149829?

[katre]

@iirina Sure, I'll include that with the fix from @buchgr in the next RC.

[meteorcloudy]

@katre https://github.com/bazelbuild/bazel/commit/803801d1494f06f0ce977a1f2241ef6a4d85df09 is ready for cherry-pick to fix #7794

[katre]

Creating 0.24.0rc8:

scripts/release/release.sh create --force_rc=8 0.24.0 3eb889508c8c41e5e7a04913044e8d82878cdf13 109810a8a7d6b23a8a2f3001306af1f6e2bf6c3e ba3ebcb21da47c487fd132f0a1f3ad0a93149829 803801d1494f06f0ce977a1f2241ef6a4d85df09

(Using 0.24.0rc7 as baseline because of the large number of conflicts in the previous cherrypicks.)

[katre]

And cherrypicks failed, due to the presence of "remote_java_tools_java_import".

@iirina Please advise on what else needs to be part of this.

[iirina]

d4651e86d0469f12ee2a09b02c112a14b13a6c3d is the commit that added remote_java_tools_import, which also requires 9554a033e39667be8330aa28dda3c5274ad32278 (contains third_party/* changes). 2d446750b114555ad37918cf30153eb9d15d2537 and f3944c85ab4fbd8b98b3c8d5278f54017af53238 fix some breakages that the first two commits introduced.

So the additional cherry picks that are needed are:

d4651e86d0469f12ee2a09b02c112a14b13a6c3d 9554a033e39667be8330aa28dda3c5274ad32278 2d446750b114555ad37918cf30153eb9d15d2537 f3944c85ab4fbd8b98b3c8d5278f54017af53238

[katre]

I don't feel comfortable cherrypicking that many commits, especially since in a quick test they don't apply cleanly.

Is there another fix for this issue? Can it wait for 0.25.0?

[iirina]

That were also my thoughts. I can try rewriting third_party/jarjar/BUILD.tools without using remote_java_tools_java_import, then we'll have only 3 commits to cherry pick. Would that be reasonable?

[iirina]

Actually it won't be possible to rewrite the BUILD file because we made the switch from platform-independent Java tools releases to platform-specific Java tools releases... I think the easiest right now is to wait for bazel 0.25.0.

[katre]

Creating 0.24.0rc9:

scripts/release/release.sh create --force_rc=9 0.24.0 3eb889508c8c41e5e7a04913044e8d82878cdf13 803801d1494f06f0ce977a1f2241ef6a4d85df09

[katre]

Downstream tests for 0.24.0rc9 passed: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/872

[katre]

Barring any late-arriving release blockers, I plan to finalize the release tomorrow.

[katre]

Release 0.24.0 is finalized and exists on Github: https://github.com/bazelbuild/bazel/releases/tag/0.24.0

Can external package maintainers (@vbatts @petemounce @excitoon) please update their packages when they are able to?

[vbatts]

On 26/03/19 16:52 +0000, katre wrote:

Release 0.24.0 is finalized and exists on Github:
[1]https://github.com/bazelbuild/bazel/releases/tag/0.24.0

Can external package maintainers ([2]@vbatts [3]@petemounce
[4]@excitoon) please update their packages when they are able to?

https://copr.fedorainfracloud.org/coprs/build/873974

[katre]

@vbatts Looks like there's a failure there on fedora-rawhide. I can't figure out the logs: is it a Bazel issue or a package config issue?

[katre]

Re-pinging @petemounce and @excitoon about the chocolatey and scoop packages.

[petemounce]

Published.

[vbatts]

@katre looks like the relevant section of log is:

ERROR: /builddir/build/BUILD/bazel-0.24.0/third_party/grpc/BUILD:386:1: C++ compilation of rule '//third_party/grpc:gpr_base' failed (Exit 1): gcc failed: error executing command 
BUILDSTDERR: 
ERROR: /builddir/build/BUILD/bazel-0.24.0/third_party/grpc/BUILD:386:1: C++ compilation of rule '//third_party/grpc:gpr_base' failed (Exit 1): gcc failed: error executing command 
BUILDSTDERR:   (cd /tmp/bazel_XzaXLMOY/out/execroot/io_bazel && \
BUILDSTDERR:   (cd /tmp/bazel_XzaXLMOY/out/execroot/io_bazel && \
BUILDSTDERR:   exec env - \
BUILDSTDERR:   exec env - \
BUILDSTDERR:     PATH=/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin \
BUILDSTDERR:     PATH=/builddir/.local/bin:/builddir/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin \
BUILDSTDERR:     PWD=/proc/self/cwd \
BUILDSTDERR:     PWD=/proc/self/cwd \
BUILDSTDERR:   /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.d '-frandom-seed=bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/k8-opt/genfiles -iquote bazel-out/k8-opt/bin -isystem third_party/grpc -isystem bazel-out/k8-opt/genfiles/third_party/grpc -isystem bazel-out/k8-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/k8-opt/genfiles/third_party/grpc/include -isystem bazel-out/k8-opt/bin/third_party/grpc/include -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/gpr/log_linux.cc -o bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.o)
BUILDSTDERR:   /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.d '-frandom-seed=bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.o' '-DGRPC_ARES=0' -iquote . -iquote bazel-out/k8-opt/genfiles -iquote bazel-out/k8-opt/bin -isystem third_party/grpc -isystem bazel-out/k8-opt/genfiles/third_party/grpc -isystem bazel-out/k8-opt/bin/third_party/grpc -isystem third_party/grpc/include -isystem bazel-out/k8-opt/genfiles/third_party/grpc/include -isystem bazel-out/k8-opt/bin/third_party/grpc/include -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/grpc/src/core/lib/gpr/log_linux.cc -o bazel-out/k8-opt/bin/third_party/grpc/_objs/gpr_base/log_linux.o)
BUILDSTDERR: Execution platform: @bazel_tools//platforms:host_platform
BUILDSTDERR: Execution platform: @bazel_tools//platforms:host_platform
BUILDSTDERR: [535 / 544] //third_party/protobuf/3.6.1:protoc_lib; 1s local
BUILDSTDERR: [535 / 544] //third_party/protobuf/3.6.1:protoc_lib; 1s local
BUILDSTDERR: 
third_party/grpc/src/core/lib/gpr/log_linux.cc:43:13: error: ambiguating new declaration of 'long int gettid()'
BUILDSTDERR: 
third_party/grpc/src/core/lib/gpr/log_linux.cc:43:13: error: ambiguating new declaration of 'long int gettid()'
BUILDSTDERR:    43 | static long gettid(void) { return syscall(__NR_gettid); }
BUILDSTDERR:    43 | static long gettid(void) { return syscall(__NR_gettid); }
BUILDSTDERR:       |             ^~~~~~
BUILDSTDERR:       |             ^~~~~~
BUILDSTDERR: In file included from /usr/include/unistd.h:1170,
BUILDSTDERR: In file included from /usr/include/unistd.h:1170,
BUILDSTDERR:                  from third_party/grpc/src/core/lib/gpr/log_linux.cc:41:
BUILDSTDERR:                  from third_party/grpc/src/core/lib/gpr/log_linux.cc:41:
BUILDSTDERR: /usr/include/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
BUILDSTDERR: /usr/include/bits/unistd_ext.h:34:16: note: old declaration '__pid_t gettid()'
BUILDSTDERR:    34 | extern __pid_t gettid (void) __THROW;
BUILDSTDERR:    34 | extern __pid_t gettid (void) __THROW;
BUILDSTDERR:       |                ^~~~~~
BUILDSTDERR:       |                ^~~~~~
BUILDSTDERR: third_party/grpc/src/core/lib/gpr/log_linux.cc:43:13: warning: 'long int gettid()' defined but not used [-Wunused-function]
BUILDSTDERR: third_party/grpc/src/core/lib/gpr/log_linux.cc:43:13: warning: 'long int gettid()' defined but not used [-Wunused-function]
BUILDSTDERR:    43 | static long gettid(void) { return syscall(__NR_gettid); }
BUILDSTDERR:    43 | static long gettid(void) { return syscall(__NR_gettid); }
BUILDSTDERR:       |             ^~~~~~
BUILDSTDERR:       |             ^~~~~~

(from https://copr-be.cloud.fedoraproject.org/results/vbatts/bazel/fedora-rawhide-x86_64/00873974-bazel/build.log.gz)

And from https://copr-be.cloud.fedoraproject.org/results/vbatts/bazel/fedora-rawhide-x86_64/00873974-bazel/root.log.gz
it looks like it is using gcc 9.0.1-0.11

[katre]

Scoop appears to have updated to bazel 0.24.0 (https://github.com/lukesampson/scoop/blob/master/bucket/bazel.json).

[katre]

Pinging @hlopko as the C++ expert. Do we support gcc 9?

[hlopko]

I suspect this is caused by https://github.com/bazelbuild/bazel/commit/458ab037e395f872fece67403a2a46a3ad480112#diff-6f9f3b0ebf3a405262ce1170d74d73bb (update of grpc). I will have to look into it, try to repro, isolate, and then see what can be done. If there is a fix on Bazel side, I vote for having a patch release. If it's on the grpc side, we will see how quickly we can get the fix. Stay tuned :)

[katre]

Thanks

[katre]

Should we open a tracking bug so this isn't forgotten for 0.25?

[vbatts]

Thankfully that build failure is on rawhide, which is the development prerelease (specifically to help roust early issues like this) so it isn't blocking actual users

[hlopko]

Yup :) https://github.com/bazelbuild/bazel/issues/7874

[hlopko]

@vbatts So it means Bazel will break on next Fedora release?

[vbatts]

@hlopko fedora 30 is soon to release, and the current "rawhide" will be fedora 31 about 6 months later. I bet there will be a fix before then

[werkt]

I'd like to suggest that a 0.24.1 be released which incorporates changes from #7860 - it will not apply cleanly, as it comes after 9813c58bc59fd7da990ed9f24c73016e339a334d, but it represents a serious bug in 24 tracked in #7856 that we'd much rather not wait for a cut of 25 for a fix of.

[katre]

@werkt Is that issue a release blocker? Asking @buchgr to help triage.

[werkt]

I'll defer to @buchgr on that, but we have been broken since 0.22 with this particular issue (0.23 introduced the break), are still broken on 0.24, won't be fixed until 0.25 without intervention, and have only had a chance to properly diagnose it now (we were late to adopt 0.23 as well).

[katre]

Makes sense for a patch release. I want to wait for the result on #7874 before I cut it, however.

[hlopko]

Oh wait gcc-9 is not yet released :) Ok I say https://github.com/bazelbuild/bazel/issues/7874 doesn't warrant a patch release.

[vbatts]

On 29/03/19 03:50 -0700, Marcel Hlopko wrote:

Oh wait gcc-9 is not yet released :) Ok I say [1]#7874 doesn't warrant
a patch release.

yeah. Fedora Rawhide, is literally that. It's meant as an early canary.
:-)

[katre]

SInce #7874 isn't critical, I will cut a patch release today that includes 80bff99811356ea5ec2a66f0217df75a56923069 (to address #7856).

[katre]

I tried to create 0.24.1rc1:

./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 80bff99811356ea5ec2a66f0217df75a56923069

However, I cannot merge this cherry-pick: the code has changed too far in between and I have no idea what should actually be fixed.

@werkt and @buchgr Please give me directions (or, better yet, a commit I can apply to 0.24.0) on how to resolve this. Or wait for the 0.25.0 RC to be delivered early next week.

[werkt]

Working one up based on release-0.24.0 now

[werkt]

https://github.com/werkt/bazel/tree/retry-exception-fix-0.24 contains a single commit that addresses the issue and applies to the base commit of 0.24

[katre]

Okay, second attempt at 0.24.1rc1:

$ git branch -D release-0.24.1
$ git remote add werkt [email protected]:werkt/bazel.git
$ git fetch werkt
$ ./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 c56c489119e6587975964c44ceb9e429ad950736

Waiting on tests and binaries now.

[katre]

0.24.1rc1 is now out: https://releases.bazel.build/0.24.1/rc1/index.html
@werkt Please test this.

[werkt]

Tested, this correctly generates an error that would result in a fallback, excellent!

[werkt]

This release candidate looks acceptable from my end, is there any update on this?

[katre]

I am releasing it today (two business days have passed with no new release blockers).

[katre]

0.24.1 is now released (https://github.com/bazelbuild/bazel/releases/tag/0.24.1).

@petemounce, @vbatts, and @excitoon: can you update the packages you each maintain? Thanks!

[vbatts]

fedora and centos build:
https://copr.fedorainfracloud.org/coprs/build/876626

[petemounce]

Pushed.

[menny]

Bazel crash immediately with error:

/var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: relocation error: /var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: symbol _ZTVNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEE, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference

On Linux gew1-buildagent-ssd-0933-d.gew1.spotify.net 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

0.23.1 works fine.

[katre]

What version of glibcxx are you using?

@hlopko, is this something that is easy to fix?

[hlopko]

@menny, do you use your own C++ toolchain? Did you migrate for https://github.com/bazelbuild/bazel/issues/6861, including enabling features that used to be enabled by default in the past:

• dependency_file
• random_seed
• module_maps
• module_map_home_cwd
• header_module_compile
• include_paths,
• pic
• preprocessor_define

[hlopko]

Oh ignore me, I misuderstood the issue.

[katre]

@menny @hlopko Please file this as a separate issue to ensure it isn't missed for 0.25.

[hlopko]

Hi @menny - is this still an issue? Can you repro with a clean install of Ubuntu 14.04 (docker will be fine)? Would you mind creating a new issue and pinging me there so we can investigate properly? Thanks!

[menny]

@hlopko, It's still an issue on some of our CI pools (the ones that have ubuntu 14).
I'll try to reproduce it on a Docker container.

[katre]

I am closing this issue since 0.24.1 is fully released. Once the followup issue about GLIBCXX_3.4.21 is triaged and we determine if a further patch to 0.24 is required, I will re-open.

[menny]

@hlopko I've decided to upgrade the old CI machine to Ubuntu 18.
The crash is not an issue any longer.