Bazel: Release - December 2017 - Target RC date: 2017-12-01 - name: 0.9.0

[damienmg]

Release tracking bug for December 2017:

• Target cut should be 1st of December (first business day of December)
• Name: 0.9.0

/cc @dslomov. You are planned to be the RM for this one but as I said I could exchange would fit my schedule better.

[damienmg]

So the nightly seems pretty broken right now: https://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/bazel/job/nightly/Downstream_projects/

• rules_docker is broken by 0.8.0 #4173
• rules_go has failing tests which is probably the cause for buildifier and bazel-watcher (#4172) failure
• Intellij is also broken

I guess we won't be able to get a clean release tomorrow. I won't be able to follow up much as I am traveling today.

@philwo, current buildcop, FYI

@dslomov @ulfjack I guess this is case for having the build cop be the release manager too?

[philwo]

What is a build cop? Is that something different than the sheriff?

[damienmg]

Sorry for the confusion. It is the same. I just tend to use build cop on
the issue tracker because I find the term is clearer that sheriff on what
the sheriff does.

On Thu, Nov 30, 2017, 9:32 AM Philipp Wollermann [e-mail]
wrote:

What is a build cop? Is that something different than the sheriff?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3773#issuecomment-348117820,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf7RgTWgBHBEpxmHTrWOyIpCcOQKxks5s7mgdgaJpZM4PdwG-
.

[dslomov]

let's stabilize before Monday, I think that's doable - and let's cut on
Monday if we can

On Thu, 30 Nov 2017, 10:11 Damien Martin-Guillerez, <
[email protected]> wrote:

Sorry for the confusion. It is the same. I just tend to use build cop on
the issue tracker because I find the term is clearer that sheriff on what
the sheriff does.

On Thu, Nov 30, 2017, 9:32 AM Philipp Wollermann >
wrote:

What is a build cop? Is that something different than the sheriff?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<https://github.com/bazelbuild/bazel/issues/3773#issuecomment-348117820
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ADjHf7RgTWgBHBEpxmHTrWOyIpCcOQKxks5s7mgdgaJpZM4PdwG-

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3773#issuecomment-348126950,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AExrX2dSXTGx4SwiziRPcxUzyKHpOEhLks5s7nE3gaJpZM4PdwG-
.

[philwo]

Ok! I’ll have a look at the open breakages and coordinate with Dmitry once I’m in the office in a few minutes.

[philwo]

The bazel-tests failure in process_wrapper-test should be fixed in the next nightly thanks to https://github.com/bazelbuild/bazel/commit/89ec374424a585c4923df942c848c6a88ddd9db7

[philwo]

Is there anything that I can do to help get this sorted out?

[dslomov]

See this update: https://github.com/bazelbuild/bazel/issues/4173#issuecomment-348232897
@iirina will investigate and we'll make a decision based on her fix.

[aj-michael]

4208 is breaking Android, and we don't have a fix yet.

[damienmg]

So adding @iirina this week sheriff.

Status: candidate not possible, there was no nightly due to the disconnected mac slaves. The nightly is now running, I will update the status once it has run.

[damienmg]

So the nightly failed in bootstrapping on a python error on gflags...

[damienmg]

It happens only on Ubuntu 14.04, list of changes in the nightly:

• Properly enforce one version for android_robolectric_test targets by (commit: 3c5581d)
• Remove unused adb_jobs flag. (commit: bda9ada)
• Match upstream python-gflags-3.1.0 imports (fixes #3816) (commit: fb15f0f)
• Update python-gflags to version 3.1.1 (commit: 8fd0f4a)
• Windows: Don't leak any handles from Bazel client to Bazel server (commit: 9c97bf9)
• Avoid NPEs when providers are not found in JavaInfo. (commit: 49008a3)
• Windows: Expose find_vc_path and find_msvc_tool function (commit: ff12a22)
• Prevent git filegroup from choking on gitk temporary files. (commit: 062da5c)
• Make AndroidSdkProvider.verifyPresence throw. (commit: 6a5669a)
• Move IDL constant out of AndroidSemantics (commit: c69068a)
• Remove duplicated method from AndroidBinary. (commit: dec0329)
• Add user and system execution time to provide total CPU execution time (commit: 26bc282)

So fb15f0f and 8fd0f4a as likely culprits

[damienmg]

So I got more time to investigate failures of the latest nightly that ran fully before the mac issues:

• rules_go (and buildifier and buildtools) are failing likely due to 337f19cc54e77c45daa1d5f61bf0a8d3daf8268f: i'll probably rollback and rollforward with fixing rules_go.
• intellij is broken due to --incompatible_disallow_set_constructor disappearing coming from /cc @vladmos

The other failure seems to be rules_docker which should be fixed.

[jin]

4208 is also fixed.

[laszlocsomor]

Status: looking for a suitable baseline commit for the release candidate.

Details:

• Last internal Blaze Release Candidate was cut on 2017-11-29. That's too old to base a Bazel RC on, because we'd need to cherrypick some recent bugfixes, such as the fix for #4173. My decision: look for a commit based on recent, relatively successful bazel-nightly CI runs, because bazel-nightly uses the nightly binary (built near HEAD) to run tests.
• Last bazel-nightly run: https://ci.bazel.io/view/Dashboard/job/bazel/job/nightly/144/
  
  
  
  • baseline: https://github.com/bazelbuild/bazel/commit/ddd5ac16aeffa6c4693c348f73e7365240b1abc5
  
  
  • breaks some targets.
  
  

TODO(@laszlocsomor):

• [x] create RC from ddd5ac16aeffa6c4693c348f73e7365240b1abc5
• [x] triage bazel-nightly breakages
• [x] initiate pushing out rc1
• [x] confirm rc1 is pushed
• [x] cherrypick bugfixes for bazel-nightly breakages
  
  
  
  • [x] https://github.com/bazelbuild/bazel/issues/4232: rules_go, bazel-watcher
  
  
  • [x] https://github.com/bazelbuild/bazel/issues/4233: TF, TF serving, bazel-tests
  
  
  • [x] https://github.com/bazelbuild/bazel/issues/4240: bazel-tests -- not blocking
  
  
• [x] triage internal release blockers
  
  
  
  • [x] 70181354
  
  
  • [x] 70189704
  
  
  • [x] 70243541 -- doesn't affect Bazel code
  
  
  • [x] 70249461 (update 2017-12-11: this is still open but looks like it just fails an internal performance test)
  
  
• [x] close release blockers

[laszlocsomor]

Release notes edits:

• removed "RELNOTES: None for Blaze users." entry
• removed both (duplicate) relnotes coming from a7a9d56539a467ef095d3003a16d33922d9505a5 and its rollback 944d4f754446dba33a83b003035fbae32f855bd0
• removed duplicate relnotes from a rollback-commit 0dd27ab4506baced80d0cd6d135934e363598437 and its roll-forward 1e3d568ea68e3d4b597dd40e6100295548f44e8a
• with advice from @vladmos , removed the relnotes from c5e9a4790288a24b22bc72761994775c029a676e in favor of 9bb93ee8c0edae911f9e2adeaca8aebd406788b6
• removed relnotes from 7e5c4af28115553c8076ab632b9ace8a0566b01d because it has no observable effect on Bazel

[laszlocsomor]

Created 0.9.0rc1:

(...)/release.sh create 0.9.0 ddd5ac16aeffa6c4693c348f73e7365240b1abc
Created 0.9.0rc1 on branch release-0.9.0.

[petemounce]

curl https://releases.bazel.build/0.9.0/rc1/index.html

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
</Error>

[laszlocsomor]

Not yet pushed.

[petemounce]

Ah, ok.

[laszlocsomor]

RC is now pushed:

• announcement: https://groups.google.com/forum/#!msg/bazel-discuss/js7iR3f9oew/PNbQBpK0BAAJ
• binaries: https://releases.bazel.build/0.9.0/rc1/index.html

[damienmg]

FTR the global test for 0.9 is there: https://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/bazel/job/release/491/Downstream_projects/

[laszlocsomor]

Need to cherrypick for RC2:

• https://github.com/bazelbuild/bazel/commit/2cf560f83922e6df9626ba3ee063c1caf6797548 (fixes https://github.com/bazelbuild/bazel/issues/4233)
• https://github.com/bazelbuild/bazel/commit/a2d2615362c65be98629b39ce39754a325ed1c42 (@aj-michael said: "This fixes a bug introduced in https://github.com/bazelbuild/bazel/commit/e5fb5f07ba9fe0dc40fc2d6039973dab5794f876")
• https://github.com/bazelbuild/bazel/commit/68c577afc2fb33b5e66b820bcc9043fed1071456

Maybe need to cherrypick (waiting for answer):

• ~https://github.com/bazelbuild/bazel/commit/e4eb23fe5512760b0e4a80c19a98f43875051fb1~: update: @haxorz confirmed that no cherrypick is necessary
• ~https://github.com/bazelbuild/bazel/commit/67566e5ed617469ac0b4fdaf18631ae701df92af~: update: @lberki confirmed that this is unimportant and no cherrypick is necessary

[laszlocsomor]

RC2 is cut and pushed:

$ (...)/release.sh create 0.9.0 ddd5ac16aeffa6c4693c348f73e7365240b1abc 2cf560f83922e6df9626ba3ee063c1caf6797548 a2d2615362c65be98629b39ce39754a325ed1c42 68c577afc2fb33b5e66b820bcc9043fed1071456
$ (...)/release.sh push

[laszlocsomor]

Test results of rc2: https://ci.bazel.io/view/Bazel%20bootstrap%20and%20maintenance/job/bazel/job/release/533/Downstream_projects/

[petemounce]

https://chocolatey.org/packages/bazel/0.9.0-rc1 & https://chocolatey.org/packages/bazel/0.9.0-rc2 are available.

[dslomov]

4250 is not blocking

[laszlocsomor]

Need to cherrypick for rc3:

• commit 766ba8adc4487f17ebfc081aeba6f34b18b53d6c

[laszlocsomor]

Creating rc3. I'll update the thread when it's pushed.

  $ (...)/release.sh create 0.9.0 \
          ddd5ac16aeffa6c4693c348f73e7365240b1abc \
          2cf560f83922e6df9626ba3ee063c1caf6797548 \
          a2d2615362c65be98629b39ce39754a325ed1c42 \
          68c577afc2fb33b5e66b820bcc9043fed1071456 \
          766ba8adc4487f17ebfc081aeba6f34b18b53d6c
(...)
Created 0.9.0rc3 on branch release-0.9.0.

  $ (...)/release.sh push
(...)
To github.com:bazelbuild/bazel
   8e1f466e3..b1903dcb0  refs/notes/cherrypick -> refs/notes/cherrypick

[laszlocsomor]

Pushed.
Post-submit test results: https://ci.bazel.io/job/bazel/job/release/559/Downstream_projects/

[laszlocsomor]

The only post-submit failures are:

• protobuf: already failing with 0.8.1, see https://github.com/bazelbuild/bazel/issues/4250
• TF serving: same as https://github.com/bazelbuild/bazel/issues/4252

[petemounce]

https://chocolatey.org/packages/bazel/0.9.0-rc3 is up.

[laszlocsomor]

Great, thanks Peter!

[laszlocsomor]

@davidstanke and @ola-rozenfeld requested cherrypick for RC4:

• https://github.com/bazelbuild/bazel/commit/a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe

[laszlocsomor]

Created new RC and initiated push:

  $ (...)/release.sh create 0.9.0 \
          ddd5ac16aeffa6c4693c348f73e7365240b1abc \
          2cf560f83922e6df9626ba3ee063c1caf6797548 \
          a2d2615362c65be98629b39ce39754a325ed1c42 \
          68c577afc2fb33b5e66b820bcc9043fed1071456 \
          766ba8adc4487f17ebfc081aeba6f34b18b53d6c \
          a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe

I needed to resolve a merge conflict:

diff --cc src/test/shell/bazel/remote_execution_test.sh
index 601bcc9e0,5eadb1d1a..000000000
--- a/src/test/shell/bazel/remote_execution_test.sh
+++ b/src/test/shell/bazel/remote_execution_test.sh
@@@ -29,11 -30,10 +30,15 @@@ function set_up() 
    while [ $attempts -le 5 ]; do
      (( attempts++ ))
      worker_port=$(pick_random_unused_tcp_port) || fail "no port found"
++<<<<<<< HEAD
 +    hazelcast_port=$(pick_random_unused_tcp_port) || fail "no port found"
 +    "${bazel_data}/src/tools/remote_worker/remote_worker" \
++=======
+     "${bazel_data}/src/tools/remote/worker" \
++>>>>>>> a22d0e9c1... Fix: uploading artifacts of failed actions to remote cache stopped working.
          --work_path="${work_path}" \
          --listen_port=${worker_port} \
-         --hazelcast_standalone_listen_port=${hazelcast_port} \
+         --cas_path=${cas_path} \
          --pid_file="${pid_file}" >& $TEST_log &
      local wait_seconds=0
      until [ -s "${pid_file}" ] || [ "$wait_seconds" -eq 15 ]; do

Per advice from @ola-rozenfeld , I resolved it as:

   while [ $attempts -le 5 ]; do
      (( attempts++ ))
      worker_port=$(pick_random_unused_tcp_port) || fail "no port found"
      "${bazel_data}/src/tools/remote_worker/remote_worker" \
            --work_path="${work_path}" \
            --listen_port=${worker_port} \
            --cas_path=${cas_path} \
            --pid_file="${pid_file}" >& $TEST_log &

[laszlocsomor]

Cherrypick request for RC5:

• [x] CL 178732031 to fix bug 70522059 ("Bazel generates duplicated actions if java_common.compile is called twice")

[iirina]

CL 178732031 to fix bug 70522059 ("Bazel generates duplicated actions if java_common.compile is called twice")

Fix commit is 03964c8ccb20d673add76c7f37245e837c3899b6

[laszlocsomor]

@iirina : Thanks!

RC4 push failed, and I messed up my local git clone, so I'm recreating RC4 from scratch. (I'm updating the release guide for future Bazel release managers; I'm pretty sure I know what I'm doing.)

  $ (...)/release.sh create 0.9.0 \
          ddd5ac16aeffa6c4693c348f73e7365240b1abc \
          2cf560f83922e6df9626ba3ee063c1caf6797548 \
          a2d2615362c65be98629b39ce39754a325ed1c42 \
          68c577afc2fb33b5e66b820bcc9043fed1071456 \
          766ba8adc4487f17ebfc081aeba6f34b18b53d6c \
          a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe \
          03964c8ccb20d673add76c7f37245e837c3899b6

[laszlocsomor]

Pushed: https://releases.bazel.build/0.9.0/rc4/index.html

[laszlocsomor]

Unfortunately, the cherrypick a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe needs the commit that added the //src/tools/remote package, namely 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf. I'll create RC5 with that in it. /cc @ola-rozenfeld

https://ci.bazel.io/job/Global/job/bazel-tests/403/consoleFull

[node=linux-x86_64] ERROR: /home/ci/workspace/Global/bazel-tests-node=linux-x86_64/src/test/shell/bazel/BUILD:423:1: no such package 'src/tools/remote': BUILD file not found on package path and referenced by '//src/test/shell/bazel:remote_execution_rest_test'

[laszlocsomor]

Ha! Including https://github.com/bazelbuild/bazel/commit/2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf got rid of the merge conflict from https://github.com/bazelbuild/bazel/commit/a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe in RC4.

Created RC5 and initiated push. CI started building: https://ci.bazel.io/job/bazel/job/release/603/

  $ (...)/release.sh create 0.9.0 ddd5ac16aeffa6c4693c348f73e7365240b1abc 2cf560f83922e6df9626ba3ee063c1caf6797548 a2d2615362c65be98629b39ce39754a325ed1c42 68c577afc2fb33b5e66b820bcc9043fed1071456  766ba8adc4487f17ebfc081aeba6f34b18b53d6c 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe 03964c8ccb20d673add76c7f37245e837c3899b6                                                                 
remote: Counting objects: 3443, done        
remote: Finding sources: 100% (28/28)           
remote: Total 28 (delta 11), reused 28 (delta 11)        
From sso://bazel/bazel
 * branch                master     -> FETCH_HEAD
Creating new release branch release-0.9.0 for release 0.9.0
Switched to and reset branch 'release-0.9.0'
Applying cherry-picks
  Cherry-picking 2cf560f83922e6df9626ba3ee063c1caf6797548
  Cherry-picking a2d2615362c65be98629b39ce39754a325ed1c42
  Cherry-picking 68c577afc2fb33b5e66b820bcc9043fed1071456
  Cherry-picking 766ba8adc4487f17ebfc081aeba6f34b18b53d6c
  Cherry-picking 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf
  Cherry-picking a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe
  Cherry-picking 03964c8ccb20d673add76c7f37245e837c3899b6
Creating release notes
Created 0.9.0rc5 on branch release-0.9.0.

  $ (...)/release.sh push
...
To github.com:bazelbuild/bazel
   243833b91..ac65b44c5  refs/notes/cherrypick -> refs/notes/cherrypick

[ola-rozenfeld]

It should not need that. The rest_test can reference the worker from the
old path. Note how the /src/tools/remote is not a new package, I just
moved /src/tools/remote_worker to /src/tools/remote.

If you do cherrypick the move, be mindful to change the references in
remote_execution_test and remote_execution_rest test appropriately (and the
BUILD file as well).

On Tue, Dec 12, 2017 at 10:44 AM, László Csomor [e-mail]
wrote:

Ha! Including 2f3d7df
https://github.com/bazelbuild/bazel/commit/2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf
got rid of the merge conflict from a22d0e9
https://github.com/bazelbuild/bazel/commit/a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe
in RC4.

Created RC5 and initiated push. CI started building:
https://ci.bazel.io/job/bazel/job/release/603/

$ (...)/release.sh create 0.9.0 ddd5ac16aeffa6c4693c348f73e7365240b1abc 2cf560f83922e6df9626ba3ee063c1caf6797548 a2d2615362c65be98629b39ce39754a325ed1c42 68c577afc2fb33b5e66b820bcc9043fed1071456 766ba8adc4487f17ebfc081aeba6f34b18b53d6c 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe 03964c8ccb20d673add76c7f37245e837c3899b6
remote: Counting objects: 3443, done
remote: Finding sources: 100% (28/28)
remote: Total 28 (delta 11), reused 28 (delta 11)
From sso://bazel/bazel

• branch master -> FETCH_HEAD
  Creating new release branch release-0.9.0 for release 0.9.0
  Switched to and reset branch 'release-0.9.0'
  Applying cherry-picks
  Cherry-picking 2cf560f83922e6df9626ba3ee063c1caf6797548
  Cherry-picking a2d2615362c65be98629b39ce39754a325ed1c42
  Cherry-picking 68c577afc2fb33b5e66b820bcc9043fed1071456
  Cherry-picking 766ba8adc4487f17ebfc081aeba6f34b18b53d6c
  Cherry-picking 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf
  Cherry-picking a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe
  Cherry-picking 03964c8ccb20d673add76c7f37245e837c3899b6
  Creating release notes
  Created 0.9.0rc5 on branch release-0.9.0.

$ (...)/release.sh push
...
To github.com:bazelbuild/bazel
243833b91..ac65b44c5 refs/notes/cherrypick -> refs/notes/cherrypick

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3773#issuecomment-351090548,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYoKuLRIwS69RDhF_SdOOjA_R6lxuFtgks5s_p9rgaJpZM4PdwG-
.

[laszlocsomor]

OK, I created RC6 without 2f3d7df5750ec7f3ac293c92c31d2969af2dd1cf and with manual edits while cherrypicking a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe in order to resolve the merge conflict (same way as earlier) and to change references of src/tools/remote{,/remote} to src/tools/remote_worker{,/remote_worker}:

  $ (...)/release.sh create 0.9.0 ddd5ac16aeffa6c4693c348f73e7365240b1abc 2cf560f83922e6df9626ba3ee063c1caf6797548 a2d2615362c65be98629b39ce39754a325ed1c42 68c577afc2fb33b5e66b820bcc9043fed1071456  766ba8adc4487f17ebfc081aeba6f34b18b53d6c a22d0e9c14e58b29d81f5a83bdcc6e5fce52eafe 03964c8ccb20d673add76c7f37245e837c3899b6

[petemounce]

RCs 4-6 inclusive are available from chocolatey.

[laszlocsomor]

Great, thank you Peter!

[laszlocsomor]

Post-submit tests are https://ci.bazel.io/job/bazel/job/release/617/
The only errors are:

• TF serving: this looks like https://github.com/bazelbuild/bazel/issues/4289, and it's not a Bazel issue
• TF syntaxnet: this is https://github.com/bazelbuild/bazel/issues/4290, and I think that's a flake.

I'll go ahead and push RC6 to release.

[laszlocsomor]

Ooops, not yet. 2 weeks have not yet passed since RC1.
I'm pushing it tomorrow then.

[laszlocsomor]

Pushed release:

• https://github.com/bazelbuild/bazel/releases/tag/0.9.0
• https://releases.bazel.build/0.9.0/release/index.html

@petemounce : Could you please update Chocolatey? Thanks!

@ilovezfs : I heard you were maintaining the Homebrew recipe. If that's indeed the case, thank you! Could you please update with 0.9.0? If you aren't, do you know who is?

[ilovezfs]

Voilà https://github.com/Homebrew/homebrew-core/pull/21879

[laszlocsomor]

The last thing before we can close this bug is to update Chocolatey.

[petemounce]

Sorry, did that earlier, forgot to actually comment. https://chocolatey.org/packages/bazel/

[laszlocsomor]

Yay, thanks to both of you!