Bazel: Bazel installer stopped working after d3f8efc, if project has .bazelversion file

[davido]

After installing of bazel 2.0.0rc3 I am seeing this:

  $ cd gerrit
  $ cat .bazelversion
  2.0.0rc3
  bazelisk build :gerrit
  019/12/02 21:55:13 Downloading https://releases.bazel.build/2.0.0/rc3/bazel-2.0.0rc3-linux-x86_64...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 893fd804-3bed-41d5-8b7c-dda793e67520
INFO: Analyzed target //:release (419 packages loaded, 8377 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/sandbox
INFO: From Compiling 656 JavaScript files to polygerrit-ui/app/polygerrit_ui_closure_bin.js:
WARNING - Failed to resolve sourcemap at bazel-out/k8-fastbuild/bin/polygerrit-ui/app/custom-style-interface.min.js.map: bazel-out/k8-fastbuild/bin/polygerrit-ui/app/custom-style-interface.min.js.map
  Codes: SOURCEMAP_RESOLVE_FAILED

0 error(s), 1 warning(s)

Target //:release up-to-date:
  bazel-bin/release.war
INFO: Elapsed time: 154.755s, Critical Path: 99.96s
INFO: 435 processes: 3 remote cache hit, 330 linux-sandbox, 8 local, 94 worker.
INFO: Build completed successfully, 457 total actions

As expected, bazel was downloaded by the bazelisk:

  $ ls -all ~/.cache/bazelisk/bin/bazelbuild/bazel-2.0.0rc3-linux-x86_64
-rwxr-xr-x 1 davido users 43780359 Dec  2 21:55 /home/davido/.cache/bazelisk/bin/bazelbuild/bazel-2.0.0rc3-linux-x86_64

Now, if I try to build with bazel directly, bypassing bazelisk, it is failing:

  $ bazel build :gerrit
ERROR: The project you're trying to build requires Bazel 2.0.0rc3 (specified in /home/davido/projects/gerrit2/.bazelversion), but it wasn't found in /home/davido/.bazel/bin.

Bazel binaries for all official releaeses can be downloaded from here:
  https://github.com/bazelbuild/bazel/releases

You can download the required version directly using this command:
  (cd "/home/davido/.bazel/bin" && curl -LO https://releases.bazel.build/2.0.0rc3/release/bazel-2.0.0rc3-linux-x86_64 && chmod +x bazel-2.0.0rc3-linux-x86_64)

Now, I have to fetch bazel twice? It also doesn't change if I install Bazel locally, I still see the same breakage, even though bazel-2.0.0rc3 was installed:

  $ bash bazel-2.0.0rc3-installer-linux-x86_64.sh --user
  [...]
  Uncompressing.......

Bazel is now installed!

Make sure you have "/home/davido/bin" in your path. You can also activate bash
completion by adding the following line to your :
  source /home/davido/.bazel/bin/bazel-complete.bash

See http://bazel.build/docs/getting-started.html to start a new project!

And bazel still doesn't work:

  $ bazel version
  Build label: 2.0.0rc3
  $ cd gerrit

  $ bazel build :gerrit
ERROR: The project you're trying to build requires Bazel 2.0.0rc3 (specified in /home/davido/projects/gerrit2/.bazelversion), but it wasn't found in /home/davido/.bazel/bin.

Bazel binaries for all official releaeses can be downloaded from here:
  https://github.com/bazelbuild/bazel/releases

You can download the required version directly using this command:
  (cd "/home/davido/.bazel/bin" && curl -LO https://releases.bazel.build/2.0.0rc3/release/bazel-2.0.0rc3-linux-x86_64 && chmod +x bazel-2.0.0rc3-linux-x86_64)

[meisterT]

cc @meteorcloudy @philwo

[meteorcloudy]

This is caused by https://github.com/bazelbuild/bazel/commit/d3f8efca72a0e2013467c033df1bf58dd76e8a10

@philwo
Looks like this is not only affecting users installing Bazel through APT repo, but also with bash bazel-2.0.0rc3-installer-linux-x86_64.sh --user. Maybe we should add a feature to check the current installed Bazel already matches the given version in .bazelversion? So that users don't run into the having to install Bazel twice problem.

[meteorcloudy]

Maybe we can package the version.txt file into the debian package and installer, then make the shell script check the Bazel version here?

[davido]

Interestingly, bazel project itself is not affected, and:

  $ cd bazel
  $ bazel build src:bazel-bin-dev

just works. However, all projects that have .bazelversion stopped working after d3f8efc.

So the workaround for now is to remove .bazelversion file from the tree.

Also, I would have expected, that higher version number semantic is still preserved. If .bazelversion file has "1.0.0" version, and I have installed bazel version 3.2.1, then it should just work. And not telling me that the .bazelversion has version 1.0.0, but currently higher version 3.2.1 is installed. The only reason to refuse running bazel, if the installed version is lower than the one referenced in .bazelversion file.

[dslomov]

Is this a release blocker for 2.0?

[philwo]

@dslomov Yes. I'm looking into this now.

[philwo]

So, here are some thoughts on this:

If a project has a .bazelversion file, it means it "should" be built with that exact Bazel version. Because the .bazelversion file is used to basically "version your build tool with your source code", I do not want to implement fuzzy matching like "as long as it's the same major version, it's fine". It will use and require the exact version, so this is working as intended.

Let's assume you have a project with a .bazelversion file that specifies "1.1.0" as the required Bazel version, then here's what happens:

• If you use Bazelisk, it will automatically and transparently use the correct version.
• If you installed Bazel via apt-get and you don't have the bazel-1.1.0 package installed, it will suggest to run the command to install that package.
• If you install Bazel via the installer, it will suggest the command to download Bazel 1.1.0 and put it into the correct place.
• If you use a plain Bazel binary (no wrapper script), it will ignore the .bazelversion file and just use whatever Bazel version you have.

I think this behavior makes sense and is pretty user friendly?

However, I agree that there's one optimization we should add: If the .bazelversion asks for Bazel X.Y and the Bazel version that was installed via apt-get install bazel or the installer script is already Bazel X.Y, then we do not need to download the same version once again. (Basically what Yun suggested.)

I'll think of some logic for the wrapper script that supports this, then we can do a cherry-pick and it should be fine.

[philwo]

Also, I don't understand this reasoning:

However, all projects that have .bazelversion stopped working after d3f8efc.
So the workaround for now is to remove .bazelversion file from the tree.

No, of course not - the workaround is not to remove the .bazelversion file, the "workaround" is to simply do what the (IMHO pretty helpful) error message suggests - install the required Bazel version :) I don't see how this is "broken" or how "Bazel stopped working".

Edit: @davido Sorry, to be clear, I totally understand why having to download the same Bazel version twice (once as the "latest" Bazel and once as the "specifically versioned package" Bazel) is weird and not a good UX. I'll fix this before we release Bazel 2.0. The general behavior should be fine and an improvement, though, or what do you think?

[davido]

The general behavior should be fine and an improvement, though, or what do you think?

+1. Totally agreed. Thank you for the clarification!

Sorry for not very clear bug report, but I was totally confused by the Bazel installer call to download the same Bazel version second time, so confused, that I have not even tried to do, what the error message told, but went ahead and created the error report.

[Helcaraxan]

We would like to kindly request this issue to be reopened as it does appears to have broken both Bazel and bazelisk at the same time in a significant manner without acceptable workarounds with the 2.0.0 release.

The requirements of CI and an organisation of 100+ engineers mean that the way in which we manage versions for bazel must be _fully automated across all three supported platforms_ in a seamless way. This includes:

• Detecting what version of bazel needs to be used for a given invocation depending on configuration. The version might be different depending on the repository that is being built / tested.
• If the required version is not available on the current machine (local workstation, laptop, CI agent, …) then download it.
• Invoke the required version’s bazel binary with the original command and arguments passed to the top-level bazel invocation.

To achieve this:

• We have historically (3+ years) been using our in-house wrapper tools/bazel script to pin and auto-manage the versions required to build our various repositories. It provides nearly identical features to what bazelisk came to provide at a later point in time.
• In an effort to reduce tech-debt we want to move to simply use bazelisk as an off-the-shelf solution.

While attempting the move to bazelisk we are introducing the required .bazelversion file. While testing this on a Ubuntu workstation that has bazel 2.0.0 installed we found out that the new logic that was added to the wrapper script for bazel’s Debian package makes our attempt at migrating to bazelisk impossible.

The repo under tests currently relies on bazel 1.2.1. When running bazel version with our tools/bazel script stripped to simply invoke bazelisk, we would expect it to invoke bazelisk version which would:

1. Detect the required 1.2.1 version.
2. Detect that it is not currently available in bazelisk’s cache.
3. Download the 1.2.1 binary for Linux.
4. Invoke the freshly downloaded binary with the version command passed to bazelisk at the start.

However, instead, we found that we never even get intobazelisk or even tools/bazel and instead are presented with the already reported error message:

 -> bazel version
ERROR: The project you're trying to build requires Bazel 1.2.1 (specified in /<redacted>/.bazelversion), but it wasn't found in /usr/bin.

You can install the required Bazel version via apt:
  sudo apt update && sudo apt install bazel-1.2.1

If this doesn't work, check Bazel's installation instructions for help:
  https://docs.bazel.build/versions/master/install-ubuntu.html

Although the suggested solution is

to simply do what the (IMHO pretty helpful) error message suggests - install the required Bazel version

this manual actions is clearly not an option given the requirement of automating the entire process, which is the very goal of a tool like bazelisk, as well as Bazel itself.

From reading the code in the wrapper script we have also found that there is no way of deactivating this new logic.

As a result we are now faced with the fact that we cannot at any point in the foreseeable future migrate to bazelisk, even in the theoretical case if this feature was rolled back in a 3.0.0 release. It would still require a .bazelversion file and the mere presence of this file will trigger errors on any Debian-based machine that happens to have bazel 2.0.0 installed, since this feature does its checks before any other wrapper script or user configuration can intervene.

The questions that we have are:

1. What has been the reason to add this .bazelversion logic into the bazel wrapper script? It intermingles the functionality of bazel and bazelisk in an awkward way. The collision on the filename makes it virtually impossible to undo the implications of this change. Releasing a 3.0.0, or 2.0.1 for that matter is not sufficient as people can still have 2.0.0 installed.
2. Why is bazel trying to implement the features that bazelisk already provides in a clean and cross-platform way? The bazelisk tool is already part of the bazelbuild organisation / ecosystem on GitHub and can thus be clearly advertised for people needing the automated version management. All while leaving bazel itself do what it’s good at: building stuff in a fast and reproducible manner.
3. Why has this attempt actually been targeted at only one platform, resulting in bazelisk’s key-feature of automated version management not actually being supported all while actually breaking existing users of bazelisk on that platform?

_To be clear: if we had already been using bazelisk before the release of 2.0.0 instead of our in-house script then the release of this “feature” would have actively broken all our Linux-based developers overnight._

[philwo]

Hi @Helcaraxan and others,

sorry for the trouble you and others are having with the new feature. I'll respond tomorrow to all of your questions. I designed the new feature to not break any existing users, but obviously this didn't work out. I reopened this issue - let's see that we can fix it together to also work for you and others who check in Bazelisk as tools/bazel into their repo.

[Helcaraxan]

Hello @philwo. Thank you very much for the quick reply. It is much appreciated. 👌

We'll be waiting for the answers and are happy to collaborate in finding the best way forward. We were genuinely surprised when we realised the conundrum we had just stumbled into with this change in the behaviour of bazel. 🙂

[Helcaraxan]

cc @laurentlb for the 2.1.0 release.

[philwo]

Thanks everyone for commenting and/or upvoting @Helcaraxan's post so that I can get a good feeling for the impact and urgency here and communicate this to my colleagues. This really helps prioritize this, so that we can hopefully find a good solution for all users quickly. :)

Before Bazel 2.0, users could install Bazel using a variety of ways:

1. They could download the Bazel binaries directly (e.g. bazel-2.0.0-linux-x86_64) and check them into their source code repository or put them on their $PATH.
2. They could download the "installer script" (e.g. bazel-2.0.0-installer-linux-x86_64.sh) and install Bazel using that. This will actually install a shell-script as the bazel command, which tries to execute $REPO/tools/bazel if it exists, otherwise will execute Bazel.
3. They could install Bazel using package managers (apt-get, yum, brew, pacman, choco ...). These packages also install the above mentioned shell-script.
4. They could install Bazelisk (usually as their bazel command on the PATH), which will automatically download the latest Bazel version (or the one specified in $REPO/.bazelversion), then check if a $REPO/tools/bazel script exists and if yes, run that, otherwise run the downloaded Bazel binary.

When the wrapper or Bazelisk run $REPO/tools/bazel they pass the location of the actual Bazel binary via the $BAZEL_REAL environment variable. The tools/bazel script is then expected to do whatever pre/post steps it wants to do (e.g. strip environment variables, run Bazel in a sandbox, upload logs after each invocation, ...) and execute $BAZEL_REAL "$@" at the appropriate point in time.

All users from above list except the Bazelisk users would now regularly run into this problem: They somehow end up with a version of Bazel that doesn't match the project that they want to build (e.g. downloaded the wrong version or their package manager automatically updated Bazel over night and whamm the entire team's build is broken).

How do we fix that? Two ideas:

1. Ask people to check in Bazelisk as tools/bazel.
2. Change the wrapper script so that it understands .bazelversion and tells the user to install the correct version of Bazel using their native packaging solution (e.g. it will tell Debian users to apt-get install bazel-1.1.0, installer-script users to wget -O ... https://..., ...).

Although I would have loved option 1, it didn't sound too convincing in practice:

• The most important reason: Some people might not want to / are not allowed to / cannot download Bazel binaries from the internet in an automated way. For example, Google and other companies have IT security policies that prohibit downloading binaries from the internet as part of your build. (Yes, that means we can't use Bazelisk even though I wrote it... 🤦‍♀) My previous company ran their entire CI without internet access for security reasons and to guarantee that e.g. "PyPI is down" or "Library foobar deleted their binaries from the internet" does not take the entire company down. These people need a way to version Bazel with their code that does not rely on Bazelisk downloading stuff from the internet.
• Bazelisk is a 6 MB binary per platform and if you check that in you could just as well check your Bazel binary in, right? Whereas if you don't check-in Bazelisk but instead some script that gets Bazelisk for your current platform and then runs it - couldn't your script just directly get Bazel instead?
• Checking in Bazelisk as tools/bazel doesn't actually guarantee that it gets called - if your user is on Windows or installs Bazel by downloading Bazel binaries directly, their bazel invocation will not run tools/bazel.

So I went for option 2. The users who cannot use Bazelisk are happy that their Bazel versioning problem is now finally solved, yay. 🎉 The users who actually were using option 1 are now broken. Not-so-yay. 😞

Let's figure out how to support this without breaking anyone. Here's all combinations of users and repository setups. In the third column "Wrapper" refers to a script that simply changes the environment a bit and then delegates to $BAZEL_REAL, while "Bazelisk" refers to either Bazelisk or any other script that tries to find the correct Bazel version and then executes that (ignores $BAZEL_REAL).

My bazel is | $REPO/.bazelversion? | $REPO/tools/bazel? | Resolution
--- | --- | --- | ---
Binary | No | No | Not affected, works fine 🆗
Binary | Yes | No | Unsupported anyway 🤷‍♂
Binary | No | Wrapper | Unsupported anyway 🤷‍♂
Binary | Yes | Wrapper | Unsupported anyway 🤷‍♂
Binary | No | Bazelisk | Unsupported anyway 🤷‍♂
Binary | Yes | Bazelisk | Unsupported anyway 🤷‍♂
Shell-script | No | No | Not affected, works fine 🆗
Shell-script | Yes | No | Fixed by Bazel 2.0 🎉
Shell-script | No | Wrapper | Not affected, works fine 🆗
Shell-script | Yes | Wrapper | Fixed by Bazel 2.0 🎉
Shell-script | No | Bazelisk | Not affected, works fine 🆗
Shell-script | Yes | Bazelisk | Broken by Bazel 2.0 😭
Bazelisk | No | No | Not affected, works fine 🆗
Bazelisk | Yes | No | Not affected, works fine 🆗
Bazelisk | No | Wrapper | Not affected, works fine 🆗
Bazelisk | Yes | Wrapper | Not affected, works fine 🆗
Bazelisk | No | Bazelisk | Not affected, works fine 🆗
Bazelisk | Yes | Bazelisk | Not affected, works fine 🆗

It looks like we have to fix the case where:

• User installs Bazel using apt-get or similar that causes their bazel command to be Bazel's shell script wrapper.
• Repository has a .bazelversion file.
• Repository has a tools/bazel script that is either Bazelisk or something similar.

Do you have suggestions how to fix this? Some ideas:

1. The shell-script wrapper could detect that tools/bazel is Bazelisk or a variant and then skip its own .bazelversion check. However I can't immediately see a reliable way how to detect this.
2. We sacrifice the "Bazel is shell-script, repo has .bazelversion, tools/bazel is a simple pre-post-wrapper script" case and just always skip .bazelversion if tools/bazel exists. This might not be too bad - it would basically restore pre-2.0 behavior for this specific case only. I have to think about this a bit. 🤔
3. We completely remove the .bazelversion feature again from the shell-script wrapper and instead change Bazelisk so that it can be used by all users, even those in offline-only or security restricted scenarios. For example we could add a feature to Bazelisk so that it fetches its binaries and version-information from an internal location.
4. Other ideas?

1. What has been the reason to add this .bazelversion logic into the bazel wrapper script? It intermingles the functionality of bazel and bazelisk in an awkward way. The collision on the filename makes it virtually impossible to undo the implications of this change.

The reason was that users who could not use Bazelisk (or download binaries from the internet) still wanted a way to version Bazel with their source code (= the .bazelversion idea). The current solution makes it easy for users who install Bazel via apt-get to work with repositories that use .bazelversion files (but breaks the case where project owners assume that when a user runs bazel it will actually just run their $REPO/tools/bazel script).

Releasing a 3.0.0, or 2.0.1 for that matter is not sufficient as people can still have 2.0.0 installed.

I don't see this quite as dark - if people installed Bazel 2.0.0 in a way that gave them the shell-script wrapper that you relied on, then it usually means that they used a package manager, which will automatically update their Bazel to later versions. Also, any Bazel version could potentially be so broken that it doesn't work with your project - it's not great, but we have to fix it and then go on.

1. Why is bazel trying to implement the features that bazelisk already provides in a clean and cross-platform way? The bazelisk tool is already part of the bazelbuild organisation / ecosystem on GitHub and can thus be clearly advertised for people needing the automated version management. All while leaving bazel itself do what it’s good at: building stuff in a fast and reproducible manner.

See above - I would love if everyone could just Bazelisk. Maybe I should have pushed more for this. If you and others think that adding offline support etc. to Bazelisk is the right direction, we can try to do that instead.

1. Why has this attempt actually been targeted at only one platform, resulting in bazelisk’s key-feature of automated version management not actually being supported all while actually breaking existing users of bazelisk on that platform?

I don't think this is the case - all platforms that ship the shell-script wrapper (which are all platforms for which a tools/bazel approach would have worked) now support the .bazelversion file, too. Bazelisk supports more platforms, which is why I recommend users to directly install Bazelisk as their bazel.

If the question is about "why didn't you just add this directly to Bazel" - because I didn't want to add something that "phones home" to the internet to Bazel. User trust is super important for me and others on the Bazel team and we think your build tool shouldn't rely on the internet in order to work.

Hope this already answers some questions - I'll follow-up tomorrow in case I have any other ideas / thoughts about this and of course for any comments and questions that come up :)

[meteorcloudy]

1. The shell-script wrapper could detect that tools/bazel is Bazelisk or a variant and then skip its own .bazelversion check. However I can't immediately see a reliable way how to detect this.

What do you think we just do a grep "\.bazelversion" tools/bazel? It will even match a binary file that uses .bazelversion (tested with bazelisk, it works). If the shell wrapper found tools/bazel uses .bazelversion, then it skips checking .bazelversion itself.

Update: Just realized it won't work if tools/bazel doesn't have read permission.

[davido]

My bazel is | $REPO/.bazelversion? | $REPO/tools/bazel? | Resolution
-- | -- | -- | --
Shell-script | Yes | No | Fixed by Bazel 2.0

This broke one specific use case: too new Bazel version used, as documented in .bazelversion. Say it is 1.2.0 in .bazelversion, but Bazel 2.0.0 is used. Now it wouldn't work, unfortunately.

However, the projects may already implement bazel vesion check in WORKSPACE file, and in fact Gerrit already does that (the Starlark method would read .bazelversion and verify that the used Bazel version is higher or equal). So that the check in the use case above would succeed, because the Bazel version is higher. You can see this as minimum version check and not strict version check. As the project build tool chain maintainer, I would like to decide what is the minimum Bazel version check is supported by my stable branch. So may be exact version match is too strict check for my project?

Anyway, one way to solve the conflict for Bazel shell wrapper between:

• be dumb and just call bazel-real or
• be smart, parse .bazelversion and refuse to call bazel-real and report an error

is to make the check in shell wrapper conditional. It could be SKIP_BAZELVERSION_CHECK environment variable or just .skip_bazel_version_check_in_shell_wrapper marker file in the workspace directory.

[Helcaraxan]

Thank you for the detailed walkthrough @philwo and your thoughts on it. This is very useful to understand the context in which the various implementation choices were made. ❤️

Based on the information that you've shared we definitely understand the use-case that drove the feature forward, as well as the pain-point that it is trying to address.

---

Some general thoughts from our side:

They somehow end up with a version of Bazel that doesn't match the project that they want to build (e.g. downloaded the wrong version or their package manager automatically updated Bazel over night and whamm the entire team's build is broken).

This is exactly (one of) the problem(s) that bazelisk is trying to solve. From that perspective improving its offline support would have, in our opinion, made more sense. As it stands, the chosen implementation feels like a lack of _separation of concerns_ between bazel and bazelisk.

Checking in Bazelisk as tools/bazel doesn't actually guarantee that it gets called - if your user is on Windows or installs Bazel by downloading Bazel binaries directly, their bazel invocation will not run tools/bazel.

That is indeed true. However we're circumventing that issue on Windows specifically by ensuring that the local tools/ appears in the PATH before anything else.

See above - I would love if everyone could just Bazelisk. Maybe I should have pushed more for this. If you and others think that adding offline support etc. to Bazelisk is the right direction, we can try to do that instead.

Yes, from our perspective that would definitely be the right direction. It would match Bazelisk's aspirations very well as a tool for managing Bazel versions.

---

The crux of the issue lies in the fact that the error that is being raised is a hard one, instead of simply an explicit warning. If it had been a warning, or if the error could be deactivated as @davido suggests, then there would have been a straightforward fix for the broken users.

The preferred way forward for us would be a mix between suggestions 2 & 3:

• Simply call tools/bazel if it exists and skip any checks, etc. The presence of a tools/bazel would tend to indicate that a user "knows what they are doing" as it's not a widely advertised & well-known feature (it's actually nearly impossible to find in Bazel's documentation). Hence these users should know what to do if things break because of mismatched Bazel versions: use bazelisk or at the least do their own version checks within their WORKSPACE.
• Extend bazelisk's feature set for offline support through things like configurable download locations, etc.

• Removing the .bazelversion check from the script might be a good thing but we're not that adamant about it. It would now be possible to skip it via the tools/bazel approach. Or if really necessary we could also add a way to deactivate the check via an environment variable (please not yet another marker file).

[philwo]

Thanks for the input! I'll go with @Helcaraxan's plan and will make sure that the fixed wrapper (skip version check if tools/bazel exists) gets into Bazel 2.1.

[davido]

@philwo Thanks for adding the option to skip the check in the script wrapper.

Now, to test a minor Bazel upgrade, say 2.1.0rc4 from [1] in Gerrit workspace, where .bazelversion contains 2.0.0, I could add this tool/bazel to skip the check:

  $ cd gerrit
  $ cat tools/bazel
echo ${BAZEL_REAL}
$HOME/.bazel/bin/bazel-real "$@"

Now, after installing bash bazel-2.1.0rc4-installer-linux-x86_64.sh --user locally, I can actually test it, and the check is skipped:

  $ bazel version
/home/davido/.bazel/bin/bazel-2.0.0
Starting local Bazel server and connecting to it...
Build label: 2.1.0rc4

Of course, I could temporary update the .bazelversion in the tree, temporary remove it, or just pass USE_BAZEL_VERSION=2.1.0rc4 bazel version...

[1] https://releases.bazel.build/2.1.0/rc4/index.html

[philwo]

Thanks for testing, @davido! :)

[Helcaraxan]

Just wanted to follow up @philwo about the resulting fix that now is making its way into 2.1.0. The reactiveness and communication has been greatly appreciated.

A great thanks on behalf of the entire team. ❤️

[dslomov]

Great resolution @philwo! Do we want to have a 2.0 patch release for this?

[philwo]

@Helcaraxan What do you think - Bazel 2.1.0 should be released tomorrow unless some unexpected blocker shows up today. Do we still need a patch release in that case? I would say no, because the patch release would also have to bake for a day or so, so it wouldn't be out any quicker. 🤔

[Helcaraxan]

@philwo I agree that it wouldn't be required for the release speed of the fix. However the argument can be made that as it stands 2.0.0 is broken to some degree and thus should be patched. That's a much more subtle debate and I trust the Bazel team to make the call on that. A patch would always be appreciated but it'd be understandable if you don't want to bear with the extra work.

Sorry. Am aware I am not necessarily making things easier but that's my honest perspective.

[philwo]

Your input and perspective is always appreciated. :)

@dslomov I'll see how many cherry-picks we would need for this and will comment on the release bug of Bazel 2.0!

[philwo]

Bazel 2.0.1, 2.1.1 and 2.2.0 have been released with the fix, so I'm closing this issue.