Bazel: Bazel excessively re-fetches external repositories

[AlessandroPatti]

Description of the problem / feature request:

Since 2.2.0, Bazel fetches external repository more frequently. The culprit seems to be a3a221597, as the STARLARK_SEMANTIC value stored in the marker file changes with the commands.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I am able to constantly reproduce it executing a run or build command followed by a query command.

What operating system are you running Bazel on?

MacOS 10.15.3

What's the output of bazel info release?

release 2.2.0

[laurentlb]

This might happen if you have different flags for build and for query.
Do you have a bazelrc file? If so, there might be an issue there.

If you have:

build --some_flag

You can enable it for query too. "common" means both "build" and "query", so you can try:

common --some_flag

[AlessandroPatti]

Syncing all flags in StarlarkSemanticsOptions.java using common did solve the issue indeed, thanks!

[keith]

I can confirm this fixed it for us too, in our case it was --incompatible_allow_tags_propagation.

There's more conversation about this here too https://bazelbuild.slack.com/archives/CA31HN1T3/p1583983830261700

But I wonder if we should force these flags to only ever be in a common group in the .bazelrc so no one else can accidentally make this mistake?

[aiuto]

Changing the flags passed to Bazel can force the reevaluation of the WORKSPACE. This is expected, but Bazel can spend a lot of time redownloading git repositories.

When I run bazelisk --migrate test ... in Stardoc, the command takes 15 minutes. Almost all the time is spent fetching and refetching git repositories.

Small repro:

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "io_bazel",
    commit = "b2b9fa00d7d168e9553cfc9fe381928e8e246176",  # 2020-07-28
    remote = "https://github.com/bazelbuild/bazel.git",
    shallow_since = "1595950714 -0700",
)

BUILD:

load("@io_bazel//:distdir.bzl", "distdir_tar")

Example commands to run:
Copied from #11864

$ bazel build ...
$ bazel build ... --record_rule_instantiation_callstack
$ bazel build ... --record_rule_instantiation_callstack --incompatible_restrict_string_escapes

Most of the incompatible flags force a redownload. Each of the command spends around one minute downloading io_bazel, according to the console output:

     Fetching @io_bazel; Cloning b2b9fa00d7d168e9553cfc9fe381928e8e246176 of https://github.com/bazelbuild/bazel.git 58s

[jin]

There's a new report filed at #11895.

[TheButlah]

Same issue here, except it doesn't appear as though my build and query commands _can_ be unified to use the same arguments, so the workaround given doesn't help in my case.

Bazelrc File:

# Allows for build:macos syntax
common --enable_platform_specific_config

# Build with clang-11 by default on linux (mac uses xcode clang)
build:linux --repo_env=CXX=clang++-11 --repo_env=CC=clang-11

# Clang is missing the global sized delete operator, but gcc has it by default. Make
# clang also use it same as gcc.
build --cxxopt="-fsized-deallocation"

# Timeout after 5 seconds for remote
build --remote_timeout=5
fetch --remote_timeout=5

# Fixes issue where debug symbols point to sources in deleted sandbox, by not deleting the sandbox
build:macos --sandbox_debug #--spawn_strategy=local

# Ensures warnings are always printed out, even when cached
build --experimental_replay_action_out_err

# Be more aggressive about avoiding questionable code
build --cxxopt="-Wall"

I've gone through them and they don't appear to be valid commands for query, unless I missed something. In this case, is there a fix or workaround? What is the source of my woes?

This is a big issue for my teams workflow because we use bazel query every time we import a python package in order to properly import built artifacts, and re-fetching dependencies each time slows things down a large amount

[laurentlb]

Can you identify which flag causes the problem?

Bazel will do a reload if a flag (that affects the loading phase) changes. If you identify which flag is problematic, we can look for a solution (it's possible that a Bazel flag doesn't behave well; or it can be a problem in your bazelrc).

[TheButlah]

@laurentlb it is the bazel build --repo_env that causes the fetch/query thrashing. Both bazel query and bazel fetch do not have a --repo_env flag, but its presence in bazel build causes bazel to consider the fetch and build steps to be different, hence the thrashing.

[laurentlb]

Thanks, I've filed a separate issue for this.

[igorgatis]

Is there a way to spot the flag which is causing this issue? Say, comparing logs?