Bazel: Why does Bazel navigate symbolic links when finding BUILD files?

[crorvick]

Description of the problem / feature request:

Bazel navigates into symbolic links when searching for BUILD files. This can be divided into two scenarios:

1. The symbolic link points to another location within the workspace. In this case, all paths reachable through the link are already reachable without it, thus there is no reason to continue.
2. The symbolic link points to a location outside of the workspace. This seems like a dubious use case. Is this really an intended feature? I would expect everything outside of the workspace should be pulled in using a workspace rule.

Description of the problem / feature request:

Do not traverse symbolic links when looking for BUILD files in a workspace.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ git clone https://github.com/crorvick/bazel-hello.git
Cloning into 'bazel-hello'...
remote: Enumerating objects: 22, done.
remote: Total 22 (delta 0), reused 0 (delta 0), pack-reused 22
Unpacking objects: 100% (22/22), done.
$ mkdir empty
$ cd empty/
$ touch WORKSPACE
$ ln -s ../bazel-hello .
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 1f6bc1d1-68d2-4c64-ab9c-20f55cfb9ba7
Loading: 
Loading: 0 packages loaded
ERROR: error loading package 'bazel-hello/src/hello': Unable to load package for '//src/optimize:optimize.bzl': BUILD file not found on package path
INFO: Elapsed time: 1.294s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded)
FAILED: Build did NOT complete successfully (2 packages loaded)

A slightly more complex, but less contrived scenario:

$ git clone https://github.com/crorvick/bazel-hello.git
Cloning into 'bazel-hello'...
remote: Enumerating objects: 22, done.
remote: Total 22 (delta 0), reused 0 (delta 0), pack-reused 22
Unpacking objects: 100% (22/22), done.
$ cd bazel-hello/
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: d8aee92d-b92a-4198-8ad1-f2db4cde17b2
Loading: 
Loading: 0 packages loaded
Analyzing: 4 targets (5 packages loaded)
Analyzing: 4 targets (5 packages loaded, 0 targets configured)
INFO: Analysed 4 targets (20 packages loaded, 505 targets configured).
INFO: Found 4 targets...
[0 / 5] no action
INFO: Elapsed time: 3.158s, Critical Path: 0.52s
INFO: 7 processes: 7 linux-sandbox.
INFO: Build completed successfully, 20 total actions
INFO: Build completed successfully, 20 total actions
$ cd ..
$ mv bazel-hello bazel-hello-bak
$ cd bazel-hello-bak
$ bazel build ...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 5c1430a6-4ff2-47c1-bf3c-fb3c5600672d
Loading: 
Loading: 0 packages loaded
ERROR: error loading package 'bazel-bazel-hello/external/bazel_tools/tools/jdk': Unable to load package for '//tools/jdk:default_java_toolchain.bzl': BUILD file not found on package path
INFO: Elapsed time: 1.343s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (8 packages loaded)
FAILED: Build did NOT complete successfully (8 packages loaded)

Bazel is tripping over its bazel-<project> symbolic link in the above example after the parent directory is renamed. A confound in resolving this particular issue is that it adding /bazel-* to the list of ignored file patterns by Git (or whatever VCS is used) is a reasonable thing to do since the bazel-<project> link is not predictable from what is under Git's control.

[laszlocsomor]

/sub

[crorvick]

bump

[laszlocsomor]

This is great question!

I don't know for sure why Bazel does this. My theory is that following these symlinks lets Bazel track changes in them or in their target, and correctly rebuild rules that depend on the link (e.g. a genrule with srcs=["a"] where a is a symlink). I also suspect that this behavior may have historical reasons, like supporting a latest symlink in some tool's package that always points to the newest checked-in version of that tool (while old versions are also kept in the depot).

Whether symlinks in the source tree are good practice or not is another question. (Especially if said source tree is also built on Windows, not just Linux.)

@ulfjack : do you have any thoughts on this topic?

[ulfjack]

Funny, I just discussed this with @buchgr. He was working on related problems, but I'm not sure what his current plan is.

[crorvick]

@buchgr @ulfjack, do you have any further thoughts on this? This continues to be an infrequent source of confusion, and it's still not clear to me why Bazel should be navigating symbolic links at all. Thanks!

[crorvick]

@jin @ulfjack, is there a good story for why Bazel navigates symbolic links? If not, I think this deserves a 1.0 tag so this can be cleaned up before it becomes difficult to do so.

[ulfjack]

Is there a good story for why it shouldn't treat symlinks like files or directories? I'm not convinced that changing this will be less confusing overall.

[crorvick]

Well, I tried to tell one in the description of this ticket. It seems like the only use case for navigating a symbolic link is a hacky way to pull something into a workspace. This feels like an odd feature for Bazel to support.

I'm not really sure what it means for Bazel to navigate links within a workspace ... it seems to just result in Bazel compiling and linking the same thing multiple times.

In summary, I cannot see how symbolic links provide anything beyond the ability to shoot yourself in the foot. It seems out of character for the Bazel project to not have a strong opinion about such a thing.

[laszlocsomor]

Related bug: https://github.com/bazelbuild/bazel/issues/6350

[laszlocsomor]

/cc @philwo

[acecilia]

I see a usecase for following symlinks:

Imagine having a complex build setup: for example, I use bazel for building swift, and the setup for having it up and running takes quiet some time and code (I have custom macros in order to avoid duplicating code all around in the BUILD files).

Now, I want to take this custom setup and reuse it for building swift code located in multiple different repositories: the code will not be inside the workspace. One way I can do this is by having all the bazel setup in a submodule, and a symlink pointing to a folder one level up, that in each of the repositories contains different packages.

This has another benefit: separation of concerns. The build setup is isolated from the code: the details of how to build the source code are isolated from the source code, which I think is good because gives the ability to change the build setup without modifying anything inside the source code files.

If not in this way, how can I reuse a bazel setup to build code in multiple repositories?

[crorvick]

If not in this way, how can I reuse a bazel setup to build code in multiple repositories?

I would think you could use a workspace rule to pull in your custom macros, something like:

local_repository(
    name = "my_custom_macros",
    path = "/home/acecilia/src/my_custom_macros",
)

And then you would pull them into a BUILD file with:

load("@my_custom_macros//:macros.bzl", "some_macro")

[acecilia]

Okey @crorvick, I did not think about that 😅 Thanks!