Bazel: Add ability to exclude subdirectories from the workspace

[alexeagle]

We have a dist folder in the Angular project, which is in the .gitignore and we have configured editors and so on to ignore this folder. It's a very common setup for frontend projects. This folder also happens to get a copy of the @angular/bazel npm package, which has some BUILD files.

We want to teach Angular developers to bazel build //... but the expansion of the label wildcard descends into all subdirectories that are not external repositories.

I could make a dist/WORKSPACE file as a workaround, and add local_repository(name="_ignore", path="dist") to /WORKSPACE but then we can't tell git to ignore the whole dist folder.

I have the same problem with some dependencies, which frontend projects typically install into a node_modules/ directory in their project - then I hit things like grpc which ship to npm with a half-working Bazel setup.

It has also been observed by others in the ecosystem, for example this bug when the rxjs library is installed in node_modules and breaks the wildcard expansion: https://github.com/ReactiveX/rxjs/issues/3334

[aehlig]

I'm not sure, I would classify this as related to external repositories; the feature request to ignore certain directories is valid even if not external repositories are involved. Basically, a more flexible form (e.g., allowing to specify only via prefixes) of the --deleted_packages option is requested.

[dslomov]

Hmm, seems like the "ignored" directories should still go into the WORKSPACE file.
What would be the good design?

[ittaiz]

Hey,
This is a long standing issue as I'm fairly certain #1083 and #2460 are related if not actual duplicates.

1083 is more than 2 years old.

We have projects that are only jvm but contain node_modules in the repo. Because one git-ignores the node_modules folder, the nested WORKSPACE workaround isn't applicable.
What do you advise?

[BenTheElder]

For Kubernetes being able to ignore _output from the other (bash/make/docker based) builds would fix needing to make clean before bazel building. Bazel currently chokes on an infinite symlink expansion in _output even though we don't use anything in _output for the Bazel build.

I think a lot of projects supporting Bazel and another build system could use some functionality like ignoring directories via an option in WORKSPACE.

[ixdy]

The --deleted_packages flag doesn't work for kubernetes' use case currently.

As @BenTheElder notes, our make-based build system creates an infinite symlink to ensure it has a valid gopath, _output/local/go/src/k8s.io/kubernetes -> [workspace root].

bazel build -- //... -//vendor/... complains about this symlink chain, and --deleted_packages doesn't seem to help here, as it fails on the infinite symlink before considering it, apparently. (It's also worth noting that _output isn't a bazel package, so I'm not sure how this would work, anyway.)

[petroseskinder]

Hey,
@dslomov I have another use case that might inform how to design a solution.

At my company, we have configured our package path to be

--package_path=%workspace%:%workspace%/__fuse__

This configuration supports a workflow where users perform sparse-checkouts of our large monorepo, while still being able to bazel build code that has not been locally checked out. Packages that are not locally-checked out, are served via a read-only fuse mount.

Overall, we are fairly satisfied by this approach. However, users often run into issues when moving or deleting BUILD files. Unless they add a --deleted_packages flag, bazel will continue to pick up the package in %workspace%/__fuse__. This sometimes leads to seemingly bewildering build behavior.

If bazel could exclude subdirectories, perhaps through configuring the bazelrc or WORKSPACE, we could simply exclude the fuse version of the package whenever users deleted or move around a package.

Note, this is a supplementary issue that this fix would resolve. We are _also_ dealing with similar issues as @alexeagle that block us from executing bazel build ...

[aehlig]

Hmm, seems like the "ignored" directories should still go into the WORKSPACE file. What would be the good design?

If we want the ignored directories to be mentioned in the WORKSPACE, at least we have a hack to work around it by adding

local_repository(
  name = "ignore_dist",
  path = "dist",
)

to the WORKSPACE. Still not a proper solution, but maybe something we can live with for the time being.

[ixdy]

@aehlig that doesn't work for kubernetes' needs, at least:

$ git diff
diff --git a/build/root/WORKSPACE b/build/root/WORKSPACE
index 8c6a8c6b6d..d043d75dba 100644
--- a/build/root/WORKSPACE
+++ b/build/root/WORKSPACE
@@ -92,3 +92,8 @@ docker_pull(
 load("//build:workspace_mirror.bzl", "export_urls")

 export_urls("workspace_urls")
+
+local_repository(
+    name = "ignore_output",
+    path = "_output",
+)
$ bazel build ...
ERROR: infinite symlink expansion detected
[start of symlink chain]
[HOME]/code/kubernetes/src/k8s.io/kubernetes
[HOME]/code/kubernetes/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes
[end of symlink chain]
ERROR: Failed to get information about path, for _output/local/go/src/k8s.io/kubernetes, skipping: Infinite symlink expansion
ERROR: Infinite symlink expansion
INFO: Elapsed time: 0.226s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded)
    currently loading: hack ... (90 packages)

[aehlig]

Ah, good point. Thanks.

I'll try and see if I can find a way to change bazel to have the ability to really ignore subdirectories. I'm not sure, how feasible it is, though.

[aehlig]

Interestingly enough, bazel kind-of has support to ingore certain directories. E.g., we
could support a file .bazelignore by simply adding

diff --git a/src/main/java/com/google/devtools/build/lib/skyframe/BazelSkyframeExecutorConstants.java b/src/main/java/com/google/devtools/build/lib/skyframe/BazelSkyframeExecutorConstants.java
index f7b19ce28f..a8582ed56e 100644
--- a/src/main/java/com/google/devtools/build/lib/skyframe/BazelSkyframeExecutorConstants.java
+++ b/src/main/java/com/google/devtools/build/lib/skyframe/BazelSkyframeExecutorConstants.java
@@ -29,7 +29,7 @@ public class BazelSkyframeExecutorConstants {
       ImmutableSet.of();

   public static final PathFragment ADDITIONAL_BLACKLISTED_PACKAGE_PREFIXES_FILE =
-      PathFragment.EMPTY_FRAGMENT;
+    PathFragment.create(".bazelignore");

   public static final CrossRepositoryLabelViolationStrategy
       CROSS_REPOSITORY_LABEL_VIOLATION_STRATEGY = CrossRepositoryLabelViolationStrategy.ERROR;

However, that only mostly ignores the directories mentioned in the .bazelignore file.
Symlink loops are still not allowed in the "ignored" directories.

--
Klaus Aehlig
Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[ixdy]

Are there any plans to address symlink loops?

[ghorn]

@ixdy upgrade to latest bazel and add a .bazelignore file with a list of directories you want to ignore

[TheButlah]

This doesn't properly ignore subdirectories of the given folder