Intellij: ijwb stores entire bazel workspace as single IntelliJ project

[roblg]

ijwb's use of a single project to represent the entire bazel workspace can result in disagreements between the IDE and bazel as to whether the code compiles properly, which results in more than a little confusion for us internally.

One easily reproducible issue is that in a workspace with two projects, only one need declare a dependency on an external jar and both projects will see the jar as a dependency according to IntelliIJ, though bazel will fail to build. I posted a simple example workspace (bazel doesn't build, but ijwb says everything is fine) on github that illustrates the issue. There's also separate, related issue that we're experiencing internally, where IntelliJ will resolve dependencies in a particular file to jars that are dependencies of an entirely unrelated project that shares no dependencies (transitive or otherwise). Bazel finds the right jars while building and running.

Are there any plans to have more granular classpaths for bazel packages in ijwb?

edit: typos

[roblg]

More context:

We're migrating a large-ish codebase (300+ projects) from maven to bazel. In some cases, different top-level projects depend on different versions (or "flavors" of the same version) of an open source library. For example, we have one project that depends on guice-4.1, and another that depends on guice-4.0-no_aop.

Concretely, what we have is:

Foo → guice-4.1.jar
Bar → guice-4.0-no_aop.jar

The two projects that depend on those jars are unrelated; there's no dependency leakage or anything. Bazel builds cleanly, and correctly; bazel query to examine the dependencies of each project individually show that they're correct.

However, if we open a file in Foo that depends on an API that only exists in guice-4.1.jar, IntelliJ marks the API usage in red, and says that it can't find that API. I suspect that this is due to the ordering in the idea workspace.xml file, because I can fix it (or break it) by twiddling that.

While I was playing around with trying to come up with a reduced test case, I also discovered that it's possible to get into situations where IntelliJ says everything is fine, but bazel doesn't build. This is an easier case to repro, so I posted that one to github.

[brendandouglas]

We have no plans for multiple IntelliJ modules per project. At the moment, the best workaround is to use multiple IntelliJ projects per Bazel workspace.

You can set the scope of the IntelliJ project to whichever set of Bazel targets constitute a sensible module, in the project view file (Bazel > Project > Open project view file).

[dfabulich]

@brendandouglas If we do that, how would we use IntelliJ to refactor? Currently we load all of our Java code into one workspace, so when we refactor, IntelliJ finds all call sites and updates them; if we split that up into multiple projects, refactorings in one project won't find call sites in the other projects, right?

This bug seems _really_ bad to me. Refactoring is one of the top features people use IDEs for. The whole point is to do the refactor and verify in the IDE that the whole project compiles without errors. If there's only one classpath for the entire workspace, then there's no way to automatically make changes that affect multiple projects and be sure that they safely compile.

[brendandouglas]

The problem is that there's a fundamental mismatch between the build units in IntelliJ and Bazel.

In IntelliJ, you can only configure settings like java language level on a per module basis. This implies one Bazel target == one IntelliJ module.

However Bazel targets operate on lists of files, IntelliJ modules on recursive directories.

[dfabulich]

I don't think I understand your point. I agree there is some mismatch between Bazeld and IntelliJ, but that mismatch is not relevant to this bug.

Just to clarify, let's walk through the test case Robert posed. The WORKSPACE declares two maven_jar rules: aopalliance and guice_4_1. There are only two BUILD files: project1/BUILD and project2/BUILD (no nested packages in subdirectories).

The project2 build file declares a simple java_library rule:

java_library(
    name='project2',
    srcs=glob(["src/java/**/*.java"]),
    deps=[
        "@aopalliance//jar",
        "@guice_4_1//jar",
    ],
)

//project2:project2 successfully builds both in the bazel CLI and also in ijwb.

The project1 build file is a copy of project2, except the dependency on @guice_4_1//jar is commented out. The Module1.java file imports a Guice class, so //project1:project1 fails the Bazel CLI build, as you'd expect.

But the project1 module compiles without errors in ijwb, because project1 has access to _all_ of the jars in the WORKSPACE, including jars that were only declared as dependencies of project2.

Your point about recursive directories is interesting but is it relevant to this bug? If project1 were a subdirectory of project2, I can see how this would not be something that ijwb could reconcile.

Since project1 and project2 are totally separate directories, AFAIK nothing about IntelliJ's module system should prevent these modules from having separate classpaths (or separate Java language levels, if it came to that).

[brendandouglas]

I agree there are cases where Bazel and IntelliJ's build units can be compatible. My point is that there are also many cases where they're not.

Any time you have build rules in the same package with conflicting dependencies or build flags, there's no way to correctly include them both in a single IntelliJ project.

As things stand, it's pretty much infeasible to autogenerate intellij modules from a bazel project. The only sensible build unit is a bazel target, but:

• there are typically multiple targets in a single directory
• typical projects have many, many targets, and IntelliJ scales poorly with number of modules (O(n^2) performance)

We could do something like introduce a special kind of bazel rule corresponding to a compilation module, and have the bazel plugin use those to generate modules (would likely require buy-in from the bazel team). We could also allow users to manually specify modules in some other way (e.g. in the .bazelproject file).

I suspect we'll eventually implement something along these lines, but not anytime soon (it's a decent amount of work, and there's little demand for it internally).

In the meantime, some possible workarounds:

• use the same version of a given dependency, wherever possible
• if practical, have the entire workspace open in a separate project, and use that for refactoring
• otherwise, have multiple projects open, and do the same refactoring in each (this is what we do -- it's not pleasant, but I don't find it overly painful)

[igorgatis]

Maybe people don't need a tool to automagically map targets into modules. It would be nice to have a rule like:

intellij_java_module(
  name = "foo",
  srcs = glob(["src/**/*.java"]),
  deps = [...],
  tests = glob(["test/**/*.java"]),
  testdeps = [...],
)

Which would map into a intellij module. Dependencies would be handled as any jars.

This plugin, as it is right now, is impractical for any real use case scenario, IMHO.

[cgruber]

I think the above is a harsher position than I'd take (having used the plugin at YouTube, a year ago, to great effect). But I do think it's a crucial feature to making the IDE experience superlative. It represents a case where the IDE and the command line are substantially out of kilter in their view of reality, and that gap is a problem.

I think there are some possible ways out of this, including possibly a code inspector that is much more aware of dependency relationships between targets, and can trigger the symbol-not-found behavior.

[cgruber]

I also think that "little demand for it internally" made a lot of sense when the comment was made, but at this point, those of us considering bazel adoption outside of Google need to know if our use-cases will be understood and taken on board by Google, so I'm really hoping that this question can be re-prioritized in that newer light, and that the "internal" (inside Google) lack of motivation for a thing doesn't always result in it being de-prioritized.

[tobad357]

To add our comments, we wouldn't either mind configuring the modules we want to avoid the problem with mismatch between bazel and a intellij project
Even better would be to be able to configure X projects with Y modules as Intellij gets a bit slow if running with to many modules

[KaoruDev]

Naive comment: What about using tags to identify what the user considers a "module" in Intellij? Something like intellij-module=module-name and then running a query like: bazel query "attr(tags, intellij-module=*, //...:*)

[ewiner]

Some promising news from the IntelliJ team, at https://blog.jetbrains.com/idea/2019/12/intellij-platform-roadmap-for-2020/ (emphasis mine):

Project Model Redesign

The project model is how the IDE represents the structure of your project – which files belong to the project, how they depend on each other, which libraries are used, and so on. IntelliJ IDEA’s project model has served us well over the years, however, it has certain limitations. First, it doesn’t support arbitrary mixing of projects of different types. For example, AppCode can open an Xcode project and Rider can open a Visual Studio solution, but there is no way to open a Gradle project and an Xcode project in the same IDE frame. Second, the project model works on the level of directories, not files, and it can’t represent different files in the same directory having different dependencies. This makes it difficult to integrate build systems such as Bazel into the IDE, and presents challenges in other scenarios.

The redesigned project model, which we internally call “workspace model”, will remove these limitations. It also brings additional benefits such as better performance during project opening, smoother synchronization with Maven and Gradle, and a nicer programming model. We’ll start with changing the internal implementation to the workspace model, and once this is stable, we’ll proceed to adding user-visible features such as combining projects of arbitrary types in the same IDE frame.

[cwaeland]

We have no plans for multiple IntelliJ modules per project. At the moment, the best workaround is to use multiple IntelliJ _projects_ per Bazel workspace.

You can set the scope of the IntelliJ project to whichever set of Bazel targets constitute a sensible module, in the project view file (Bazel > Project > Open project view file).

@brendandouglas curious how you would have multiple projects for a given Bazel workspace. When I attempt to create a new project with a different .bazelproject file I get an .ijwb already exists error.