Intellij: External dependencies are not resolved in Intellij

[brownry1986]

External dependencies are consistently not able to be resolved in Intellij with the Bazel plugin.

I have a sample project that demonstrates the issue: https://github.com/brownry1986/BazelIssues

The project builds successfully via command line and syncs successfully in Intellij, however upon opening SampleMain.java the classes imported from external dependencies are not resolved (e.g. org.slf4j.Logger).

The current example uses maven_jar to define the external dependencies, however we see the same behavior when using java_import_external.

[brownry1986]

Forgot to include environment information.

IntelliJ IDEA 2018.3.2 (Ultimate Edition)
Build #IU-183.4886.37, built on December 17, 2018

JRE: 1.8.0_152-release-1343-b26 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.14.2

Bazel Plugin: v2018.12.03.0.2

[uribwix]

Anyone found a workaround for this until it'll get fixed?

Reproduced on my env with bazel 0.21.0 and latest plugin version

[brendandouglas]

I added a comment to #252 -- we can't support random genrules, as we rely on getting detailed information from starlark java providers not exposed by genrules.

[brownry1986]

I'm not sure this is related. I'm using the same sample project for simplicity, but the issue is with the slf4j dependency not being resolved. If I remove the genrule the issue still persists. I can update the sample project to reflect this if needed.

[brownry1986]

@uribwix we have found two workarounds:

1. Modify BUILD file and resync:
2. Add whitespace to the BUILD file in the package where you are seeing the issue
3. right-click on the package directory, and choose "Partially Resync"
4. this forces the package to be seen as part of the VCS working set

5. this is not ideal as once you revert or commit the whitespace change, the package is no longer in the VCS working set

6. Extend the plugin:

7. we wrote our own plugin to extend BlazeJavaSyncPlugin.java to modify the call to updateSyncState
8. in our overriding method we call super.updateSyncState passing null as the workingSet
9. this causes the javaWorkingSet in BlazeJavaWorkspaceImporter.java to also be null and activates the following code

    // Add all deps if this target is in the current working set
    if (workingSet == null || workingSet.isTargetInWorkingSet(target)) {
      // Add self, so we pick up our own gen jars if in working set
      workspaceBuilder.directDeps.add(targetKey);
      for (Dependency dep : target.getDependencies()) {
        if (dep.getDependencyType() != DependencyType.COMPILE_TIME) {
          continue;
        }
        // forward deps from java proto_library aspect targets
        TargetIdeInfo depTarget = targetMap.get(dep.getTargetKey());
        if (depTarget != null
            && JavaBlazeRules.getJavaProtoLibraryKinds().contains(depTarget.getKind())) {
          workspaceBuilder.directDeps.addAll(
              depTarget.getDependencies().stream()
                  .map(Dependency::getTargetKey)
                  .collect(Collectors.toList()));
        } else {
          workspaceBuilder.directDeps.add(dep.getTargetKey());
        }
      }
    }

I can understand why the plugin may not want to resolve all dependencies in the entire project if you are only working on a subset, but it doesn't seem to be properly detecting which dependencies to resolve. That being said, should bazel only need to resolve/download the dependency once anyway?

[brownry1986]

@brendandouglas I updated the sample project to remove the genrule and show that the issue persists.

[brendandouglas]

Hmm, I tried to reproduce with the new project, but I can't even get it to build.

I'm seeing "Worker process did not return a WorkResponse" and "Unrecognized VM option 'CompactStrings'... Could not create the Java Virtual Machine" errors during sync.

[brownry1986]

Sorry, I think this has something to do with java toolchain and the bazel version. I've added this to the .bazelrc in the project to suppress this error:

build --javacopt=-XepDisableAllChecks

It should build now.

[IljaKroonen]

Our team is having this problem since we started using bazel. I haven't reported the issue because I could not consistently reproduce it, and because we use kotlin, I thought this was some obscure bug related to kotlin support.

Our observations:

• It happens consistently in the master branch
• We can make the problem go away by creating a new branch. In the linked repo, if you open the project in IntelliJ, sync on master you will see the errors. If you then go to your terminal, and run git checkout -b toast, and sync again, the errors are gone!
• Other branches than master sometimes break aswell, and require us to make a new branch to make it work again. It's not clear to us what breaks it.
• Deleting the .git folder fixes the issue
• Copying the .git folder back makes the issue come back
• Deleting the .ijwb folder does not "reset" the broken branches

[brendandouglas]

I can't reproduce this on my own test projects, and still can't sync your project @brownry1986, even with that flag. Which OS and bazel version are you using?

[IljaKroonen]

bazel 0.21.0
plugin v2019.01.02.0.2
IntelliJ Community 183.5153.38
Ubuntu 18.04.1 LTS

[brownry1986]

@brendandouglas
I believe the "Unrecognized VM option 'CompactStrings'... error is related to jdk8 versus jdk9

Following is my environment info:
bazel: 0.20.0-homebrew
java: 1.8.0_181
os: MacOS 10.14.2
Intellij: 2018.3.2 (Ultimate Edition) Build #IU-183.4886.37, built on December 17, 2018
Bazel Plugin: 2019.01.02.0.2

I also have the following in my .bazelrc which is impacting the jdk that is used by bazel:
build --host_javabase=@local_jdk//:jdk
build --java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_vanilla
build --javacopt=-XepDisableAllChecks

This could explain why you are still getting the CompactStrings error while I am not (since my local jdk is 8). Would it be possible for you to install jdk8 and point your JAVA_HOME to it to attempt to build this sample?

However, when I modify the .bazelrc to the following the dependency resolution issues actually goes away:
build --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8

Any ideas why changing the java_toolchain would impact the dependency resolution?

[brownry1986]

Following up on this, I opened the Intellij bazel plugin project from a fresh checkout and set my project jdk to 9. Now I am getting the "Unrecognized VM option 'CompactStrings'... error when I try to sync.

@brendandouglas are you using a specific jdk when attempting to recreate the issue with my sample project? I would like to sync up my environment with yours as closely as possible.

I actually have to add a .bazelrc with the following to build the plugin from command line:

build --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
build --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8

[brendandouglas]

I think I'm using java v1.80.91.

No idea why changing the toolchain would impact dependency resolution -- I can't reproduce that bit. Adding either set of build flags to my project (java 8 or java 9) fixes the build error and doesn't exhibit the external dependency error.

[IljaKroonen]

@brendandouglas The working set seems to be of importance here (as we can see in brownry1986's workarounds). I made a screencap that illustrates the problem: https://www.youtube.com/watch?v=wHIXIchTPpc

As you can see:

• I clone from VCS and open the project: the issue is there
• I add the flags to my build file: the issue disappears
• I remove the flags: the issue reappears

But we can not conclude that the flags are the problem. If the original repo contained those flags on master, the problem would be there; if I would then remove the flags after cloning, the issue would disappear.

EDIT: more explanations and typo

[brendandouglas]

Thanks for posting the screencast. That definitely shows the problem.

You've set up JDK 9, but the plugin can't find a corresponding project jdk (you can set one up in 'File > Project Structure').

In the video, you can see the 'Sync failed' message, along with this error in the 'blaze problems' view:

"Unable to find a JDK 9 ... After configuring in the project structure dialog, sync the project again"

[IljaKroonen]

@brendandouglas It's true there is an error message but that is not what causes the problem. To demonstrate that, I made a capture with JDK 11 installed.

This time I also show how the .git folder affects the resolution:

• I import the project after cloning it: the problem is there
• I rename the .git folder to not_git: the problem disappears
• I restore the not_git folder to .git: the problem reappears

https://youtu.be/_oRoPgWXy58

[brendandouglas]

Thanks for the extra details, I can reproduce with JDK 11. The problem is that the java_library target (//module:sample-lib) isn't successfully writing out a jdeps file. We rely on jdeps to know which deps are actually required to resolve the source code.

Without it, you'll only get deps added to the project libraries for targets in the 'working set' (the set of modified files/directories) -- hence why modifying the root directory fixes the issue.

So this is an upstream problem with bazel / JDK 11. Could you file a bug on bazelbuild/bazel?

For reference, with JDK 8, bazel-bin/module/libsample-lib.jdeps contains (modulo formatting):

bazel-out/k8-fastbuild/genfiles/external/org_slf4j_slf4j_api/jar/_ijar/jar/external/org_slf4j_slf4j_api/jar/slf4j-api-1.7.7-ijar.jar
//module:sample-lib
com.sample

Building with JDK 11, it contains:

//module:sample-lib

[brownry1986]

Thanks for the additional info @brendandouglas .

It does appear that the vanilla toolchain has the same deficiency as the jdk11 toolchain.

When setting my toolchain to @bazel_tools//tools/jdk:toolchain_hostjdk8 the .jdeps file contains the dependencies. When setting my toolchain to @bazel_tools//tools/jdk:toolchain_vanilla the .jdeps file only contains the library being built. My takeaway is that we need to move away from the vanilla toolchain.

Just to summarize for anyone else; there appears to be two paths for loading dependencies into Intellij:

1. The .jdeps file for each library is loaded and external dependencies listed therein are loaded into Intellij.
2. For version controlled projects; the plugin will load additional external dependencies from packages that are determined to be in your CSV working set (changes since last commit).

Some java toolchains do not appear to properly produce the jdeps file which would result in these dependencies not being properly loaded into Intellij.

Please correct me if the above is not accurate; otherwise I consider this closed (but may file an issue with bazel for the vanilla toolchain).

[IljaKroonen]

Ooh thank you very much for the insight @brendandouglas, I now know how I can track down the issue with our kotlin project.

As for the bug in bazel, I did a quick search and found this, so I guess they are aware.

https://github.com/bazelbuild/bazel/issues/5723#issuecomment-437138252

[ghost]

Please try File | Invalidate Caches | Invalidate and Restart. It work for me.

[gergelyfabian]

Is this still a valid bug? Was this fixed on Bazel's side?

[gergelyfabian]

Coming back to this topic in May 2020. This is still an issue with Bazel 3.0.0 and the latest plugin version (2020.04.13).
With Bazel 3.0.0 it's not any more possible to provide a host_javabase with Java versions 9 and 10 (and in my experience also Java 8 does not work): https://blog.bazel.build/2020/03/31/bazel-3.0.html, and so work it around on Bazel's side.

Providing a --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java8 also does not fix this problem.

The only option that is available as a workaround is to fork the plugin as e.g. in https://github.com/uri-canva/intellij/pull/1 to enforce adding all external dependencies.

Btw, who should be fixing this bug? Bazel or the IntelliJ plugin? Or if it cannot be fixed, should there maybe be an official workaround?

[gergelyfabian]

I added a fresh fork for forcing the load of external libraries in: https://github.com/gergelyfabian/intellij/tree/v2020.05.08 (basing on 2020.04.13).

[gergelyfabian]

Coming back to this topic in May 2020. This is still an issue with Bazel 3.0.0 and the latest plugin version (2020.04.13).
With Bazel 3.0.0 it's not any more possible to provide a host_javabase with Java versions 9 and 10 (and in my experience also Java 8 does not work): https://blog.bazel.build/2020/03/31/bazel-3.0.html, and so work it around on Bazel's side.

Providing a --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java8 also does not fix this problem.

The only option that is available as a workaround is to fork the plugin as e.g. in uri-canva#1 to enforce adding all external dependencies.

Btw, who should be fixing this bug? Bazel or the IntelliJ plugin? Or if it cannot be fixed, should there maybe be an official workaround?

It seems what I wrote about is a different issue with the same symptom.

The issue in my case is that there seems to be a bug in the cooperation between intellij plugin and rules_scala.
If I use the pattern documented by rules_scala to define a scala target using Scalafmt (https://github.com/bazelbuild/rules_scala/blob/master/docs/phase_scalafmt.md), then intellij plugin won't detect it as a Scala target, but rather a Java one (because intellij plugin has hard-coded the targets it expects from rules_scala).

The Scala targets generated by rules_scala do not expose a jdeps interface, hence intellij BlazeJavaWorkspaceImporter cannot find the dependencies for them (assuming they are not in the working set).

At the same time I checked, that both proper Java targets get proper dependencies with BlazeJavaWorkspaceImporter and (detected) Scala targets get proper dependencies with intellij BlazeScalaSyncPlugin (Java and Scala use separate mechanisms for import, but both work on their own). The only problem is if a Scala target is misdetected as Java.

I created a new issue for this: https://github.com/bazelbuild/intellij/issues/1824.