E.g.
<component name="libraryTable">
<library name="grpc-core-1.7.0-ijar_b557f595">
<CLASSES>
<root url="jar:///private/var/tmp/_bazel_michael.schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/bazel-out/darwin-fastbuild/genfiles/external/org_pubref_rules_protobuf/java/_ijar/grpc_compiletime_deps/external/io_grpc_grpc_core/jar/grpc-core-1.7.0-ijar.jar!/" />
</CLASSES>
<JAVADOC />
<SOURCES />
</library>
</component>
-> % ls /var/tmp/_bazel_michael.schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/bazel-out/darwin-fastbuild/genfiles/external/org_pubref_rules_protobuf/java/_ijar/grpc_compiletime_deps/external/io_grpc_grpc_core/jar/
grpc-core-1.7.0-ijar.jar
compared to what is found in bazel-repo
-> % ls /private/var/tmp/_bazel_michael.schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/external/io_grpc_grpc_core/jar/
BUILD.bazel grpc-core-1.7.0-sources.jar grpc-core-1.7.0.jar
Is there a reason we should not be using the repo location for libraries?
We cache jars locally for correctness (the contents of bazel-out can change between syncs) and performance (not relevant if bazel-out is on the local disk).
The underlying problem here is that the source jar wasn't detected during sync. Do you have an example project showing how it was produced?
Ive setup a small example project as an example, but I see that it is resolving sources correctly. I will reimport my main project where I have been having issues, and will check if sources are properly resolved after this.
After removing the project workspace and re-importing from scratch, I am able to view dependency sources. I'm not sure what was wrong with the first project creation. Ill close this issue for now.
I'm still having issues with this.
When I try to replicate the problem with a smaller project sources are correctly resolved, however I see some differences that might be worth noting.
In my main project, some source files are correctly found and others are not.
For classes whose sources are correctly discovered, the path is always in:
/private/var/tmp/_bazel_schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/external/...
For classes whose sources are not correctly discovered, the path is always in:
/private/var/tmp/_bazel_schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/bazel-out/darwin-fastbuild/genfiles/external/...
In regards to my original comment in the issue, in either case the intellij libraryTable record refers to the the cached copy
/private/var/tmp/_bazel_schiff/dcad08dfc7b1c329418c14a917d7f378/execroot/__main__/bazel-out/darwin-fastbuild/genfiles/external/...
For classes whose sources are not discovered, I can manually attach them by navigating to the source in the same root as for classes which can be discovered.
Please let me know if there is any further information I can provide. I will add more here as I find it.
I'm having the same issue.
I made sure that all dependencies where imported using java_import_external and have a srcjar.
Those srcjars are present in /private/var/tmp/...../execroot/__main__/external, but are not used by intellij.
@brendandouglas is there any kind of debug logging that I could turn on that might help me to understand why the source jars are not being picked up during sync?
It's a bit tricky to debug -- ideally we'd have a project reproducing this, so we can debug it ourselves.
The steps I'd follow:
okay, here's the intellij txt for commons_codec:
Intellij always sends me to the ijar when I try to lookup the source code for a class in this package.
build_file_artifact_location {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "BUILD"
root_execution_path_fragment: "external/commons_codec"
}
deps {
dependency_type: 0
target {
label: "//bazel:java8_toolchain"
}
}
java_ide_info {
jars {
interface_jar {
is_external: true
is_new_external_version: true
is_source: false
relative_path: "_ijar/commons_codec/external/commons_codec/commons-codec-1.11-ijar.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/genfiles/external/commons_codec"
}
jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "commons-codec-1.11.jar"
root_execution_path_fragment: "external/commons_codec"
}
source_jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "commons-codec-1.11-src.jar"
root_execution_path_fragment: "external/commons_codec"
}
source_jars {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "commons-codec-1.11-src.jar"
root_execution_path_fragment: "external/commons_codec"
}
}
}
key {
label: "@commons_codec//:commons_codec"
}
kind_string: "java_import"
tags: "mvn:artifact:commons-codec:commons-codec:1.11"
i have two intellij-info.txt files for my external dependencies. I am using bazel-deps to manage maven dependencies, not sure if this the reason for having two files. Both are included below.
build_file_artifact_location {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/BUILD.bazel"
root_execution_path_fragment: "external/commons_cli_commons_cli"
}
deps {
dependency_type: 0
target {
label: "@bazel_tools//tools/jdk:toolchain_jdk8"
}
}
java_ide_info {
jars {
interface_jar {
is_external: true
is_new_external_version: true
is_source: false
relative_path: "jar/_ijar/jar/external/commons_cli_commons_cli/jar/commons-cli-1.3.1-ijar.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/genfiles/external/commons_cli_commons_cli"
}
jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/commons-cli-1.3.1.jar"
root_execution_path_fragment: "external/commons_cli_commons_cli"
}
}
}
key {
label: "@commons_cli_commons_cli//jar:jar"
}
kind_string: "java_import"
build_file_artifact_location {
is_external: false
is_new_external_version: true
is_source: true
relative_path: "3rdparty/jvm/commons_cli/BUILD"
root_execution_path_fragment: ""
}
deps {
dependency_type: 0
target {
label: "@bazel_tools//tools/jdk:toolchain_jdk8"
}
}
deps {
dependency_type: 0
target {
label: "@commons_cli_commons_cli//jar:jar"
}
}
java_ide_info {
jars {
jar {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_cli/libcommons_cli.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
source_jar {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_cli/libcommons_cli-src.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
source_jars {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_cli/libcommons_cli-src.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
}
}
key {
label: "//3rdparty/jvm/commons_cli:commons_cli"
}
kind_string: "java_library"
The build target string I am using to reference this library is
//3rdparty/jvm/commons_cli:commons_cli
the jar referenced by the source_jar path does exist
-> % ls -l bazel-out/darwin-fastbuild/bin/3rdparty/jvm/commons_cli/libcommons_cli-src.jar
-r-xr-xr-x 1 schiff wheel 288 Aug 3 12:31 bazel-out/darwin-fastbuild/bin/3rdparty/jvm/commons_cli/libcommons_cli-src.jar
interestingly, the source jars here are empty
-> % jar -tf bazel-out/darwin-fastbuild/bin/3rdparty/jvm/commons_cli/libcommons_cli-src.jar
META-INF/
META-INF/MANIFEST.MF
More information:
I removed all bazel directories: rm -rf /private/var/tmp/_bazel_schiff/
ran bazel clean --expunge
and then synced the project. Intellij now finds sources.
checking the content of the intellij-info.txt again shows that both now include references to the source jars. Both listed below in the state in which intellij is able to find sources
build_file_artifact_location {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/BUILD.bazel"
root_execution_path_fragment: "external/commons_lang_commons_lang"
}
deps {
dependency_type: 0
target {
label: "@bazel_tools//tools/jdk:toolchain_jdk8"
}
}
java_ide_info {
jars {
interface_jar {
is_external: true
is_new_external_version: true
is_source: false
relative_path: "jar/_ijar/jar/external/commons_lang_commons_lang/jar/commons-lang-2.6-ijar.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/genfiles/external/commons_lang_commons_lang"
}
jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/commons-lang-2.6.jar"
root_execution_path_fragment: "external/commons_lang_commons_lang"
}
source_jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/commons-lang-2.6-sources.jar"
root_execution_path_fragment: "external/commons_lang_commons_lang"
}
source_jars {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/commons-lang-2.6-sources.jar"
root_execution_path_fragment: "external/commons_lang_commons_lang"
}
}
}
key {
label: "@commons_lang_commons_lang//jar:jar"
}
kind_string: "java_import"
The source jars specified here have all of the expected classes
-> % jar -tf /private/var/tmp/_bazel_schiff/dcad08dfc7b1c329418c14a917d7f378/external/commons_cli_commons_cli/commons-cli/commons-cli/1.3.1/commons-cli-1.3.1-sources.jar
META-INF/
META-INF/MANIFEST.MF
org/
org/apache/
org/apache/commons/
org/apache/commons/cli/
org/apache/commons/cli/AlreadySelectedException.java
org/apache/commons/cli/AmbiguousOptionException.java
org/apache/commons/cli/BasicParser.java
org/apache/commons/cli/CommandLine.java
org/apache/commons/cli/CommandLineParser.java
org/apache/commons/cli/DefaultParser.java
org/apache/commons/cli/GnuParser.java
org/apache/commons/cli/HelpFormatter.java
org/apache/commons/cli/MissingArgumentException.java
org/apache/commons/cli/MissingOptionException.java
org/apache/commons/cli/Option.java
org/apache/commons/cli/OptionBuilder.java
org/apache/commons/cli/OptionGroup.java
org/apache/commons/cli/Options.java
org/apache/commons/cli/OptionValidator.java
org/apache/commons/cli/overview.html
org/apache/commons/cli/package-info.java
org/apache/commons/cli/ParseException.java
org/apache/commons/cli/Parser.java
org/apache/commons/cli/PatternOptionBuilder.java
org/apache/commons/cli/PosixParser.java
org/apache/commons/cli/TypeHandler.java
org/apache/commons/cli/UnrecognizedOptionException.java
org/apache/commons/cli/Util.java
META-INF/LICENSE.txt
META-INF/NOTICE.txt
build_file_artifact_location {
is_external: false
is_new_external_version: true
is_source: true
relative_path: "3rdparty/jvm/commons_lang/BUILD"
root_execution_path_fragment: ""
}
deps {
dependency_type: 0
target {
label: "@bazel_tools//tools/jdk:toolchain_jdk8"
}
}
deps {
dependency_type: 0
target {
label: "@commons_lang_commons_lang//jar:jar"
}
}
java_ide_info {
jars {
jar {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_lang/libcommons_lang.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
source_jar {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_lang/libcommons_lang-src.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
source_jars {
is_external: false
is_new_external_version: true
is_source: false
relative_path: "3rdparty/jvm/commons_lang/libcommons_lang-src.jar"
root_execution_path_fragment: "bazel-out/darwin-fastbuild/bin"
}
}
}
key {
label: "//3rdparty/jvm/commons_lang:commons_lang"
}
kind_string: "java_library"
The source jars specified here are still empty.
-> % jar -tf bazel-out/darwin-fastbuild/bin/3rdparty/jvm/commons_lang/libcommons_lang-src.jar
META-INF/
META-INF/MANIFEST.MF
For another reason, I had to "Invalidate Caches / Restart" - again sources are not resolving. I checked the intellij-info.txt after syncing on restart, source jars are all still referenced. @brendandouglas your next suggestion would be to step through BlazeJavaWorkspaceImporter#importWorkspace ?
I'm using bazel-deps. I've noticed BlazeSourceJarNavigationPolicy does the right thing, attaching the source jar, when bazel-$workspace/external/ ($execution_root/external from bazel info) contains an entry for the external library. Sometimes $execution_root/external doesn't contain the library. That's when source navigation for external libraries fails for me. The set of libraries in $execution_root/external depends on which bazel command was last run (IIUC). So after bazel build //... all libraries are there but if some other command is run, then some libraries won't be linked. So some source jars won't attach..
$output_base/external from bazel info always contains the jar and sources jar, once bazel-deps has downloaded them.
Would it make sense to use bazelInfo.getOutputBase() instead of bazelInfo.getExecutionRoot() here? https://github.com/bazelbuild/intellij/blob/c279bd605818d660a37e5a118c5a30761e821fb5/base/src/com/google/idea/blaze/base/sync/workspace/ArtifactLocationDecoderImpl.java#L42-L46
The hack above completely breaks Sync from IJ. Source navigation works but IJ is missing class files for all external libraries after Sync. Oops. The hack in https://github.com/uri-canva/intellij/commit/eea28777f3428efa9d93cd9661dca6ee3b7dd79b#commitcomment-31465511 seems to work slightly better: sources can be resolved and Sync seems to work.
Would it make sense to always access source jars from $execution_root/external? Their presence under $execution_root/external appears to depend on what bazel command was last run. Making source navigation in IJ appear broken with the bazel plugin.
I also have this issue that external library sources can not be found in our project. So I was trying to reproduce an issue at the minimal sample. Unfortunately I still wasn't able to do it and I have to mention that somehow magically navigaton to sources happened to work in the main project also ( But it breaks down from time to time, I didn't manage to find the reason).
But I found something weird regarding to external dependencies. I'm not sure that my concern is an issue at all. So I have a sample project - https://github.com/hsestupin/external-deps-sources-ijwb-issue
What user should expect in regards to navigation to external dependencies? In my case the sync project action generates the following guava descriptor (file .ijwb/.idea/libraries/guava_27_0_jre_ijar_ecb3e810.xml):
<component name="libraryTable">
<library name="guava-27.0-jre-ijar_ecb3e810">
<CLASSES>
<root url="jar:///private/var/tmp/_bazel_jetbrains/33536c80f81de6f09ac9062b0300b1b3/execroot/__main__/bazel-out/darwin-fastbuild/genfiles/external/guava/jar/_ijar/jar/external/guava/jar/guava-27.0-jre-ijar.jar!/" />
</CLASSES>
<JAVADOC />
<SOURCES />
</library>
</component>
As you see there is no SOURCES defined. It leads to the following behaviour - when I try to navigate to com.google.common.base.Strings from the example.Application class I see sources of Strings.java but they are not properly parsed and analyzed. That means, for example, that I'm not able to invoke find usages on methods of this class. It's just a plain text.
So the question is - is it correct to expect that external dependencies should be properly indexed just like in maven-based projects?
Bazel version - 0.18.1-homebrew
Plugin - 2018.10.22.0.2
@hsestupin IIUC this explains why there is no SOURCES. The IJ plugin uses a different API to intercept and display sources. One that's marked as being removed in 2019.. https://github.com/bazelbuild/intellij/blob/c279bd605818d660a37e5a118c5a30761e821fb5/java/src/com/google/idea/blaze/java/libraries/BlazeSourceJarNavigationPolicy.java#L44-L56
I'm just a user trying to understand.. Hopefully a Googler or maintainer can confirm.
breaks down from time to time
My belief about how this happens https://github.com/uri-canva/intellij/commit/eea28777f3428efa9d93cd9661dca6ee3b7dd79b#commitcomment-31465511. There's a bazel aspect that writes intellij-info.txt files containing paths. Depending on what bazel command you (or the bazel plugin) last ran some of those paths won't exist under $execution_root. They will always exist under $output_base. Once the libraries have been downloaded.
Internally, we're currently running a build of the plugin with these changes https://github.com/uri-canva/intellij/commits/2018-12-11. Ideally we can contribute a solution that can be upstreamed.
The IJ plugin uses a different API to intercept and display sources. One that's marked as being removed in 2019.
I've read that javadoc and it actually makes perfect sense. But are there actually any plans on supporting library indexing? For example, may be it's possible to have a separate lib output folder which won't be reshuffled after every blaze build.
I don't think there's a general solution to persisting output artifacts indefinitely under $output_base.
We try to work around this for jars by maintaining a local cache, populated at sync time. If this is working as intended, it should prevent the issue you're seeing.
Do you have 'Settings > Other Settings > Bazel Settings > Use a local jar cache' enabled?
With option Settings > Other Settings > Bazel Settings > Use a local jar cache enabled it can't even find sources. Navigation brings me to the class file.
However I see guava jar file was added - .ijwb/.blaze/libraries/guava-27.0-jre-ijar_525231bb.jar. And library description .ijwb/.idea/libraries/guava_27_0_jre_ijar_ecb3e810.xml looks like this:
<component name="libraryTable">
<library name="guava-27.0-jre-ijar_ecb3e810">
<CLASSES>
<root url="jar://$PROJECT_DIR$/.blaze/libraries/guava-27.0-jre-ijar_525231bb.jar!/" />
</CLASSES>
<JAVADOC />
<SOURCES>
<root url="jar://$PROJECT_DIR$/.blaze/libraries/guava-27.0-jre-src_525231bb-src.jar!/" />
</SOURCES>
</library>
</component>
It seems that sync action didn't manage to find a source jar.
p.s. All this I'm trying on the sample project which was mentioned above - https://github.com/hsestupin/external-deps-sources-ijwb-issue
That library description looks correct -- the source jars have been both copied locally and attached to the library.
So that suggests the problem is associating a particular class from the ijar with the source jar. That's not something specific to the bazel plugin, though perhaps bazel is formatting the ijar in some way which isn't handled?
Ignore that last comment, it does look like there's a problem copying the source jar locally -- I'll take a look.
The root cause of these problems was artifacts derived from external workspaces being improperly resolved, triggered by a change to the bazel output directory structure combined with some old compatibility code in our aspect.
Should be fixed for the next release (at least, the issues with that sample project appear to be fixed)
Should be fixed for the next release (at least, the issues with that sample project appear to be fixed)
Is this only when using a local jar cache, only when not using it, or both?
Either, though if you're not using the local jar cache there's still the possibility output files may disappear before they're accessed.
Tested https://github.com/bazelbuild/intellij/commit/02b148713798c71d86121e3c22ca5ef5470cb08b and https://github.com/bazelbuild/intellij/commit/f1b794bab165ed95683916f242e9d18b5022ef14, both with and without local jar cache, still don't have sources.
Given the changelog in https://github.com/bazelbuild/intellij/commit/f1b794bab165ed95683916f242e9d18b5022ef14 that one should have the fix mentioned above, but it doesn't seem to cover our use case.
I can confirm that it still doesn't work. The sample project is this one. Enabling local jar cache doesn't have any impact.
I can't reproduce any issues with that exact same project, but I don't know how you're building the plugin.
I'd wait and see if it's still broken in the next release uploaded to JetBrains' plugin repo (scheduled for next Friday).
I mean I'm trying on the very this project https://github.com/bazelbuild/intellij
I don't build plugin by myself. I use idea 2018.3 and the corresponding plugin (the last one).
Ah, that makes sense then. The fix is not in the latest release, but will be in the next one (v2018.12.03).
I rebuilt the plugin from 02b148713798c71d86121e3c22ca5ef5470cb08b on IntelliJ 2018.3 and am still seeing the same issues in my project. intellij-info.txt files seem to find the correct source files, but they are not attached automatically.
build_file_artifact_location {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/BUILD.bazel"
root_execution_path_fragment: "external/com_amazonaws_aws_java_sdk_core"
}
deps {
dependency_type: 0
target {
label: "@bazel_tools//tools/jdk:toolchain_hostjdk8"
}
}
java_ide_info {
jars {
interface_jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/aws-java-sdk-core-1.11.319.jar"
root_execution_path_fragment: "external/com_amazonaws_aws_java_sdk_core"
}
jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/aws-java-sdk-core-1.11.319.jar"
root_execution_path_fragment: "external/com_amazonaws_aws_java_sdk_core"
}
source_jar {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/aws-java-sdk-core-1.11.319-sources.jar"
root_execution_path_fragment: "external/com_amazonaws_aws_java_sdk_core"
}
source_jars {
is_external: true
is_new_external_version: true
is_source: true
relative_path: "jar/aws-java-sdk-core-1.11.319-sources.jar"
root_execution_path_fragment: "external/com_amazonaws_aws_java_sdk_core"
}
}
}
key {
label: "@com_amazonaws_aws_java_sdk_core//jar:jar"
}
kind_string: "java_import"
When I manually attach these jars, for example external/com_amazonaws_aws_java_sdk_core/jar/aws-java-sdk-core-1.11.319-sources.jar all classes are correctly displayed
@brendandouglas what do you mean by the contents of bazel-out can change between syncs? Doesn't the content change symmetrically (class jar + sources)?
The contents of blaze-out aren't guaranteed to be unchanged in the time between syncs -- any blaze commands run separately to sync (on the command line, or running a test) could interfere with the contents of blaze-out.
Do you mean that while the plugin tries to read the files something can
change the files under its feet?
On Fri, Dec 21, 2018 at 9:21 PM brendandouglas notifications@github.com
wrote:
The contents of blaze-out aren't guaranteed to be unchanged in the time
between syncs -- any blaze commands run separately to sync (on the command
line, or running a test) could interfere with the contents of blaze-out.—
You are receiving this because you commented.Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/intellij/issues/225#issuecomment-449475949,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFzWrUqzAR19lFiag-aS-99EteRefks5u7TSegaJpZM4RzIL5
.
Yes, if it's relying on the files being in bazel-out. That's why we use a local jar cache by default, representing a snapshot of what was in bazel-out at sync time.
we use a local jar cache by default
I don't have that setting enabled, I suspect it used to not be the default before? Should I enable that setting manually even if I never changed it? It's unclear what happens when defaults are changed across plugin versions. Is this something we can add to the .bazelproject file so I can check it in and have it under control?
You're right, that feature is disabled for bazel by default. It's certainly fine to enable it, especially if you're making changes to bazel-out between syncs.
Most helpful comment
The root cause of these problems was artifacts derived from external workspaces being improperly resolved, triggered by a change to the bazel output directory structure combined with some old compatibility code in our aspect.
Should be fixed for the next release (at least, the issues with that sample project appear to be fixed)