Bazel: How bazel deal with fat jar and relocating issue?

[zhexuany]

I am working on a java project. We were use maven for building until we cann't bear with the slowness. We decide to move to bazel. We figures almost everything out but two things we can not find any docs about these. First one is fat jar. Second one is shading. I hope someone from the community could give us some hits about how to do these.

We already googled everywhere but did not find satisfied solution. We noticed that there are some open issue across GitHub similar with our situation. But these issues have not been done yet. Maybe it is time to collect some thoughts and give an introduction about how to do these things.

Thanks in advance. :)

[ittaiz]

Have you tried deploy_jar? It's a fat jar
On Tue, 16 May 2017 at 9:39 Yang Zhexuan [e-mail] wrote:

pingcap/tikv-client-java#30
https://github.com/pingcap/tikv-client-java/issues/30

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3011#issuecomment-301689689,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF75Vk2wSZvkaRMoGWo_FPfB7ssDFks5r6USkgaJpZM4NcASa
.

[zhexuany]

@ittaiz I could not find any docs related with deploy_jar.

[ittaiz]

https://bazel.build/versions/master/docs/be/java.html#java_binary_implicit_outputs
Specifically:

name_deploy.jar: A Java archive suitable for deployment (only built if explicitly requested).
Building the _deploy.jar target for your rule creates a self-contained jar file with a manifest that allows it to be run with the java -jar command or with the wrapper script's --singlejar option. Using the wrapper script is preferred to java -jar because it also passes the JVM flags and the options to load native libraries.

The deploy jar contains all the classes that would be found by a classloader that searched the classpath from the binary's wrapper script from beginning to end. It also contains the native libraries needed for dependencies. These are automatically loaded into the JVM at runtime.

If your target specifies a launcher attribute, then instead of being a normal JAR file, the _deploy.jar will be a native binary. This will contain the launcher plus any native (C++) dependencies of your rule, all linked into a static binary. The actual jar file's bytes will be appended to that native binary, creating a single binary blob containing both the executable and the Java code. You can execute the resulting jar file directly like you would execute any native binary.

[zhexuany]

We actually tried that. But we could not figure out how to trigger this implicit output. Could you please tell me how to?

[ittaiz]

Sure thing-
Given a target named baz in package //java/com/example/foo you should call bazel build //java/com/example/foo:baz_deploy.jar. Note the _deploy.jar suffix.
In general btw I think you should really try to use SO and tag your question with Bazel since this doesn't sound like an issue with Bazel itself.

[zhexuany]

@ittaiz That works. Maybe docs need to be updated? To me, after reading doc I could not figure out to do fat jar in this way. Actually, if bazel team does not have free hand, I can do this instead.

How about another issue? The relocating issue? We used maven-shade plugin to do this. Is there any thing in bazel can do the same thing as we did before with maven?

[ittaiz]

There is a solution but unfortunately I don't remember it off the top of my
head.
I suggest you ask that on SO.
Bazel team members (as well as community members) monitor the tag and help
out.

On Tue, May 16, 2017 at 3:27 PM Yang Zhexuan [e-mail]
wrote:

@ittaiz https://github.com/ittaiz That works.

How about another issue? The relocating issue? We used maven-shade plugin
to do this. Is there any thing in bazel can do the same thing as we did
before with maven?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3011#issuecomment-301766377,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF57O96FVaD-NtS4kWl9LB2H8H3G8ks5r6ZZFgaJpZM4NcASa
.

[zhexuany]

Thanks. I did not find any useful info from SO. I leaves this issue open in case of any others people run into same situation.

[ittaiz]

Did you ask in SO a question about how to do shading?
On Mon, 22 May 2017 at 9:04 Yang Zhexuan [e-mail] wrote:

Thanks. I did not find any useful info from SO. I leaves this issue open
in case of any others people run into same situation.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3011#issuecomment-303006883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF-iKo6x-ty9NRcFPOTVkzEBndbo5ks5r8SVmgaJpZM4NcASa
.

[zhexuany]

I did. But no one gives me answer about this particular issue.

[ittaiz]

That's strange. I try to monitor the tag bazel and don't remember seeing
it. Link? I'll try to see (though I'm really really not sure I know the
answer)
On Mon, 22 May 2017 at 15:09 Yang Zhexuan [e-mail] wrote:

I did. But no one gives me answer about this particular issue.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3011#issuecomment-303081862,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF2tYkrvjbry-lXofk0fTLNZsGJYpks5r8XrdgaJpZM4NcASa
.

[promiseofcake]

This is still an issue when attempting to bundle applications for the Spark runtime and it's provided Hadoop transitive dependencies (protobuf) being out of date (2.x, need 3.x).

[ittaiz]

I think @johnynek has a jarjar rule which might be what you're looking for.
@damienmg where would you put such a rule? java_rules? Standalone repo?

[promiseofcake]

@ittaiz that would be great! I tried playing with jarjar but it seemed to get caught up in the generated protobuf code. ~Is it already a Bazel rule, that would be wonderful.~ Going to give it a try, thank you: https://github.com/johnynek/bazel_jar_jar

[ittaiz]

I don't have such a rule but like I mentioned I've heard others who have it.
Sounds like something we need in the ecosystem.
I've mentioned the two people I know who might be able to help
On Fri, 10 Nov 2017 at 6:14 Lucas Kacher [e-mail] wrote:

@ittaiz https://github.com/ittaiz that would be great! I tried playing
with jarjar but it seemed to get caught up in the generated protobuf code.
Is it already a Bazel rule, that would be wonderful.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/3011#issuecomment-343483934,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF6NY0ptb_qLc0myy96tiCEUiMNOsks5s1FpAgaJpZM4NcASa
.

[johnynek]

we use it in a few places. Works so far for us.

[ittaiz]

@lberki
@hmemcpy from my team has been working on the having parity with maven shade/assembly plugins and it seems that between singlejar (the utility that java/scala deploy jars use) and @johnynek's jarjar rule we're almost set.
The one thing we need is the ability to filter out some jars from the deploy jar.
One use-case is that your org uses jackson with version X but your runtime has a different version Y. Assuming they're compile time compatible you don't want to use jarjar as that's a heavy tool and just want to filter out the jackson jars from the uber jar.
In maven we would use provided/exclude from assembly for that.

[hmemcpy]

Yes! Turns out that in our particular case, a combination of using SingleJar (via the deployable.jar implicit target) + applying jar_jar solves our needs pretty well!
The only thing that's missing is the ability to exclude things from the final deployable. I've just opened an issue, requesting an additional parameter be added to scala_library to support that (https://github.com/bazelbuild/rules_scala/issues/560)

[thundergolfer]

@ittaiz, @hmemcpy

If you could publish code/writing on getting feature parity with Maven's shade plugin and Gradle's shadow plugin that would be great. I know there's bazel_jar_jar, but haven't found it straightforward to use, and it seems that it requires having the shaded name in your source code, as in

import com.google.common must become import com.myorg.google.common in source. This is something that isn't necessary with the Gradle plugin. With that plugin the shading happens when creating the fat-jar.

[smparkes]

it seems that it requires having the shaded name in your source code

FWIW, we're starting to use bazel_jar_jar and instead of changing source code, we run bazel_jar_jar on the client jars as well as the implementation jars. This avoids having to change source code at the cost of having to add an additional shading step and factor our dependences a bit more ...

[thundergolfer]

@smparkes Can you expand on this? It sounds interesting but I don't quite know your terminology in this:

on the client jars as well as the implementation jars.

[sid374]

@hmemcpy @smparkes Could you please describe your build process/give some examples of how you got bazel jar_jar working. The documentation isn't very clear for someone who's new to bazel.

The problem I'm trying to solve is this: My project imports 2 external libraries (azure-sdk and cassandra-driver) that depend on different versions of guava (19 and 20). These versions of guava are not backwards compatible. I can not upgrade the cassandra-driver to a newer version, so I'm looking to shade azure-sdk with the version of guava that this requires.

I'm using bazel-deps to download and include the external libraries. Any help/pointers are much appreciated. This is the most active discussion I've found on this topic yet.

[thundergolfer]

Hey @sid374,

I did further work in this area, so can try help out.

Say you want to shade guava inside azure-sdk. First you want to pull the azure-sdk and it's dependencies into your WORKSPACE:

maven_install(
    name = "maven_deps_with_guava_conflicts",
    artifacts = [
        "com.azure-sdk:1.23.0",
    ],
    fetch_sources = True,
    repositories = [
        "https://jcenter.bintray.com/",
    ],

Then you want to gather up the azure-sdk jar and all it's dependencies by using the _deploy.jar implicit target. Like this:

# This is only to generate a fat jar for further shading
java_binary(
    name = "azure-sdk",
    main_class = "none",
    runtime_deps = [
        "@maven_deps_with_guava_conflicts//:com_azure_sdk",
    ],
)

then you shade the fat jar.

shade_rules is a files with something like this -> rule com.google.common.** shaded.guava.@1.

jar_jar(
    name = "shaded-fat-azure-sdk",
    # Fat jar
    input_jar = ":google-cloud-storage_deploy.jar",
    rules = "shade_rules",
)

Now you have a shaded library, :shaded-fat-azure-sdk , that you can put in your deps list.

[thundergolfer]

The above strategy has been used to solve the kind of problem you describe. Let me know if anything is unclear or if you need further details. There really should be a blog post about this...