Gradle: JPMS Support

[lacasseio]

Add support for building, running and testing Java Modules (JPMS - Java Platform Module System) to Gradle.

[adammurdoch]

This epic contains the issues to fix to allow Gradle to run without error on recent Java 9 jigsaw jvm snapshots. These issues are in addition to those in epic #719, which contains the issues to fix to allow Gradle to run the non-jigsaw snapshots.

[lacasseio]

Following the migration from Jira to GitHub, GRADLE-3565 needs to be considered and broken down in the context of this epic.

[reinhapa]

Can a do something on this epic in order to get this forward?

[sdfelts]

FYI Tested 3.1 using Jigsaw b162 with --permit-illegal-access.

The following warning is produced.
WARNING: Illegal access by org.gradle.internal.reflect.JavaMethod (file:gradle-3.1/lib/gradle-base-services-3.1.jar) to method java.lang.ClassLoader.getPackages()

[reinhapa]

It's just a temporary solution, that gives more time to replace the actual wrong code using the proper API calls introduced in JDK 9...

[sdfelts]

I posted the note to make sure that someone knows there is a problem in JavaMethod.

[sdfelts]

Although all packages are opened in the final build 181 of JDK 9, this warning still occurs.
We get thousands of them when we build our product.

[sormuras]

JUnit 5 tracks Illegal Reflective Access warnings here: https://github.com/junit-team/junit5/issues/1063

Most of them originate from the Groovy runtime used in Gradle: "Gradle 4.1 with Groovy groovy-all-2.4.11.jar"

[HoldYourWaffle]

So what's the current status on this? There's a lot of outdated and conflicting information online, and I don't want to start implementing jigsaw in my projects until gradle supports it properly.

[manu-technology]

any chances we have a stable (non-experimental) jigsaw plugin soon?

[HoldYourWaffle]

@eljobe I seem to have somehow unassigned you by commenting. No idea how that happened but I think github just crapped out.

[zyxist]

@noobestein -> I'm maintaining a fork of experimental-jigsaw called chainsaw which properly supports a number of use cases, such as unit tests, javadocs, multimodule projects, annotation processors, and dealing with Jigsaw incompatibilities in thirdparty Java libraries. There are still some quirks, but the plugin is quite usable already and I'm using it in regular development. You can check it out until Gradle gets a native support.

[nikolaybotevb]

Any update on this?

[DigitalSmile]

Let me put another +1 on this. It's java 10 already and this fall will be java 11 LTS. Jigsaw support has to be implemented anyway, guys...

[lucav76]

Any update? Java 11 will soon be out...

[joeha480]

The "workaround" for modules (https://guides.gradle.org/building-java-9-modules/) doesn't work with Eclipse, so I really need proper module support in Gradle. Please fix this as soon as possible!

[Ealrann]

Let me add another +1. The lack of native support of java 9 modules force us to use Gradle in a dirty way.

[marcphilipp]

@Ealrann How so? Have you tried using @zyxist's plugin?

[rgoldberg]

@joeha480 @Ealrann

You can try my fork of the plugin.

I've never tried it with Eclipse, but it works with IntelliJ IDEA.

It supports most of the important features from Chainsaw, plus it automatically handles additional JPMS situations that Chainsaw doesn't. It also avoids various bugs from Chainsaw (most notably, Chainsaw's task dependency bugs that can cause tasks to be run out of order, ruining the build).

The main Chainsaw feature that it lacks is support for manually specifying JPMS command options from Gradle scripts. Maybe I'll add a DSL convention in the future if lots of people need it, and if I can easily determine how to implement an elegant-enough solution with the existing Gradle DSL (presumably mainly in Action block arguments to dependency registration calls, if possible).

FYI: my plugin's name has not been finalized; I need to sort out a domain name for myself…

[DJViking]

The Gradle Modules Plugin already provide good support for building Java modules.

[DJViking]

The Gradle Core(Built-in) Plugin for Java, should have support for Java modules at least with Gradle 5.0 (just released). There should not be a need for installing third-party plugin !!
-- Regards Manu Thekkoot

If you say so. Then this issue could be closed.
I haven't tested it with Gradle 5.0 yet, but if that is the case it is great seeing gradle finally have gotten full support for Java modules.

Edit: I have just tried with Gradle 5.0 and without the Gradle Modules Plugin. It didn't work. It could not find the modules it required. With the plugin compileJava works fine.

[enveeed]

@noobestein could you elaborate further on how to use it? I am using Gradle 5.0 and I can't get it to work.

[swaechter]

@noobestein I am also interested in more information/a real-word Gradle 5 example. At the moment this is keeping me back with Maven (Java 11 + JPMS works great there).

[aleksanderlech]

Hey there, is this support dropped? We are already at Java 13 and there is a big push for modules recently. Is Gradle going to ignore that?

[codedcosmos]

It's 2020. Has there been any progress towards this? Modules don't seem to be going anywhere and there seems to be demand for module support.

[sormuras]

_There's a light at the end of the tunnel..._

https://twitter.com/CedricChampeau/status/1223924890341912577

[...] One of the (many) areas we'll work on is improving encapsulation, which includes, but is not limited to JPMS.

🚀

[Ealrann]

It's really a good news !!

[jjohannes]

Java Module System Support in 6.4

Big news!!! 🎉

Java module support has landed in the latest 6.4 nightly!

./gradlew wrapper --gradle-version 6.4-20200407232121+0000 (updated)

At this stage, we have not documented the feature yet and there are still some quirks and inconveniences we plan to address before the initial release of the feature in 6.4. Nevertheless, it should be fully functional for major use cases:

• Compiling modules that have a module-info.java file with dependencies that are modules (have a module-info.class or a Automatic-Module-Name manifest entry) on the module path rather than the classpath.
• Testing modules, by running the test workers themselves as modules such that the module visibility is respected.
• Executing modules (JavaExec task or project.javaexec {}) using the module path and respect module visibility.

If you have been waiting for this, we would appreciate your feedback! Does this work in real world projects as expected? Is there some basic functionality we overlooked? Are there certain patterns/use cases that could be supported better in the future?

This feedback will help us to ensure a better quality of the initial release and also help us to craft meaningful documentation and samples.

To use the module support, you currently have to opt-in for each task that should be aware of the module path. We plan to add a single place to opt-in for 6.4 to make this easier. In 7.0, this will become the default and no opt-in will be required then anymore.

With the current nightly, do this for all projects (updated):

java {
  modularity.inferModulePath.set(true)
}

Gradle will then automatically detect if a dependency, local project or external, is a module and put it on the module path.

Here is a more complete example also covering some advance test setup:
https://github.com/jjohannes/gradle-jpms-experiments

Feel free to comment on this issue with any kind of feedback! Thank you ❤️ !

[iherasymenko]

Thanks! This is amazing.

Will the option ALL_MODULE_PATH be considered for a future release? Like you did in the spike.
https://github.com/gradle/gradle/pull/12257/files#diff-e5de6cb2768bd128055d804fd5df4e47R43

I'd like to keep all the dependencies on the modulepath even the ones that do not have a module-info.class or the Automatic-Module-Name defined.

[jjohannes]

@iherasymenko thanks for the feedback!

We did not add that option (yet), because we were were missing practical use cases. But we can certainly consider it. I assume the main case is when you want a module to depend on a "library" and add the corresponding requires to the module-info.java. Are there any other reasons why you need it?

Can you share example(s) of existing OSS libraries for which you would use/need that option?

[iherasymenko]

Yes. That's one of the reasons.

In my particular scenario, I wanted to use Bootique, but that is not a really good example, since the library is being actively maintained and I believe that at some point they will either add a module-info.java or at least an Automatic-Module-Name.

A more relevant example would be some OSS library that is in a sustainable state, i.e it was released a while ago, it does its job well (no need in a new version) but yet it does not have a proper module definition.

I'd still like to keep this library on the modulepath and avoid having the classpath altogether in order to make sure:

• There are no split packages
• The strong encapsulation is enforced for this library
• The application benefits from the faster class loading (more details here)

And yes, as you mentioned, I'd also like to have the requires directive just to clearly see what a module depends on and avoid any implicit (i.e. classpath) dependencies within the module.

[jjohannes]

Thanks @iherasymenko! That is very convincing. We should add an option for it then: https://github.com/gradle/gradle/issues/12630.

[JojOatXGME]

Thanks @jjohannes for you work. I have some questions or comments on the topic.

1. Will there be options to override the target for individual dependencies? For example when you encounter that you need some module in the module-path which does not have Automatic-Module-Name or a module-info. @iherasymenko and you already discussed ALL_MODULE_PATH. However, that might cause issues with other dependencies. Such issues might be split packages or invalid module names derived from the file name.

2. Are there plans for APIs which provide more precise control? I was working on a personal Gradle plugin for Java modules some time ago. It would be nice when such plugin could just tell JavaExec and other tasks whether the application shall run as module (--module) and what should go into the module-path. This would especially help with CreateStartScripts which was quite hacky to configue in the past.

3. Are there considerations to to support adding the ModuleMainClass attribute to the module descriptor? This would usually be done by the jar tool. The user would specify the class name with --main-class=CLASSNAME. However, I noticed that Gradle does not use the jar tool of the JDK when it packages the Jar. It might therefore be more work to implement (and is probably not that important).

[jjohannes]

Thanks @JojOatXGME for asking the right questions. :)

1. Will there be options to override the target for individual dependencies? For example when you encounter that you need some module in the module-path which does not have Automatic-Module-Name or a module-info. @iherasymenko and you already discussed ALL_MODULE_PATH. However, that might cause issues with other dependencies. Such issues might be split packages or invalid module names derived from the file name.

That's a very good point and these are also the reasons for the current behavior of splitting module path and classpath. I see that it would be useful to exclude single libraries if you choose the "ALL MODULES" approach. At least temporarily until these libraries are fixed. I'll add a comment about that #12630. Maybe we can do this by having a simple "exclusion" mechanism. More concrete ideas can be discussed on that issue.

1. Are there plans for APIs which provide more precise control? I was working on a personal Gradle plugin for Java modules some time ago. It would be nice when such plugin could just tell JavaExec and other tasks whether the application shall run as module (--module) and what should go into the module-path. This would especially help with CreateStartScripts which was quite hacky to configue in the past.

We are already working on supporting CreateStartScripts with the same approach as used in all other places (#12632). This might be merged today. The question about "precise control" is then the same as in all other "module path aware" tasks (JavaCompile, JavaExec, ...). So this is your first question - any kind of control we provide to split the input to the task into -classpath and --module-path will also apply here.

1. Are there considerations to to support adding the ModuleMainClass attribute to the module descriptor? This would usually be done by the jar tool. The user would specify the class name with --main-class=CLASSNAME. However, I noticed that Gradle does not use the jar tool of the JDK when it packages the Jar. It might therefore be more work to implement (and is probably not that important).

Good news! This already works. We do this in compileJava by writing the attribute to the class file after compilation as that fits the Gradle model of task input/outputs better and we do not use the "jar" tool at the moment as you correctly observed. If you use the application plugin with the new main class _property_, it just works:

applications {
   mainModule.set("my.appmodule")
   mainClass.set("my.App")
}

Or you can configure the option in a compile task directly. E.g:

tasks.compileJava {
  options.javaModuleMainClass.set("my.App")
}

This will all be documented soon. :)

[Vampire]

Is there a nice way to properly support pre-Jigsaw versions?
I have a library for which I support Java 8.
Currently I need to

• compile all "normal" classes with target 8
• either (as the compiler does not like compiling a module info class for packages that are not compiled alongside)
  
  
  
  • compile all classes again and additionally with module info class with target 9
  
  
  • or have some dummy classes compiled together with the module info class with target 9
  
  
• manually pack a multi-version jar with the "normal" classes taken from the Java 8 task and the module info class taken from the Java 9 task

[jjohannes]

@Vampire yes that's a use case we want to support better.

It should be done by splitting the source set in separate _java version_ folders:

src
└── main
    ├── java
    │   └── org
    │       └── java8compatible
    │           └── Java8Class.java
    └── java9
        └── module-info.java

• You can do this today, but we miss some public API. You can find the setup in the example I linked above: https://github.com/jjohannes/gradle-jpms-experiments/blob/master/module-java8/build.gradle.kts
• We want to add a public API to make that setup possible in a less verbose and stable way. We plan to add this very soon (maybe even 6.4): #727
• In the future, we generally want to support compiling, testing etc. for multiple Java versions better. In this context, this could be improved further. This might also include support for building multi-release jars out-of-the-box.

The above setup gives you a "normal" Jar. But Java8 consumers could still use that (they'll just ignore the module-info.class). Currently, if you want a multi-release Jar, you would have to also adjust the Jar task manually.

[Vampire]

Ah, nice, basically the --patch-module is what I was mainly missing for a less quirky solution, thanks.

But Java8 consumers could still use that (they'll just ignore the module-info.class).

That is not necessarily correct.
As far as I remember it didn't work without multi-release jar.
I don't remember now what was the exact problem.
Could be because it is used via CDI and then something scans all class files and cannot read the newer format when running on the old version.
With other libraries that scan files on the classpath there might be similar problems.

[jjohannes]

@Vampire would be interesting to know which setup has an issue with such a jar exactly. Normally, there should not be a problem. I know that JUnit 5 is publishing normal Jars with Java8 byte code + a Java9 module-info.class, for example junit-jupiter-api-5.6.1.jar. And that's a library that is presumably widely used with Java 8.

[sormuras]

[...] JUnit 5 is publishing normal Jars [...]

No, not exactly. Most of them are Java modules. Meaning they are modular JAR files with a compiled module descriptor as a sole entry in their root. One of them, junit-platform-commons is also a multi-release JAR. Another MR-JAR might be introduced with Project Loom, soon. -- "normal" JAR files don't contain module-info.class files.

With other libraries that scan files on the classpath there might be similar problems.

IIRC, in 2017/18 we had single issue with https://github.com/classgraph/classgraph ... all other libraries we touched since then, didnt choke.

On topic: Looking forward to Gradle's Java module system support. Especially curious, how in-module (white-box) testing is solved.

[robstoll]

Androids d8 compiler used to throw up when seeing a module-info.class file. No idea if this is still the case

[redlet]

app-1.0-final.jar
module-a-0.9-beta.jar
module-b-0.9-beta.jar
module-c-0.9-beta.jar
module-java8-0.9-beta.jar

I moved these to the same directory... am I supposed to be able to run app-1.0-final.jar or something? java.lang.ClassNotFoundException: org.modulec.ModuleC happens every time. It works with gradle app:run but I don't get how these are supposed to work outside the IDE.

[jjohannes]

@redlet If you want to run the app "manually" you need to provide the --module-path to the java command.

java --module-path app-1.0-final.jar:module-a-0.9-beta.jar:module-b-0.9-beta.jar:module-c-0.9-beta.jar:module-java8-0.9-beta.jar --module org.app

I updated the sample to the latest nightly. Now packaging the app with Gradle also works. You can run gradle app:build. Then you will get app/build/distributions/app-1.0-final.zip. That contains all libraries and a script to run them with. Extract it to a folder and run it with bin/app or bin/app.bat (on Windows).

[tkslaw]

@redlet

If you want to run the app "manually" you need to provide the --module-path to the java command.

java --module-path app-1.0-final.jar:module-a-0.9-beta.jar:module-b-0.9-beta.jar:module-c-0.9-beta.jar:module-java8-0.9-beta.jar --module org.app

Just to note, the --module-path option is documented as:

A [path-separator] separated list of directories in which each directory is a directory of modules.

Where "directory of modules" appears to be documented by ModuleFinder#of(Path...). In other words, you don't need to add each JAR file to the module-path individually. If you have all the modules in your working directory, then the following command line should be sufficient:

java --module-path . --module org.app

Can be made even shorter by using -p and -m in place of --module-path and --module, respectively.

[tkslaw]

What are the plans, if any, regarding the jlink tool? What about the jpackage tool (also see JEP 343: Packaging Tool (Incubator))? Assuming, of course, that support for those tools is covered by this issue.

[redlet]

Ah okay, the script works properly. Is it possible to have a "Java 8 JAR" load a module-based "JPMS JAR"? I'm optimistic, but this is such a major departure.

Edit: And how do Java 8 dependencies work? If you can't shade, then how do I depend on Java 8 resources? Most libraries do not support JPMS.

[jjohannes]

@tkslaw thanks for the additional infos on the java command line args.

@redlet

Ah okay, the script works properly. Is it possible to have a "Java 8 JAR" load a module-based "JPMS JAR"? I'm optimistic, but this is such a major departure.
Edit: And how do Java 8 dependencies work? If you can't shade, then how do I depend on Java 8 resources? Most libraries do not support JPMS.

There are different setups and most things do not depend on Gradle but on how java itself works with modules:

• You can put any module on the --classpath and then they work like any traditional jar (the module-info is ignored). This is what Gradle is doing if you do not turn on inferModulePath.
• If you work with modules and you want to depend on a traditional library it may have an Automatic-Module-Name configured. Then you can depend on it by that name. Such an _automatic module_ exports all its packages and sees all other modules on the module path. Many OSS libraries nowadays have the Automatic-Module-Name entry in their latest version.
  Libraries without that entry, which have never been considered to be used as modules, can be more of an issue (see discussions further up). With the current implementation, Gradle will put these libraries on the -classpath even though everything else is treated as module. In that case, the classpath is treated as one module (the _unnamed module_) by Java which is also seen by _automatic modules_. (See this automatic module in the sample).

[jjohannes]

@tkslaw

What are the plans, if any, regarding the jlink tool? What about the jpackage tool (also see JEP 343: Packaging Tool (Incubator))?

We want to support _jlink_ and _jpackage_ at some point in the future, but it is not considered part of this issue, in the sense that it is not planned for the "basic JPMS support" in 6.4.

[tkslaw]

We want to support jlink and jpackage at some point in the future, but it is not considered part of this issue, in the sense that it is not planned for the "basic JPMS support" in 6.4.

Glad support for those tools is planned, even if not as part of this issue. :thumbsup:

---

Wondering if there will be a way to easily configure (any or all):

• --limit-modules
• --add-modules
• --upgrade-module-path
• --add-reads
• --add-exports
• --add-opens
• --patch-module

Perhaps through extension properties, similar to how some current third-party plugins do it (e.g. gradle-modules-plugin)?

[DJViking]

@tkslaw

What are the plans, if any, regarding the jlink tool? What about the jpackage tool (also see JEP 343: Packaging Tool (Incubator))?

We want to support _jlink_ and _jpackage_ at some point in the future, but it is not considered part of this issue, in the sense that it is not planned for the "basic JPMS support" in 6.4.

There is always the badass-jlink-plugin that can be used for run jlink and jpackage.

[redlet]

I'm reading https://openjdk.java.net/jeps/343 (jpackage) and it seems like it does exactly what Gradle 6.4 does when you run gradle app:build (or, perhaps, just gradle build). Is jpackage supposed to create a fat jar or leave the components scattered about?

Edit: I believe jlink makes fat jars actually. Quite frankly, I am looking to bundle dependencies into a single jar since it would be unwieldy to have every dependency in its own file.

[lenborje]

I'm reading https://openjdk.java.net/jeps/343 (jpackage) and it seems like it does exactly what Gradle 6.4 does when you run gradle app:build (or, perhaps, just gradle build). Is jpackage supposed to create a fat jar or leave the components scattered about?

jpackage builds one or two things:

1. A binary launcher, native to the current platform, bundled with all the classes, resources, and a stripped-down JVM, making the resulting application independent of any system-installed JVM.
2. An installer package, e.g. an MSI, RPM, PKG, or whatever is appropriate on the current platform.

Edit: I believe jlink makes fat jars actually. Quite frankly, I am looking to bundle dependencies into a single jar since it would be unwieldy to have every dependency in its own file.

No. jlink makes a stripped-down JVM. It does not bundle any user-specified classes in this JVM. You’re supposed to use it to reduce the footprint of an application bundle, e.g. for jpackage, or for containers, e.g. Docker.

[tkslaw]

@lenborje

No. jlink makes a stripped-down JVM. It does not bundle any user-specified classes in this JVM.

That's not entirely accurate. It will bundle user-specified modules into the custom image. The jlink tool is meant to create an entirely self-contained, minimal runtime image; that includes the application module and any library modules.

@redlet

A fat/uber jar only contains all the project's code, it still requires the end user to have Java installed. But jlink can take any number modules and essentially create a (notably platform-specific) custom JRE. You can then deploy that JRE and users won't have to worry about installing the correct version of Java. Another benefit specific to jlink is, if your code uses native code, you can package the code in jmod files (used only for linking) and the native code is automatically put on the library path (without having to package it in the jar and deal with extracting it at runtime).

jpackage goes a step further and creates platform-specific executables/installers. This makes the Java application look like any other native application.

[rgoldberg]

Does the new JPMS support allow third-party tasks whose underlying utility supports modules (like KotlinCompile) to easily switch from the class path to the module path, and to easily use other module-related options?

I haven't had the time to look through the new support or documentation thereof, so sorry if this has already been answered / should be immediately apparent to people who have followed this.

[redlet]

So if I do not want an installer or a script to run my jar _and_ I have 30 "automatic module" dependencies (Java 8 libraries), I need to have 31 jar files to run my program? It seems foolish, really. The self-contained fat jar was always an easy way to ensure you had everything necessary with correct versions.

Edit: And transitive dependencies; just forget about it. You're going to be drowning in files.

[tkslaw]

@redlet

JPMS does not prevent fat/uber jars. Simply don't use modules and put everything on the classpath. Assuming the code doesn't try to break into JDK classes (e.g. ClassLoader#defineClass(...)) then the fat/uber jar should work just like it did in Java 8. Even then, adding any necessary --add-opens is a workaround; you can even configure this in your fat/uber jar's manifest file—see the "Packaging: Modular JAR files" section of JEP 261 (note the JEP mentions Add-Opens and Add-Exports are JDK-specific).

That all being said, I'm not sure if this issue is the correct place to continue discussing this (unless the problem is specific to JPMS support in Gradle).

[cilki]

Will Gradle also populate the module versions according to project.version? That was a minor annoyance I had with the JPMS plugins.

@redlet It's not ready for production yet, but jlink.online might be able to help you one day.

[redlet]

@cilki hah that's a pretty unfortunate solution to a problem that should have never developed. at worst, I think a single repository with dependencies inside could be a good solution and multiple components draw from the repository pool, but managing one centralized pool among multiple gradle projects sounds like a chore.

[jjohannes]

@cilki

Will Gradle also populate the module versions according to project.version? That was a minor annoyance I had with the JPMS plugins.

Yes it does. As soon as you set a version on the project, that version is written to the module-info.class during compilation.

You can fine tune this in a compile task by setting options.javaModuleVersion.

[jjohannes]

Thanks everyone for the interest in the topic and the valuable feedback!

We are closing in to the 6.4 release and are close to RC1. The Java Module support in 6.4 is feature complete. Please give it a spin if you like. If you find anything odd, please report.

Before RC1 is out, the latest 6.4 release nightly can already be found here:
https://gradle.org/release-nightly/

There is documentation on the topic now (it's a bit scattered due to the current structure of the Java plugins docs in Gradle's user manual - improving that structure is on our list):

• Building Java Modules
• Building modular Java Applications
• Testing Java Modules
• Samples

Regarding additional features beyond 6.4: I know that there are areas which we might want to support better in the future (e.g. whitebox testing with patching). But for that we want to find the right abstractions first. For that we need real world use case that go beyond _"there is this or that compile flag and maybe I want to set it"_ (if you get the idea).
So if you have an addition feature request beyond what is provided in 6.4, feel free to open a separate issue. Please be as concrete as you can in terms of which use case(s) you need to solve and why what Gradle 6.4 provides is not sufficient.
I also believe that what we have now is a good base to build some light-weight plugins for conveniences on top - like in this sample for instance. This could be a good start to experiment with additions we might consider to move into Gradle core in the future.

[redlet]

@jjohannes is the Samples page experiencing issues? it's appearing very strange for me on mobile and desktop.

edit: rendering is botched, but I can still see the content. "extraJavaModuleInfo" feature is amazing.

[jjohannes]

@redlet Sorry we are experiencing some issue with the style sheet for the samples page. Hopefully fixed soon. You can still find the Java Modules section. There are three samples. If you click one of them, search for Groovy DSL or Kotlin DSL for a download of the full sample project.

All other docs I linked above should be good.

[Ealrann]

@jjohannes I probably missed something, but it looks like resources (in src/main/resources) are not visible, even when the module is open.

I read them with:

module.getResourceAsStream(path)

Is there a specific configuration to make resources visibles ?

[mwkroening]

@Ealrann, https://github.com/gradle/gradle/issues/6828 has been resolved, so it should work.

See the issue reproduction repository: mwkroening/modular-gradle-resources

[redlet]

is it possible to put runtime arguments within a jar? i'm trying to load a modular jar dynamically (which has other dependencies sitting around via the buildSrc plugin), but I'm struggling because you can't simply do java -jar mj.jar and exclude the module path. it seems like it must be run from the bash script.

[jjohannes]

@Ealrann in which context do you see the problem with module.getResourceAsStream(path)? Always when the Jar is used to run the module, it should work now. It might be an issue if you "run the classes folder". But I am not sure when this would happen.
Do you see this during test running maybe? Can you provide a reproducer? We might have missed something there.

[sormuras]

[...] Is there a specific configuration to make resources visibles ?

Are the resources located in exported packages of your module?

See also the API documention for various resource loading methods:

• ClassLoader.findResource
• Class.getResourceAsStream
• Module.getResourceAsStream

[Ealrann]

@jjohannes
Here a minimal test project :
https://github.com/Ealrann/Gradle6.4_testResources

I would have expected a gradle run works here.

@mwkroening Surprisingly, using a ResourceBundle works for retrieving some properties, but not the getResourceAsStream().

@sormuras No need to export package to read resources, you should only open them.

[tkslaw]

@Ealrann If the run task of the application plugin doesn't automatically patch the resources into the module, then this is the same problem as described in https://stackoverflow.com/a/51921521/6395627

Gradle puts compiled classes in build/classes and processed resources in build/resources. Since the run task does not execute against a packaged JAR file (i.e. executes against build/classes/...) the resources are not considered part of the module unless patched in with --patch-module.

Also, keep in mind that opens is only necessary if you want to make the resource available to _other modules_. A module always has access to all its own resources.

[Ealrann]

@tkslaw Thank you for the link. I thought it was on the scope of this issue to patch the resources into the module.

[jjohannes]

Thanks @Ealrann. So we actually do not execute the JAR with the run task, but the classes. I forgot about that detail myself. So it's clear why this is not working. We need to fix this.

I guess the solution would be to either run the Jar or patch the _resources_ folder in. I am leaning towards running the Jar, because there are potentially even more folders you might add as Gradle's source set concept allows for that. I can't think of a good reason why we should not built the complete Jar and run that.

[tkslaw]

@jjohannes

because there are potentially even more folders you might add as Gradle's source set concept allows for that.

That's a good point. Running against the JAR will also prevent opens directives for resource-only packages from throwing errors at runtime, due to the package not existing (until after patching). At compile-time, the same opens directives will only cause a warning to be emitted.

[Vampire]

Actually this could be even driven further.
At work we build a tool with several additional resources in the distribution, that are not in the jar.
Theses resources should be easily editable by the user (a similar use case is logging configuration file). Our tool does not work without those tools in the proper places. So what I did is, I removed the standard run task and replaced it by a dependency to installDist and then run the application from there, as this is basically identical to how the user will run it in the end too.

So you might even consider this strategy.

Btw. either, just running the jar or running the installed dist, should also fix problems with CDI, where the current strategy does not work properly due to this separate class path entries.

[jjohannes]

Thanks for the feedback everyone. Just merged #12769. The :run task now uses the Jar when running a module (while keeping the old behavior for traditional applications).

This also makes me realize that you probably want to do the same in a blackbox test if the test itself has resources that should be part of the test's module. I updated the corresponding sample to reflect this.

This will all be in the next nightly.

[Ealrann]

@jjohannes Thank you very much!

[Ealrann]

@jjohannes Maybe I found another issue with resources, when running a blackbox test:

Let a method runModuleResource() in src/main/java read some resources with module.getResourceAsStream(path).

• Calling this method from a gradle run is working.
• But, calling this method from a @Test with gradle test is leading to a null input stream.

I'm using Gradle 6.4-20200410071617+0000.
I made another test project here:
https://github.com/Ealrann/gradleJPMS_testResources.git

[jjohannes]

@Ealrann yes that is expected. :) In the sample, I set up a separate test task/source set for blackbox testing and set up the classpath such that it uses the jar of the project.

The test task does it differently by default, but you can adjust it if you want to use it for backbox testing:

test {
   // redefine classpath to not include the main classes folder
  classpath = configurations[sourceSets.test.runtimeClasspathConfigurationName] + files(compileTestJava)
}

dependencies {
  // depend on my own 'main' variant for testing
  testImplementation project(path)
}

I know it would be more convenient if it would just work or if the default would be different. We plan to add some more conveniences to configure testing more easily soon. It should also be simpler to add an additional test set for instance. In that context, we will also consider this use case.

[msgilligan]

I'm happy to see progress being made on Module support and was excited to test this plugin out.

Unfortunately it's not very useful to me at this time for two reasons:

1. Many of my modular projects (subprojects) are using JavaFX via the JavaFX Gradle Plugin which directly depends on the Gradle Modules Plugin (In fact it only works with version 1.5.0 of that plugin and not with version 1.6.0)

2. I have very few projects that aren't using at least one non-modular JAR. The extra-java-module-info plugin should solve those problems, but I had to add an ugly hack to get it to build my Gradle multi-project (with one module converted to modular) and when I pushed that branch to CI both my GitLab and Travis jobs failed. So until there is a supported, binary, (somewhat) documented plugin with extra-java-module-info functionality, I don't think I have a single modular subproject I could use if for (without diving in to plugin development, which I really don't have time for right now.)

I understand that the point of this release is to get the model correct, and I think that is what is most needed (so I can get the JavaFX plugin, a modules-related plugin or two, and hopefully IntelliJ see IDEA-182680 to all play together nicely) but until there is at least a beta-quality version of extra-java-module-info (or equivalent) in the Plugin portal, the current module support isn't useful to me.

Thanks again for this effort, and I'm looking forward to seeing something like extra-java-module-info` from either Gradle Corp or the community!

(p.s. I'm willing to help test any such plugins and am willing to submit issues and maybe even a patch or two if necessary.)

(p.p.s If you're curious about my failing Travis build, it's here:
https://travis-ci.org/github/ConsensusJ/consensusj/builds/677524184)

[elect86]

What about libraries which don't have a main class?

Shall we fill a dummy one just to make it work?

[jjohannes]

@msgilligan Thanks for trying the release candidate and the detailed feedback. We won't add more features to 6.4, but we might do something for the next release. I linked your comment in issue #12630 which is about adding better support for the "legacy library" case in Gradle itself. (Offering a plugin on the plugin portal could also be an option.)

Regarding the _consensusj_ build: I had a look at why the transform is failing. The root cause is that configurations are resolved during project configuration, which should be avoided in general. But Gradle should throw a better error. We have an issue about this (#10484), which I will update accordingly.

To fix your issue in the _consensusj_ build, you should replace both occurrences of

'-cp', sourceSets.main.runtimeClasspath.asPath,

with a lazy string evaluation:

'-cp', "${-> sourceSets.main.runtimeClasspath.asPath}",

[jjohannes]

@elect86 I am not sure what you mean by "make it work". You don't need a main class to compile a module. It's optional.

[msgilligan]

@jjohannes Thanks for the response!

We won't add more features to 6.4, but we might do something for the next release.

Of course. I'm just sending in feedback for the next release. 🤞 Or to encourage you to move forward with extra-java-module-info as an open source plugin, etc.

To fix your issue in the _consensusj_ build, you should replace both occurrences of

'-cp', sourceSets.main.runtimeClasspath.asPath,

with a lazy string evaluation:

'-cp', "${-> sourceSets.main.runtimeClasspath.asPath}",

Thanks for this tip. Done: https://github.com/ConsensusJ/consensusj/commit/154aa180c59a34ac035d506e95139207be6b60f7
This did fix my build issue (see build #457 ), but I still need my one-line hack. I have only one problematic legacy library (commons-cli-1.4.jar) in this subproject. If I remove the hack the build fails (Not a module and no mapping defined: commons-cli-1.4.jar), see build #458.

[msgilligan]

@jjohannes I also discovered that upgrading from:

testRuntimeOnly "net.bytebuddy:byte-buddy:1.9.1"

to:

testRuntimeOnly "net.bytebuddy:byte-buddy:1.9.2"

breaks the build because byte-buddy 1.9.2 and later is a multi-release JAR.

It would also be nice if test dependencies need not be module-compatible unless you are running tests in the modular environment.

[elect86]

Sorry my bad, I was looking at the application plugin doc and I got confused

[jjohannes]

@msgilligan you basically discovered bugs/weaknesses in the sample. I fixed them and improved the syntax a bit. I now published the plugin so that it is easier for everyone to try it out:

plugins {
  id("de.jjohannes.extra-java-module-info") version "0.1"
}

Syntax: https://github.com/jjohannes/extra-java-module-info#how-to-use-this-plugin

Both problems you found are fixed. In your build, this works now (in _consensusj-jsonrpc-cli/build.gradle_):

plugins {
  // ...
  id "de.jjohannes.extra-java-module-info" version "0.1"
}
extraJavaModuleInfo {
    automaticModule("commons-cli-1.4.jar", "org.apache.commons.cli")
}

[msgilligan]

@jjohannes Thanks so much!

It's now using your published plugin and Gradle 6.4-rc-2 and it's a nice clean diff. The TravisCI build is now working for both jdk8 and jdk9. (Now that I'm only including your plugin in the conditionally-included subproject rather than the root, the jdk8 build is no longer broken.)

[msgilligan]

Now I'm thinking about the next steps on my Java modularization quest. I'm going to share my thoughts because I assume my issues are typical of those other Open Source Java library maintainers will face.

I'm building JavaFX applications and (JavaFX and non-JavaFX) libraries as well as sending upstream patches (which are generally being accepted!) to make my upstream dependencies automatic modules. (For example, I sent https://github.com/apache/commons-cli/pull/35 to Apache Commons CLI when I first starting working with Gradle 6.4 module support, but that one is still pending.)

The next steps for me that relate to Gradle tooling are:

1. Convert more subprojects/JARs that are currently automatic modules into full modules. The problem is that many of these must be usable from JDK 8. So they'll either need to be Multirelease JARs or use the Gradle variant mechanism (which were released in Gradle 6.0, right?) to provide continued support for JDK 8 and Android.

2. Start writing unit tests that are both both white-box (non-modular) and black-box (modular) so I can be sure both use cases are working.

(1) seems somewhat separate from the current Gradle 6.4 module support, but it might be nice to see some multiproject samples that use Gradle 6.4, the extra-java-module-info plugin, and either MR Jars or variants with subproject dependency relations. I'm assuming Gradle already provides usable support for MR Jars and/or variants.

For (2), I'm not sure yet what is needed, because I haven't read the current documentation thoroughly nor have I tried to do anything new with testing at this point.

[msgilligan]

On the JavaFX front, my focus is on building natively-packaged applications using the jpackage tool in OpenJDK 14. This is working well, due to 3 Gradle plugins that seem to interoperate effectively:

1. Gradle Modules Plugin
2. JavaFX Gradle Plugin
3. Badass JLink Plugin -- which runs jlink and jpackage and also provides a mechanism for converting automatic and non-modular JARs to modules at "link" time.

At some point it would be nice to see Gradle effectively replace (1) by getting (2) and (3) to interoperate with Gradle 6.4+ and the extra-java-module-info plugin.

@siordache author of the Badass JLink Plugin has been amazingly responsive and supportive -- and I just noticed he's already released Badass JLink Plugin v2.17.7 to support Gradle 6.4.

And if anyone has contacts at IntelliJ, please encourage them to fix IDEA-182680! 🙏🙏🙏

[jjohannes]

Thanks again for the feedback @msgilligan. Some pointers for you:

• There is currently no special support for building MR jars in Gradle (consuming works). So you need some manual setup. I did a sample here how to add an extra java9 folder (improving this is planned #727). I extended it to show how to build it as multi-release jar. Just putting the module-info.class into a 'normal' Jar should also work (Java8 consumers will just ignore it; it's the same pattern JUnit5 uses for example).
• This sample here is a multi-project with unit tests and blackbox tests.

[msgilligan]

@jjohannes I've been pulling my hair out for the last two days on a Gradle 6.3 & Java Module System issue that I think is relevant to the overall issue of Java Module support in Gradle.

I have a Gradle multi project build with a Java 11 module (wallettemplate) built using the Java Modularity Plugin depending on a Java 7-based Automatic Module (bitcoinj-core) that was recently upgraded from the java plugin to the java-library plugin. The problem is, of course, that now the dependency is on the output "class folder" not the output jar.

I created an issue in the Gradle Modules Plugin: https://github.com/java9-modularity/gradle-modules-plugin/issues/143, but I think there is (at least) a documentation issue in Gradle.

I've spent the last couple of hours reading through Gradle docs (and have learned about all the great work you guys have done in recent years) but can't figure out how to force the dependency to use the jar artifact of the bitcoinj-core subproject.

I thought would bring it up here, but am happy to create a separate issue for it, if appropriate.

Update: The problem still occurs with Gradle 6.4-rc-2 and IMO should be mentioned in this section of the documentation: https://docs.gradle.org/6.4-rc-2/userguide/java_library_plugin.html#sec:java_library_modular_auto

[msgilligan]

… Gradle multi project build with a[n] … Automatic Module … the problem is… the dependency is on the output "class folder" not the output jar.

@jjohannes Any ideas? Should I just create an issue for this one?

[jjohannes]

@msgilligan sorry for the late reply.

You can enforce the use of jar instead of classes for compilation in any project by this:

configurations.compileClasspath.attributes {
   attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, named(LibraryElements, LibraryElements.JAR))
}

Could you point me at how to reproduce the problem with the Module support of Gradle 6.4?
There, if a project is a module (module support turned on and module-info.java present) Gradle automatically performs the above setting.

[msgilligan]

@msgilligan sorry for the late reply.

No problem. I'm sure you've been busy!

You can enforce the use of jar instead of classes for compilation in any project by this:

configurations.compileClasspath.attributes {
   attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, named(LibraryElements, LibraryElements.JAR))
}

I assume you mean for me to do this in the dependent subproject, (wallettemplate) not the dependency (bitcoinj-core), right?

Could you point me at how to reproduce the problem with the Module support of Gradle 6.4?
There, if a project is a module (module support turned on and module-info.java present) Gradle automatically performs the above setting.

To be more clear, when I see the problem with Gradle 6.4, Gradle does not see either project as a module. The dependency subproject (bitcoinj-core) is an automatic module (and as far as I know Gradle has no awareness of this unless it decides to read the JAR manifest attributes) and the dependent subproject (wallettemplate) is using the Gradle Modules Plugin. I didn't expect that trying it with Gradle 6.4 would help, but the Modularity plugin does a lot of magic (it's magic to me, at least) so I thought I would give it a try and report the results.

[jjohannes]

I assume you mean for me to do this in the dependent subproject, (wallettemplate) not the dependency (bitcoinj-core), right?

Yes. By tweaking attributes on the compileClasspath configuration, you define what that project needs for compilation from the projects it depends on (in that case, you say: "I need the jars").

The dependency subproject (bitcoinj-core) is an automatic module (and as far as I know Gradle has no awareness of this unless it decides to read the JAR manifest attributes)

Gradle 6.4 is aware of automatic modules, but you need to have the JAR for that, yes.

Looks like you already figured all this out and it got fixed in the modularity plugin now as well: https://github.com/java9-modularity/gradle-modules-plugin/issues/143. Thanks!

[albsch]

Thanks to the JPMS support by Gradle and @jjohannes plugin (extra Java module info) I was able to fully transition to JPMS for one of my main projects (including being able to use jlink!). It hugely simplified my Gradle setup, I'm basically not using any custom gradle tasks anymore. The only thing missing is that either internal dependencies move to JPMS, too (unlikely in a timely manner), or Gradle is somehow able to generate the missing module-info.java files for me, that I'm currently doing myself with the help of that module info plugin.
Great work! I know it's hard, but I think it can pay off to transition to JPMS and hope more library developers are able to do this. I'm grateful that Gradle has good support of that now.

[RazeSoldier]

Hi, I have a question. The following code block is a JPMS solution I saw from the document before.

compileJava {
    inputs.property("moduleName", moduleName)
    doFirst {
        options.compilerArgs = [
                '--module-path', classpath.asPath,
        ]
        classpath = files()
    }
}

Is there any difference between the above solution and this solution?

java {
    modularity.inferModulePath = true
}

The old solution still works well under my current gradle version (6.5.1). But if I want to rely on certain plain libraries, I need to use plugin to handle these issues.

[JojOatXGME]

@RazeSoldier Your first solution only affacts the task compileJava which compiles your production code. I think the second solution also enables the JPMS for other tasks like javaDoc, jar, run, startScripts and maybe more.

[ljacomet]

@RazeSoldier One fundamental difference between the two solutions is that inferModulePath will only put JARs that are identified as modules on the module-path, that is JARs with a module-info or an Automatic-Module-Name in their manifest. The other solutions simply moves the full classpath to the module path.

[stefan-steinegger]

I don't know where to put this, so please forgive me if this was the wrong place. We are trying to get a more or less simple Gradle project running with modules. Here is our experience.

The project setup:

• A few own real JPMS modules
• Java 11, Gradle 6.6.1
• It uses libraries like slf4j, Freemarker and Guice from the official repository (old-school jars that do not even have a automatic module name)
• It uses the Java-Library and Java-Application plugin
• It uses unit tests

Nothing special so far. To me, it looks like most of the new projects out there probably look like (or _would_ look like if it _would_ work). Yes, it uses legacy jars. I guess, everyone has to.

• Compiling with Plain Gradle 6.6.1: Even with inferModulePath, the legacy jars are not added to the module path and the compiler complains when trying to "require" one of those.
• A proposal to make it configurable is still being discussed.
• An often proposed workaround to set the module path to the classpath doesn't work either. It compiles, but at runtime, none of the modules are loaded because they are not in the module path. The generated build script doesn't use the module path at all. Anyway, it's ugly. Because Gradle already deals with the module path, it shouldn't be used anymore IMHO.,
• The modularity plugin is older than Gradles "native module support", but still required to do things Gradle still doesn't do. We successfully used it until upgrading to Gradle 6.6.1. Unfortunately it rewrites the start script. That's a pity, because the start script changed in Gradle 6.6.1 and now it's broken. Of course, I should issue a bug at their repo (and I probably will). However, I don't understand why this plugin is still required. It does much more than necessary and starts to conflict with latest efforts in Gradle to support modules.
• I didn't find a way to make modularity stop changing the start script.
• I didn't try the great extra modules info plugin. It looks really great, but feels like taking a sledgehammer to crack a nut. I guess if it will cause problems with signed jars. I would like to use the jars as I got it for several reasons. And it doesn't look very "official", so it's likely to break with future Gradle versions, as modularity did.

So it seems that it is not possible to compile and run this kind of (very common?) project with Gradle 6.6.1 today. If there is a way or workaround, it's not documented or maybe I was too stupid to find.

I believe there is not much missing in Gradle. It's there by 99%. The remaining percent drives me crazy.

[JojOatXGME]

I guess the official JPMS support currently only works with artifact transforms (as used by the extra-java-module-info plugin). See https://github.com/gradle/gradle/issues/12630#issuecomment-610798972.

I don't have experiance with signed jars. If you want to use signed jars, I guess you could sign the modified Jars with your onw key, just like the Jars of your own project.

[Burtan]

Hi,
I'm using gradle 6.6.1 and the extra-java-module-info plugin to add non-automatic modules like simple-xml.

extraJavaModuleInfo {
    automaticModule("simple-xml-2.7.1.jar", "simple.xml")
}

However, I get the following error, any ideas how to solve this?

module simple.xml reads package javax.xml from both stax.api and java.xml

Thanks!

[iherasymenko]

@Burtan If I were to guess, I would say you have the split package problem here. Try excluding the "stax.api" jar, since "java.xml" is a standard Java module bundled with JDK which already contains the classes of this jar (if my guess is correct).

[martinmeeser]

@stefan-steinegger Your description is very precise. Thanks for that.

I wanted to use "implementation group: 'org.xerial', name: 'sqlite-jdbc', version: '3.32.3.2'" - not so uncommon I presume - but took me lots of googling to make it work, ~ 1 day.

It is compiling and running now with "extra-java-module-info plugin".

1. A plugin should really not be necessary.
2. I think everything should go to the module path by default

In case of sqlite that would have worked fine (used it this way before in a Apache Netbeans ant build).