Picocli: Add module picocli-jpms (was: Include module-info.class for full JPMS modularization and jlink support)

[remkop]

Currently the picocli jar is an automatic module (manifest has Automatic-Module-Name: info.picocli).

A JLink trimmed binary image needs explicit modules. This means the jar needs to include a module-info.class.

Update: this is a common misunderstanding. See the comment below for a tutorial for creating a JLink binary image with an automatic module.

module-info.class trade-offs

It is possible to compile only the module-info.java file with Java 9 and include it in the picocli jar where all other classes are compiled with Java 5.

However, there are quite a few tools that scan all classes in a Jar and that would choke on the module-info.class (whether it is in the root of the jar or in META-INF/versions/9/). For example: jandex, and Apache Commons BCEL.

Consider producing a picocli-jpms-3.x.x.jar that has the same contents as picocli-3.x.x.jar but in addition a module-info.class. Users looking to create a JLink custom runtime image can use the picocli-jpms jar.

Solution

The comment below is a brief tutorial for creating a JLink binary image with an automatic module. No module-info.class required in picocli or in the application.

Todo: do a more formal write-up and add it to the picocli documentation.

[remkop]

It turns out that an application does not need to be a full-fledged JPMS module with a module-info.java file to generate a custom jlink runtime image for your application.

Example application:

package hello;

import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;

@Command(name = "hello", description = "picocli demo", version = "3.6.1", mixinStandardHelpOptions = true)
public class HelloWorld implements Runnable{

    @Option(names = {"-u", "--user"}, description = "Specify a user. Default is ${DEFAULT-VALUE}.")
    String user = "unknown";

    @Override
    public void run() {
        System.out.printf("Hello, %s", user);
    }

    public static void main(String[] args) {
        CommandLine.run(new HelloWorld(), args);
    }
}

Example build.gradle:

group 'experiments'
version '1.0-SNAPSHOT'

sourceCompatibility = 1.8

repositories {
    jcenter()
}

dependencies {
    compile 'info.picocli:picocli:3.6.1'
    testCompile 'junit:junit:4.12'
}

jar {
    manifest {
        attributes 'Main-Class': "hello.HelloWorld",
                'Class-Path': 'lib/picocli-3.6.1.jar'
    }
}

Build the application:

$ gradlew clean build

If you try to run it now as an executable jar it complains because it is still missing picocli on the classpath:

$ java -jar build\libs\experiments-1.0-SNAPSHOT.jar -h
Exception in thread "main" java.lang.NoClassDefFoundError: picocli/CommandLine
        at hello.HelloWorld.main(HelloWorld.java:19)
Caused by: java.lang.ClassNotFoundException: picocli.CommandLine
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 1 more

Copy the picocli jar into build/libs/lib:

$ mkdir build/libs/lib
$ cp path/to/picocli-3.6.1.jar build/libs/lib/.

Now you can run the jar:

$ java -jar build\libs\experiments-1.0-SNAPSHOT.jar
Hello, unknown

Get the list of dependencies from jdeps:

$ jdeps --list-deps build\libs\experiments-1.0-SNAPSHOT.jar
   java.base

Our HelloWorld app only requires java.base. Let's create a runtime image for that module:

$ jlink --no-header-files --no-man-pages --add-modules java.base --output java-runtime

You now have a ./java-runtime subdirectory with a custom JRE runtime image with all that is necessary to run the application:

$ java-runtime\bin\java -jar build\libs\experiments-1.0-SNAPSHOT.jar
Hello, unknown

And we're done!

You can copy the contents of the build/libs directory to java-runtime/lib so that everything is in one place for easier distribution.
That allows you to run the application like this:

$ java-runtime\bin\java -jar java-runtime\lib\experiments-1.0-SNAPSHOT.jar
Hello, unknown

[jennettwheeler]

I'm having this issue too, but I'm using maven...

[remkop]

All that’s really needed is a jar with the HelloWorld class where the manifest has HelloWorld as the main class so that the application can be run with java -jar myjar.jar.

It would be great if you could provide a minimal maven Pom that accomplishments that, so I can include it in the documentation.

[jennettwheeler]

It's just that I can't run jLink while my module-info.java depends on picocli?

Oh wait, you mean that for now:

• Work out "JPMS supported" dependencies
• Work out non-JPMS dependencies
• Create a JRE using just the "JPMS supported" dependencies without the jar file
• Copy the jar & the non-JPMS dependencies (within class path)

[jennettwheeler]

I think can work with that, when I get to it, but I'm definitely looking forward to the "picocli-jpms" module.

I had a go at modularising it, but was having issues with groovy being found (I think it's because I'm using JDK 11). So I'm guessing it will not be an easy task to convert.

[remkop]

I got my inspiration from Simon Ritter: https://medium.com/azulsystems/using-jlink-to-build-java-runtimes-for-non-modular-applications-9568c5e70ef4

I'm considering separating the groovy stuff into a separate module (#479) but that will likely be a picocli 4.0 thing. There are other things I want to work on first. Especially since there is a workaround as described in the above comment and Simon't article.

[jennettwheeler]

Thank you. i look forward to it. For now I am happy with the workaround.

[jennettwheeler]

Besides it's better to use Picocli than not :P haha

[jennettwheeler]

For now, I've added a new module to my project called picocli with ComandLine.java & AutoComplete.java (plus module-info.java) and named the version 3.7.0-JPMS. This is letting my compile and create my JRE!

The downside is, of course, that I have to manually download changes to your source code.

[remkop]

Did following the steps in the comment (or Simon’s article) not work? You shouldn’t need a module-info.class...

[jennettwheeler]

My application is a JPMS app, so it does not work for me. My solution means that I have Linux & windows zips being created by just running mvn package.

I have made use of service loader (I don't know if this is only java 9+?) to essentially enable installing sub commands prior to run-time...

I have a folder called "jobs", in there maven puts my default jobs (sub commands for picocli) - which are excluded from jlink as they aren't depended upon at any point.

I also have a second git repo that proves to me that building the jar and dropping it into the jobs folder immediately enables the sub-command in picocli and lets me run that jar's code.

[remkop]

In picocli 4.0 I’m planning to do the following:

• Add a picocli-core-module artifact that has only the classes in the picocli package. This will be a Java 9 modular jar (not an automatic module). TBD whether to exclude the AutoComplete class and whether to compile all classes with Java 9 or just the JPMS module-related ones (module-info).
• Add a picocli-groovy artifact that has only the Groovy classes. This artifact depends on picocli-core-module.
• Also, continue to provide a picocli artifact with the same content as the current artifact (the picocli and the picocli.groovy package). All classes in this jar are compiled with Java 5, just like now. This is an automatic module just like it is now. The module name for this automatic module will be the same as that of the picocli-core-module artifact.

[jennettwheeler]

Sounds great.

[jennettwheeler]

Is this still planned for 4.0?

[remkop]

Hi, yes, this is part of the plan for 4.0.
(However, there’s a lot on that list: https://github.com/remkop/picocli/milestone/40)

I haven’t really given much thought on how to implement this. It’ll probably require some Gradle magic.

Will you be able to help out with this one?

[jennettwheeler]

I would be more than willing to but I struggled the last time I tried. I have absolutely zero experience with gradle (I use maven) and was struggling to make it work.

Do you have any pointers.

Are you planning on 4.0 being a breaking change (i.e. no longer supporting Java 5?)

[jennettwheeler]

I did have a quick go, but struggled to modify. I'll have a proper sit down soon to work out exactly how it currently works, because only then can I make adequate decisions.

It has reaffirmed my personal preference of maven haha. Each to their own eh?

[Warkdev]

Hi guys,
Please let me know if you need further assistance for this, I'm also looking forward a picocli JPMS-9 compliant library :)

Warkdev

[remkop]

@Warkdev that would be great!
I jotted down some thoughts in this comment. I was thinking to add Gradle tasks to build the extra artifacts, but haven’t really thought through the details.

Any help would be appreciated.

[jennettwheeler]

@Warkdev if you have experience with Gradle, then I would suggest you take a look at it as i have not had a proper look in earnest yet, so don't worry about taking over if that's what you want. I have absolutely no experience with Gradle, so am not the best person tackle this (unless @remkop wants to switch to Maven haha)

[Warkdev]

@jwheeler91 I think we've the same level of knowledge with Gradle. I just recently adapted a Gradle-projet to JPMS recently. I do have a preference for Maven projects :)

@remkop Will have a look, hopefully I won't make any mistake.

[remkop]

Having worked with both Maven and Gradle I much prefer Gradle. As with any technology, there is some learning involved, but that is an opportunity, right? ;-) Don't worry about making mistakes, we'll just fix them. :-)

One idea is to introduce a new Gradle subproject in the project, picocli-core-module (or maybe picocli-jpms-module?). This subproject will have its own build.gradle and its own src/main/java/ directory (or should that be src/main/java/info.picocli/?) with at least one source file: module-info.java:

module info.picocli {
    exports picocli;
}

In that subproject we need to use Java 9 to compile this source file.

We can let this subproject depend on the main project, so that the main parent project's jar is created before the subproject's jar. (See picocli-codegen/build.gradle for how to depend on the root project).

The next step would be a bit unconventional. Usually a subproject's build.gradle only works with resources in its own subproject, but in this case we may want to take the root project's jar file, and work with that. Concretely, we want to unjar it in a temp folder, remove the Groovy classes, add our module-info.class, and then re-jar it into a modular jar.

TBD:

• does the module-info.class live in the root of the jar or under /META-INF/versions/9/? (Looks like the latter is better.)
• do we (should we) create the modular jar with the --module-version option?
• Would the picocli-core-module Gradle build file also be responsible for creating a separate artifact (picocli-groovy? picocli-groovy-jpms-module?) with just the groovy classes, or should we create another subproject with its own Gradle build for that?
• I guess we should remove the 'Automatic-Module-Name' : 'info.picocli' from the jar manifest attributes of the root project, so that the picocli-4.x.x.jar is no longer an automatic module (two modules cannot contain the same package).

[Warkdev]

Great, so did I start the right way.

However, I've a first remark. To manage JPMS, I need to use gradle plug-in which is compiled with Java 11, I hope this is not an issue.

Currently, I'm also fighting with junit and gradle because I do need a second module info for the test module (why do I need to make it a module too? Or there's something really wrong with the way my IDE and gradle handles it).

Still early stages.. I've forked it to my repo and make some changes but no commit yet.

[remkop]

About the JPMS plugin, not sure why this is needed, but go ahead, we can refine later. Just out of interest, which plugin is that and what does it do?

About the tests, there is no need to make the picocli-tests jar artifact into a JPMS module.

As a side-note (since you mention Gradle plugins and this may come up), we won't be able to upgrade from Gradle 4.10.2 to Gradle 5 because it refuses to compile source code to target Java 5 byte code... Just FYI.

[Warkdev]

Hi, then we may have a problem Houston,

Im using the following:

id "org.javamodularity.moduleplugin" version "1.4.1"

I used it because that's one of the first results Google gave me. However, I see some other sources that I may try this evening without that plug-in. I keep you posted of the outcome.

Cheers,
Warkdev

[remkop]

I see. So when you mentioned tests, you didn't mean the existing unit tests, but you were talking about a separate set of tests to verify that the picocli-jpms-module artifact can be used as a valid JPMS module? And to do that, these tests themselves need to be in a module that requires and/or uses the picocli module. That makes sense. I understand now.

But perhaps we don't need to use a plugin. Plain vanilla Gradle is quite powerful and often Gradle plugins are just convenience with a nice DSL (this is different from the Maven world where one needs plugins for most any functionality). Have you had a chance to look at this guide?

[Warkdev]

Gosh you think faster than I do!

Indeed that's exactly what I meant. I'll read the guide this evening and move forward without the plug-in dependency. That's indeed too much maven style.

Sorry for my comment brievety, I'm on a phone and it's not that user friendly. I checked also this: : https://melix.github.io/javaone-2017-jigsaw/

[remkop]

No problem at all. Additionally, here is the Gradle 4.10.2 user guide to building Java/JVM projects, and another one for testing. May be useful for reference.

[jennettwheeler]

Thanks for these links. Will be useful for learning gradle for me too.

[Warkdev]

Hi guys,
I've been making some progress, at least I can compile it now using Java 9, Gradle 4.10 and with a single module-info file. However, I still have 2 pending issues:

• The module java.sql is used only at test time (not inside the src/main/picocli package) but I don't find any other way than adding requires java.sql inside the main module-info.java.
• The second one is that when I'm testing this build, I'm having some groovy-errors. I didn't expect the groovy project to be used in these tests.

java.lang.module.FindException: Unable to derive module descriptor for my-local-path\groovy-all-2.4.10.jar Caused by: java.lang.module.InvalidModuleDescriptorException: Provider class moduleName=groovy-all not in module Process 'Gradle Test Executor 2' finished with non-zero exit value 1

Any hint ?

Thx,

Warkdev

[remkop]

Good progress!

• We may need to add requires static java.sql in the main module-info.java, to make it a conditional dependency: if it's there, picocli can see it and use it, if it's not there, there's no error
• What tests are you referring to, picocli's unit tests or a separate set of module tests?

To deal with the groovy classes, I was thinking to exclude them from the main picocli JPMS module, and potentially create a separate picocli-groovy-jpms-module that would require groovy. But I'm not sure if a groovy module exists that we can require...

If this is a problem, perhaps we need to create a test setup that somehow only runs the java tests.

[Warkdev]

Hi,

sure I will update this in the module-info. Regarding your second question, that's the problem. I can't yet figure out which test is breaking, I'm getting a weird io exception with gradle executor and pipe being closed.

Could it be because of some groovy tests? I've been tempted to update the root build.gradle the remove groovy plug-in to test but didn't had any luck so far.

Im unfamiliar with groovy, even a complete stranger to it and I don't really understand the logical split in the root module for it, yet.

For info, I also had to do some cast in junit tests as some assertequals methods were ambiguous (obj, obj) and (int, int) methods.

Finally, I've completely duplicated main and test folders into the picocli JPMS module, was that what you wanted? Or did you want to keep the tests in the root project?

Thx for your hints.

Warkdev

Ps: I've a kind of mixed hate/love feeling with gradle right now. 😔😊

[remkop]

Finally, I've completely duplicated main and test folders into the picocli JPMS module, was that what you wanted? Or did you want to keep the tests in the root project?

Oh, I see, I didn't understand what was happening earlier. What I had in mind was something different. I was thinking that the picocli-jpms-module would not have any source code other than the module-info.java in src/main/java/. (So we would not copy CommandLine.java or AutoComplete.java.)

Instead, I was planning to do something sneaky: we just copy the compiled class files from the root project (minus the groovy classes) into the build/classes/java/main directory of our picocli-jpms-module before we create our modular jar. This could look something like this:

// picocli-jpms-module/build.gradle

task copyRootProjectClasses(type: Copy) {
    from("${rootProject.buildDir}/classes/java/main") {
        exclude '**/groovy/'
    }
    into 'build/classes/java/main'
}
compileJava.doLast { // this is run after compiling module-info.java
    tasks.copyRootProjectClasses.execute()
}

For the tests, I just realized that we cannot run unit tests against our module because JUnit itself is not a module...

We do want to verify that the module works, but it does not need to be as exhaustive as the root project's unit test. How about creating another "test-client" module that has some stand-alone applications, so we can put both the picocli module and this "test-client" module on the module path and run these applications (and maybe check their exit code to verify success)? We may be able to use the JavaExec Gradle task for this. (I haven't tried this...)

apply plugin: 'java'

// moduleName is the name of our "test client" module
mainClassName = "$moduleName/picocli.jpmsmodule.testclient.App1"

task runApp(type: JavaExec) {
    jvmArgs = [
            '--module-path', classpath.asPath,
            '--module', mainClassName 
    ]
    classpath = files()
    //classpath = sourceSets.main.runtimeClasspath

  // arguments to pass to the application
  args 'appArg1'
}

[Warkdev]

Hm, ok I better understand now, it was indeed a stupid idea to me to duplicate all the source code :)

I'll check tonight what's feasible. About JUnit, I'm using it with other Java 11 projects so it should be working. Some doubts remains about the ambiguous call to assertEquals with (int, T). Shall I update the root project code at all ?

Cheers,
Warkdev

[jennettwheeler]

When I was giving it a go, I was trying to create a core module with the source code which was depended upon by the jpms module and by the non jpms module. (Plus then other modules like a groovy module... I was doing that because its a pattern I follow with maven (ie. no code in the main project, having the main project be a "module of modules", with an assembly module for creating packages). I think doing that would be too large an architecture change. And may not be the best architecture for large java programs (welcome input)

[remkop]

@Warkdev About the ambiguous call error: The compiler is not giving any warnings or errors when compiling with Java 5, so it's not currently a problem. I would not mind updating though, so that the tests compile with later versions of Java as well. (But let's make sure that things still work fine when compiling with Java 5...)

About JUnit, at least with JUnit 4 we cannot execute unit tests _as modules_: JUnit is not a module so this won't work:

java --module-path picocli-jpms-module.jar;junit-4.x.jar;picocli-module-tests.jar \
     --module org.junit/org.junit.runner.JUnitCore picocli.jpmsmodule.testclient.App1Test

Maybe JUnit 5 is modular enough that something like the above can work. (This article may be useful.) But if this turns out to be too hard I was thinking we don't _have to_ use JUnit. I'm open to any ideas...

[remkop]

@jwheeler91 I'm familiar with that project structure, it works well with Maven. I haven't tried it with Gradle. I'm happy with the current project structure for picocli. Maybe what you describe would be better, but there are so many other things I want to improve in picocli that I don't want to spend time on it. :-)

[jennettwheeler]

Haha, Ok. At least you recognise the pattern, so I am not barking up the wrong tree with my other projects.

[Warkdev]

@remkop I'll let you manage Java 5. I hope it will match, I'm not fan of setting up a platform that was created before my birth ;)

I've already Java 9-10-11 on my machine (because I recently removed Java 8), it's enough ! :)

Will copy/paste my gradle config' tonight so that you can see all my mistakes.

Warkdev

[Warkdev]

@remkop When I do run the 'check' task of Gradle on the root project, I'm getting:

picocli.ArgGroupTest > testCompositeValidation FAILED java.lang.AssertionError at ArgGroupTest.java:1939 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.junit.contrib.java.lang.system.EnvironmentVariables (file:/C:/Users/Cedri/.gradle/caches/modules-2/files-2.1/com.github.stefanbirkner/system-rules/1.17.1/3642fe208063ad538ec6a2fca141d13e15f2b1f2/system-rules-1.17.1.jar) to field java.util.Collections$UnmodifiableMap.m WARNING: Please consider reporting this to the maintainers of org.junit.contrib.java.lang.system.EnvironmentVariables WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release

Am I missing something ?

Warkdev

[remkop]

Sorry about that. That was failing because of a real issue: #655.
I've @Ignore-ed the test until I have a fix.

[Warkdev]

Oh, that's why !

Thanks for the explanation, I can thus safely deny that test from the code and try to move forward.

Warkdev

[Warkdev]

Hello @remkop,

I've managed, I think, to achieve the desired behaviour for the subproject (or to be close to it).

Could you please review this commit before I send you out a PR ? https://github.com/Warkdev/picocli/commit/760217a078d9991a4d48259f167104b119a60482

Thx,

Warkdev

[remkop]

Thanks @Warkdev, great progress! I’ve put some comments.

[Warkdev]

I tried to come closer to your requirements: https://github.com/Warkdev/picocli/commit/095ea30b6c7d02cf861b6e13bd7d9ebb90cdfdf9

Now, sadly thanks to a plugin, the module-info is under a META-INF/versions/9 subfolder. Jar is now using the default JAR method and contains a MANIFEST.MF in the META-INF folder as you expected.

Warkdev

[remkop]

I like that the module-info class is now under a META-INF/versions/9 subfolder. This is not a guarantee that older tools won't choke on the Java 9 byte format, but it reduces that probability, so I really prefer this to having module-info.class in the root of the jar. Good stuff.

The moditect plugin is an interesting find. Looks like it only requires Java 8 and uses a byte manipulation library to create a Java 9 class file for the module-info class. An unusual approach, but I am okay with this for the initial cut.

_Side note: I just found this article that gives a bunch of options for what we are trying to do. The moditect plugin is one of them.)_

We can take in your work as is, but it would be great if we could test that the module works as part of the build. Not sure how easy/hard this is, it may be harder than creating the modular jar... Perhaps the moditect plugin may be able to help with this. It may also be possible to do this with JUnit 5 (but this looks very early stages: https://github.com/junit-team/junit5-samples/tree/master/junit5-modular-world), and another alternative is to create something ourselves as I described in an earlier comment.

Do you think you will be able to add a test?

[Warkdev]

Well, I can try to add some basic testing. What is the test you really want to achieve? That this subproject is well detected as a valid module?

[remkop]

I would like to verify that a JPMS modular application can be put on the module path together with our new picocli modular jar, and run as a command line application without anything on the classpath (so, module path only).

[Warkdev]

Hi,

I haven't given up but real life and need for sleep caught me :) I'm getting some headache to have junit 5 working with Gradle 4.10. You're really trying to find a difficult balance between Java 5 & JPMS system.

Warkdev

[remkop]

Sleep is important! And so is real life! :-)
No worries, take your time. If you want we can split this ticket in two parts, and have separate PRs for the module and the module tests. That way we can get community feedback on the module even if the tests are not ready.

[Warkdev]

I think it will be wiser if we want to have this being released :)

Maybe, with this start, someone else will jump on it and bring all his gradle knowledge !

[remkop]

No problem, let’s do it that way.

[remkop]

Can you submit a PR for the module bits?

[Warkdev]

Done boss!

[remkop]

PR merged. Thanks for the hard work!

Note to self: keeping this ticket open: still need to remove the automatic module name from the manifest of the picocli-4.x.jar artifact.

[remkop]

Following up on the conversation on PR #662:

If anyone wants to help out further, one thing I can think of is that it would be nice to have a README.md for this module.

Also, creating automated tests turned out to be quite challenging, but it would be good to confirm manually that the artifact produced by the build for this subproject can in fact be used in a modular java application (that is, just using jars on the --module-path, nothing on the classpath). Maybe the README could be a mini-tutorial on creating a modular CLI application with this module.

By the way, I am still undecided, but considering to rename this module to picocli-jpms-module. Feedback welcome.

[remkop]

Credit where credit is due: https://twitter.com/picocli/status/1116722078516826112?s=21
:-)

[Warkdev]

Ni need for credit but thanks. You made this awesome lib'!

[remkop]

Note to self: it turns out that the (nice and short) DSL syntax for configuring the plugin has some issue in some environments where I am testing. The legacy plugin application syntax is more verbose but does not have this issue:

buildscript {
    repositories {
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }
    dependencies {
        classpath "gradle.plugin.org.beryx:badass-jar:1.1.3"
    }
}

plugins {
    id 'java'
    id 'distribution'
    id 'maven-publish'
    id 'com.jfrog.bintray'
    id 'java-library'
    //id 'org.beryx.jar' version '1.1.3'
}

apply plugin: "org.beryx.jar"

[remkop]

Note to self:
Charles Nutter reported he had issues after putting the module-info.class in /META-INF/releases/9/ :
https://twitter.com/headius/status/1037931427000725504

So it may be safer to put it in the root of the jar after all. Since we publish two artifacts, old tools that cannot handle Java 9 bytecodes can always use the non-modular picocli.jar.

I will take a look at this later, before 4.0-GA.

[remkop]

I created #674 to follow up.