Picocli: Allow repeatable subcommands?

[idanarye]

I came here looking for a solution for a problem similar to #358 - the need for a syntax for lists of multi-field entries - but had a different solution in mind. These solutions are not mutually exclusive, so I elected to open a new ticket for it.

My idea is to have repeatable subcommands. So - if we take the example from #358 - print will be the main command and have, say, file as a subcommand. We will write it like so:

print --paper A4 \
    file A.pdf \
    file B.pdf --count 3 \
    file C.pdf --count 3 --rotate left \
    file D.pdf \
    file E.pdf --rotate right

This will create 6 CommandLine objects:

• PrintCommand(paper = "A4")
• FileCommand(name = "A.pdf", count = , rotate = )
• FileCommand(name = "B.pdf", count = 3, rotate = )
• FileCommand(name = "C.pdf", count = 3, rotate = "left")
• FileCommand(name = "D.pdf", count = , rotate = )
• FileCommand(name = "E.pdf", count = , rotate = "right")

A bit more verbose than @kravemir's original syntax (since you need to specify the options for each file) but much more readable IMHO.

I tried to do it by making file a subcommand of itself:

CommandLine printCommand = new CommandLine(new PrintCommand());
CommandLine fileCommand = new CommandLine(new FileCommand());
printCommand.addSubcommand(fileCommand);
fileCommand.addSubcommand(fileCommand);

And it did parse that command line - but it was always using the same FileCommand object so I only got E.pdf's parameters. Maybe if a CommandLine could receive a factory for objects, and construct a new one for each subcommand it parses?

[kravemir]

It's very similar to, what was suggested later on, in comment: https://github.com/remkop/picocli/issues/358#issuecomment-381321986 :-)

The idea is @CompositeParameters, which is a bit similar to @Mixin. The @CompositeParameters can contain parameters and options, and can be used as a repeatable part of command.

Anyway, I'm up for implementation of such solution. The print was just an example, the real use-case is image manipulation/transformation tool.

EDIT: not exactly same thing, but similar.

[kravemir]

Also, an issue already present: #434

[remkop]

@idanarye I can see there is definitely a need for improvement in this area.

I’m not sure yet about repeatable subcommands; I feel the hierarchy is important in a number of ways (incl. usage help and autocompletion), and worry that would be lost if subcommands could work at any level.

I will focus on #358 first.

About your example, have you tried passing in FileCommand.class instead of new FileCommand()? I believe (away from PC) that picocli will instantiate the class and use the instance as the user object for the CommandLine.

[idanarye]

About your example, have you tried passing in FileCommand.class instead of new FileCommand()? I believe (away from PC) that picocli will instantiate the class and use the instance as the user object for the CommandLine.

Didn't work. Picocli did initiate a class instance from the Class object - but it only did so once.

[remkop]

@idanarye There is a lot of merit to your suggestion. If composite options (and composite positional parameters) are implemented with commands under the hood, all the parsing logic can be reused pretty much as is.

I'm still thinking about:

• what should the annotations API look like (currently thinking a @Composite annotation on the bean class that defines the @Parameters elements)
• what should the programmatic API look like (currently thinking ArgSpec _has a_ CommandSpec attribute for composite options and composite positional parameters, which is null for non-composite options - it does not feel right to model this as an _is a_ relationship)
• usage help: what should the usage help for a composite option/positional parameter look like (difference from a normal option)? What changes in the CommandLine.Help API are needed?

[remkop]

I realized that composite ArgSpecs having a CommandSpec.Builder is a better model, especially if the builder takes just the _class_ of the composite bean in the builder constructor. Then builder.build() creates a new CommandSpec and a new composite instance each time the composite option (or composite positional parameter) is specified on the command line. The builder would act like the factory you mentioned earlier.

[remkop]

Picocli 4.0.0-alpha-1 has been released which includes support for repeating composite groups.
See https://picocli.info/#_argument_groups for details.

I believe this should should meet the requirements of the original use case.

_Please try this and provide feedback. We can still make changes._

_What do you think of the annotations API? What about the programmatic API? Does it work as expected? Are the input validation error messages correct and clear? Is the documentation clear and complete? Anything you want to change or improve? Any other feedback?_

[remkop]

Reopened as repeatable subcommands may have use cases not covered by repeatable argument groups. #635 may be an example.

[hanslovsky]

I just tried and I can add sub-commands with double-dash syntax, so that would be a feasible solution for me, as well. I could even have a sub-command with repeatable argument groups and that would be a working solution already, e.g. instead of

command sub-command --opt1=a --opt2=a sub-command --opt1=b --opt2=b

I would have something like this (I made sub-commands plural to stress the fact that it can add multiples):

command sub-commands --opt1=a --opt2=a --opt1=b --opt2=b

where opt1 and opt2 are part of an ArgGroup. This is a little less intuitive than multiple repetitions of sub-command but it should be a viable workaround.

[remkop]

To support repeating subcommands it may make sense to give commands a multiplicity attribute.

This multiplicity indicates how many times the subcommand ca be specified. The default would be "0..1", meaning that by default, a subcommand is optional but can be specified at most once.

If a subcommand can be specified multiple times, it should be defined something like the below:

@Command (multiplicity ="0..*")
class MyCommand {}

[remkop]

This is proving to be a popular feature: see this stackoverflow question.

[remkop]

TBD: after doing some prototyping:

• API: how should applications enable this? @Command(repeatable = true)? @Command(siblings = true)?
• What are the exact requirements? Do we want to allow end users to specify either subcommands or the current command or do we also want to allow sibling commands?

What is clear is that to support this, the ParseResult class needs another method subcommands() that returns the list of subcommands that were matched for the current command.

[remkop]

I am thinking to shelve this feature until the requirements are more clear.

The example in #635 and the description of this ticket are the only use cases so far., and they can be addressed with composite repeating groups (although the verb "add" in #635 suggests that a subcommand would be a better fit than than a repeating option group).

All, if you have ideas, suggestions, requirements: please comment here!

[lakemove]

To add 1 possible usecase :

eodcli \
  import --db=jdbc:oracle:local --user=sa --sql="select * from data" \
  report --out /tmp/report.csv \
  email  --recipients [email protected] 

chain several subcommands together : read db , generate report and send email notification.
jvm is too expensive to use pipe.

[remkop]

For reference:
As mentioned in https://github.com/remkop/picocli/issues/870, the python Click library uses command chaining and the command group callback feature to accomplish repeated subcommands.

The parser side of things is hopefully not too difficult to implement, but I am still thinking about what API to use to expose this functionality. Currently thinking to add a multiplicity range to the @Command annotation, similar to multiplicity in @ArgGroup.

@Command(name = "main", subcommands = { ListCommand.class })
public class Main implements Runnable {
    public void run() { }

    public static void main(String[] args) {
        CommandLine cmd = new CommandLine(new Main());
        int exitCode = cmd.execute("add", "add", "add", "list");
    }

    @Command(name = "add", multiplicity = "0..*")
    int add() {
        return 0;
    }
}

@Command(name = "list") // default multiplicity = "0..1"
class ListCommand implements Runnable {
    public void run() { }
}

This is easy to understand, easy to code, and can be picked up by the annotation processor to auto-generate documentation in the future.

One disadvantage is that by defining multiplicity on the subcommand, it cannot be re-used with a different multiplicity when added to a different parent command. If there is a need for such flexibility it could potentially be addressed with a programmatic API like CommandLine.addSubcommand(Object command, Range multiplicity).

[remkop]

After thinking some more, I now believe that my earlier thinking to use multiplicity is the wrong way (or at least insufficient) to model this.

The current model (a hierarchy of subcommands) can be represented by a _directed rooted tree_:

This ticket proposes that we additionally also allow sibling subcommands to be repeated. This is a property of the parent command node, not of the subcommand node.

When a parent command has this property, its subcommands form a fully connected graph (a _complete digraph_):

This makes me think it is more "correct" to model this as a property of the parent command.

In coding terms, we can add a subcommandsRepeatable attribute to the @Command annotation. That would allow applications to make the subcommands of specific commands "repeatable". For example:

@Command(name="A",
         subcommands = {B.class, C.class, D.class}, 
         subcommandsRepeatable = true)
class A {}

@Command(name="B",
         subcommands = {E.class, F.class, G.class}, 
         subcommandsRepeatable = true)
class B {}

The above command definitions would allow user input like this:

A B B C D B E F G E E F F

But it is not allowed to specify a child command followed by its parent command:

# incorrect: cannot move _up_ the hierarchy
A B E B

This attribute is not inherited by subcommands; each subcommand must explicitly specify whether its sub-subcommands are repeatable or not.

Some ideas for naming:

• subcommandsRepeatable
• subcommandsConnected
• subcommandsCompleteDigraph

Limitations

This model has some limitations:

• it is not possible to restrict the number of repetitions
• the "repeatable" attribute applies to all subcommands. Some applications may want to only make _some_ subcommands repeatable and exclude other subcommands.

If these limitations later turn out to be really problematic, it should be possible to address them with additional attributes. One idea is to introduce a multiplicity range attribute that a subcommand would use to define how often it can be repeated.

Implications

In the current model (a hierarchy of subcommands), each command can occur only once on the command line. Picocli currently instantiates all command objects (including the user object) at initialization time. For some applications this eager instantiation is problematic; for example, https://github.com/remkop/picocli/issues/690 was raised to request that the command's user object would not be instantiated until it is required.

In the new model, any command other than the top-level command can occur multiple times. This has implications for the life cycle of the CommandLine object graph: each subcommand occurrence may be invoked with different options and must have its own unique user object instance.

Take the following example command line:

A B B

I still need to figure what objects need to be created when in the new model.

• Do CommandLine objects need to be cloned when a subcommand is recognized? Do we need to distinguish between the first occurrence and subsequent occurrences of the subcommand?
• What about the CommandSpec of the subcommand? Does it need to be cloned or can multiple instances of the same subcommand share a CommandSpec model?
• Where should the "state" of a command live? Currently the user object is a field in CommandSpec. The ArgSpec objects of a command also have a user object field, as well as IGetter and ISetter objects that have a user object scope. This needs to be clarified and ideally simplified.

[remkop]

@idanarye, @kravemir, @hanslovsky, @lakemove, a first cut of support for repeatable subcommands has landed in master. Please give it a try and let me know what you think.

Usage:

@Command(name="A",
        subcommandsRepeatable = true,
        subcommands = {B.class, C.class, D.class})
class A {}

@Command(name="B",
        subcommandsRepeatable = true,
        subcommands = {E.class, F.class, G.class})
class B {}

@Command(name="C") class C {}
@Command(name="D") class D {}
@Command(name="E") class E {}
@Command(name="F") class F {}
@Command(name="G") class G {}

This allows input like this:

A B B C D B E F G E E F F

Which is parsed like this:

|
+--B
+--B
+--C
+--D
+--B
   |
   +--E
   +--F
   +--G
   +--E
   +--E
   +--F
   +--F

[remkop]

Status update

I believe the implementation is now complete. I added tests and updated the documentation.

Remaining work is just doc tweaking; I will reorder the sections under Subcommands a little.
When this is done I will close this ticket.

How you can help

If you are interested in repeating subcommands, you may have a use case that you want to try this on. You can test by checking out the latest master and building with:

gradlew clean publishToMavenLocal

That should publish picocli-4.2.0-SNAPSHOT to your local .m2 Maven cache. You can then try this in a project that uses the info.picocli:picocli:4.2.0-SNAPSHOT dependency.

Feedback welcome!

[hanslovsky]

This looks cool! Haven't really gotten to try this yet but I prepared a kscript from your example above which may be helpful for quick prototyping:

#!/usr/bin/env kscript

@file:DependsOn("info.picocli:picocli:4.2.0-SNAPSHOT")

import picocli.CommandLine
import picocli.CommandLine.Command

@Command(name="A",
        subcommandsRepeatable = true,
        subcommands = [B::class, C::class, D::class])
class A

@Command(name="B",
        subcommandsRepeatable = true,
        subcommands = [E::class, F::class, G::class])
class B

@Command(name="C") class C {}
@Command(name="D") class D {}
@Command(name="E") class E {}
@Command(name="F") class F {}
@Command(name="G") class G {}

val a = A()
val cl = CommandLine(a)
val parseRsult = cl.parseArgs(*args)

[hanslovsky]

I tried a repeatable subcommand with a String CommandLine.Option. If the option has arity="1", then everything works as expected but if the arity="*" then the repeated specification of the subcommand will be interpreted as an option. This is an ill-defined problem by itself and I understand that the ambiguity has to be resolved arbitrarily, i.e. parse everything as a value for the option with arity="*". Is this the intented behavior? The alternative would be for the sub-command know any other sub-command names but I can see that this can get messy quickly. As an alternative solution, is there an argument to indicate that subcommand is completet, e.g. --.

This is my example:

#!/usr/bin/env kscript

@file:DependsOn("info.picocli:picocli:4.2.0-SNAPSHOT")

import java.util.concurrent.Callable

import picocli.CommandLine
import picocli.CommandLine.Command
import picocli.CommandLine.Option
import picocli.CommandLine.Parameters

@Command(name="MainCmd",
        subcommandsRepeatable = true,
        subcommands = [Container::class])
class MainCmd : Callable<Int> {
    @Option(names=["--help", "-h"], usageHelp=true)
    var helpRequested: Boolean = false

    override fun call(): Int {
        return 0
    }
}

@Command(name="--with-container")
class Container : Callable<Int> {
    @Parameters(arity="1")
    lateinit var path: String

    @Option(names=["--dataset", "-d"], arity="*")
    var datasets: Array<String>? = null

    @Option(names=["--help", "-h"], usageHelp=true)
    var helpRequested: Boolean = false

    override fun call(): Int {
        println(this)
        return 0
    }

    override fun toString() = ContainerData(path, datasets?.toList()).toString()
}

data class ContainerData(val path: String, val datasets: List<String>?)

val mainCmd = MainCmd()
val cl = CommandLine(mainCmd)
val exitCode = cl.execute(*args)

This results in:

$ ./test-repeatable-subcommands.kts --with-container abc --dataset a --with-container xyz --dataset x
ContainerData(path=abc, datasets=[a, --with-container, xyz, x])

If I change the arity of Container.datasets to"1", this is the output (which is what I would like to see):

$ ./test-repeatable-subcommands.kts --with-container abc --dataset a --with-container xyz --dataset x
ContainerData(path=abc, datasets=[a])
ContainerData(path=xyz, datasets=[x])

[idanarye]

I need this for the plotting feature in a data collection app I should be am writing. Each plot has:

• One or more axes.
• Zero or more filters - which are sliders/combos that can be played with in the GUI.
• One or more formulas - which are the values that I show in the plot.

Even though my app is still WIP I don't want to make it depend on a snapshot (I can procrastinate wait until you officially release 4.2.0) so I copied the data classes to a new repository and wrote a CLI just for creating and printing them: https://github.com/idanarye/test-picocli-repeatable-subcommands.

What I did is create a parent CommandLine that contains the PlotEntry object, and repeatable subcommands that axes, filters and formulas to it. This allows me to write single commands that create complete plot entries:

java -jar build/libs/shadow.jar my-plot \
    axis Year sample.year \
    axis Height sample.year -u centimeters \
    filter Age 'sample.year - person.birthYear' -u years -t NUMERIC_RANGE \
    filter Gender person.gender -t TEXTUAL_SINGLE \
    formula Salary sample.salary --symbol $ -u USD -S LOGARITHMIC \
    formula Happyness 'sample.pizzaEaten + sample.beerConsumed' --symbol ':-)'

The only drawback of this approach is that I need to some more processing after CommandLine.execute() to handle the result command - could be nice if I could make picocli execute some method on the parent command after all its subcommands have finished.

[remkop]

@hanslovsky Thanks for raising the arity issue! That was a bug.
I pushed a fix to master.

I also added repeatable-subcmds-example.kts to picocli-examples/src/main/kotlin/picocli/examples/kotlin based on your example script.

I tested the fix in Java but I was unable to run your script from IntelliJ (error: invalid argument: --with-container). Can you try again with the latest master?

[remkop]

@idanarye, the simplest way to invoke a method on the top-level command object that I can think of would be to call it from the main method in your program.

Something like this:

@Command(name = "topcmd", subcommandsRepeatable = true)
class TopCmd implements Runnable {
    public void run() {
        System.out.println("topcmd called");
    }

    @Command
    void axis(@Parameters(index = "0") String str, @Parameters(index="1") File f) {
        System.out.println("axis command called");
    }

    @Command
    void filter(@Option(names = "-u") String u, @Option(names="-t") String t) {
        System.out.println("filter command called");
    }

    /* ... */
    public void postProcessing() {
        System.out.println("...and we're done!");
    }

    public static void main(String... args) {
        args = "axis Year a.year axis Height b.year filter -u=x -t=y filter -t=tt".split(" ");

        TopCmd top = new TopCmd();
        int exitCode = new CommandLine(top).execute(args);
        top.postProcessing(); // execute some method on the parent command after all its subcommands have finished.
    }
}

[hanslovsky]

@hanslovsky Thanks for raising the arity issue! That was a bug.
I pushed a fix to master.

I also added repeatable-subcmds-example.kts to picocli-examples/src/main/kotlin/picocli/examples/kotlin based on your example script.

I tested the fix in Java but I was unable to run your script from IntelliJ (error: invalid argument: --with-container). Can you try again with the latest master?

@remkop I pulled the most recent master and tested my script and now it works as expected. Thank you for the quick fix.

[idanarye]

@idanarye, the simplest way to invoke a method on the top-level command object that I can think of would be to call it from the main method in your program.

It gets a bit trickier when you want to have multiple "real" subcommands and the ones you want to do postprocessing on are not the root ones, but it's still doable.

[remkop]

@idanarye I see what you mean now.

Picocli can help find the parent command of the executed subcommands. All that is required is that we make the command methods (or the Callable for a @Command-annotated class) return something, like an int, for example. (This is a natural thing to do anyway if your application needs to control the command's exit code.)

That allows you to use the ParseResult::asCommandLineList and CommandLine::getExecutionResult methods to find the parent: the parent will have a null execution result, but the executed subcommands will have a non-null execution result. For example:

// all commands now return an int
@Command(name = "topcmd", subcommandsRepeatable = true)
static class TopCmd implements Callable<Integer> {
    public Integer call() {
        System.out.println("topcmd called");
        return 0;
    }

    @Command
    int axis(@Parameters(index = "0") String str, @Parameters(index="1") File f) {
        System.out.println("axis command called");
        return 0;
    }

    @Command
    int filter(@Option(names = "-u") String u, @Option(names="-t") String t) {
        System.out.println("filter command called");
        return 0;
    }

    /* ... */
    public void postProcessing() {
        System.out.println("...and we're done!");
    }

    public static void main(String... args) {
        args = "axis Year a.year axis Height b.year filter -u=x -t=y filter -t=tt".split(" ");

        CommandLine cmd = new CommandLine(new TopCmd());
        int exitCode = cmd.execute(args);

        // Given that all commands return something,
        // we can use the CommandLine::getExecutionResult method 
        // to find the parent command of the subcommands that were executed.
        List<CommandLine> matched = cmd.getParseResult().asCommandLineList();
        CommandLine parent = null;
        for (CommandLine commandLine : matched) {
            if (commandLine.getExecutionResult() == null) { // this command was not executed
                parent = commandLine;
            } else {
                break; // we found the first subcommand that was executed
            }
        }
        System.out.println("The parent of the executed subcommands is: " + parent.getCommandName());

        // execute some method on the parent command
        if (parent.getCommand() instanceof TopCmd) {
            TopCmd top = parent.getCommand();
            top.postProcessing();
        }
    }
}

In Java 8, the loop to find the parent can be a bit cleaner:

// Given that all commands return something,
// we can use the CommandLine::getExecutionResult method 
// to find the parent command of the subcommands that were executed.
Optional<CommandLine> parent = cmd.getParseResult().asCommandLineList().stream()
        .takeWhile(c -> c.getExecutionResult() == null)
        .reduce((cmd1, cmd2) -> cmd2);

[idanarye]

Why return a meaningless integer, when I return a value for the top command to process instead?

I ended up with something like this:

    CommandLine cli = new CommandLine(new App());
    cli
        .addSubcommand("add", new PlotCommandAdd())
        .addSubcommand("show", new PlotCommandShow())
        .execute(args);
    SubcommandPostprocessing postprocessingTarget = null;
    LinkedList<Object> postprocessingArgs = null;
    for (CommandLine cmd : cli.getParseResult().asCommandLineList()) {
        if (cmd.getCommand() instanceof SubcommandPostprocessing) {
        if (postprocessingTarget != null) {
            postprocessingTarget.postProcess(postprocessingArgs);
        }
        postprocessingTarget = cmd.getCommand();
        postprocessingArgs = new LinkedList<>();
        } else if (postprocessingArgs != null) {
        Object executionResult = cmd.getExecutionResult();
        if (executionResult != null) {
            postprocessingArgs.add(executionResult);
        }
        }
    }
    if (postprocessingTarget != null) {
        postprocessingTarget.postProcess(postprocessingArgs);
    }

I added the execute-and-switch part in the middle of the loop hoping I can run multiple top level commands:

java -jar build/libs/shadow.jar \
    add my-plot \
        axis Year sample.year \
        axis Height sample.year -u centimeters \
        filter Age 'sample.year - person.birthYear' -u years -t NUMERIC_RANGE \
        filter Gender person.gender -t TEXTUAL_SINGLE \
        formula Salary sample.salary --symbol $ -u USD -S LOGARITHMIC \
        formula Happyness 'sample.pizzaEaten + sample.beerConsumed' --symbol ':-)' \
    show \
        axis Year sample.year \
        axis Height sample.year -u centimeters \
        filter Age 'sample.year - person.birthYear' -u years -t NUMERIC_RANGE \
        filter Gender person.gender -t TEXTUAL_SINGLE \
        formula Salary sample.salary --symbol $ -u USD -S LOGARITHMIC \
        formula Happyness 'sample.pizzaEaten + sample.beerConsumed' --symbol ':-)'

But this doesn't work. Should picocli recognize that show is not a subcommand of add and pop the command stack to check if it's another top level command?

[remkop]

At the moment, it is not possible to go _up_ the hierarchy with repeatable subcommands.
I've added a note to that effect in the documentation (build/docs/html5/index.html#_repeatable_subcommands).
I think repeatable subcommands can be confusing, so for now I'd like to keep it simple.
Thoughts?

[idanarye]

I'm already abusing them enough as is, so I'm not going to request any farther complications.

[remkop]

@idanarye, @kravemir, @hanslovsky, @lakemove, All,

picocli 4.2.0 has been released, including this feature. Enjoy!