Clap: Auto generate bash completions

[kbknapp]

Thanks to @kamalmarhubi for reminding me about this and making some suggestions.

[kamalmarhubi]

Copying comments over:
@kamalmarhubi (original comment)

Is this in the feature list? I just started using clap, and the structure seems very amenable to allowing completion. Though rather than generating an auto-completion file, I had been thinking of a special completion mode enabled by flag or env var. Completion mode would take a partial command line and offer a list of completions. The benefit is that the program can use runtime information to complete values for some args, eg drawing values out of a database.

Either way, I think it's a really worthwhile feature.

[kamalmarhubi]

@kbknapp (original comment)

Yes it is....although I have created anew issue to track it since it's just been on the back burner. I've thought about doing like you said but what I'm leary of is including that functionality unconditionally in programs that don't need or care about it. Perhaps only compiling in that functionality only in debug mode? I need to think on this and am open to all suggestions :)

[sru]

@kamalmarhubi I am quite confused on the "completion mode" that you speak of. AFAIK, shells (bash, zsh...) cannot get runtime information of programs that are not yet run. They do completion by predefined completion functions. For example, here is bash completion for git.

@kbknapp Since generating completion file is one time process, it'd be more appropriate if we could do it on compile time. Nobody wants an overhead of generating completion file every time a program is run ;)

[kamalmarhubi]

I'm leary of is including that functionality unconditionally in programs that don't need or care about it. Perhaps only compiling in that functionality only in debug mode? I need to think on this and am open to all suggestions :)

I have no idea how they work or if they're appropriate, but it sounds like a possible use for features in the Cargo.toml. Disabled by default, and downstream binaries can enable it by specifying a [dependencies.clap] section:

[dependencies.clap]
version = "2"
features = ["completions"]

[kamalmarhubi]

@sru: note that the completion file calls git multiple times to construct completions. I'm suggesting that a completion file for a Clap app could call the app itself with a special mode.

[kamalmarhubi]

Equivalents from a few other languages' ecosystems:

• cobra (go): https://github.com/spf13/cobra/blob/master/bash_completions.md
• click (python): http://click.pocoo.org/6/bashcomplete/
• yargs (nodejs): https://www.npmjs.com/package/yargs#completioncmd-description-fn

[kamalmarhubi]

I just noticed that clap is already using features for a couple of extras: http://kbknapp.github.io/clap-rs/clap/index.html#optional-dependencies--features

[kbknapp]

Since generating completion file is one time process, it'd be more appropriate if we could do it on compile time. Nobody wants an overhead of generating completion file every time a program is run ;)

@sru exactly my point :) Which is kind of my thoughts on either using some sort of syntax extension or the like to generate this at compile time, or _potentially_ allowing this completion to be generated at runtime only in via features like @kamalmarhubi mentions. This would allow similar to how we run clippy, just a one time build/run to generate the completions functions, then you build without the feature enabled for actual use.

[RaitoBezarius]

Just FYI:

@kamalmarhubi I am quite confused on the "completion mode" that you speak of. AFAIK, shells (bash, zsh...) cannot get runtime information of programs that are not yet run. They do completion by predefined completion functions. For example, here is bash completion for git.

fish is theorically able to get the documentation from the manual (man), I wonder if it could constitute a solution (at least for fish which _is_ NON-POSIX -- don't kill me please, I like shiny stuff).

[clux]

Hey. So.. I am managing a fairly decent sized clap based app that actually has a custom bash completions script. It wasn't actually to bad to write a completion script for a clap app (though still not something I'd recommend) because clap it has a quite rigid structure in how arguments and subcommands are parsed. I think my current script could have easily been generated by clap provided one or two extra instructions were added to clap.

Sorry for the wall of text.

Suggested completion algorithm

The bash side, the part that will be generated and passed to complete -F.

1. if the first argument was the binary name, and the first word after that starts with dash, complete from global flags
2. if no subcommands exist in app, complete from special completions lists if it exists
3. if no subcommands have yet been provided in the current arguments, suggest the list of global subcommands
4. if a subcommand has been given and the next argument starts with a dash, complete from the subcommands' flags
5. if no subcommands exist under current subcommand, complete from special completions list if it exists

And you would iterate 4 and 5 until you are at the bottom of the tree for this. Think this kind of recursion would work. I have a proof of concept up to, but not including this point (recursion), and it feels pretty great.

All the lists mentioned would be compiled into the bash script as variables, or as scripts to be called using extensions:

Suggested Extensions

Some choices to be made here to make this work. We could easily allow:

Static Completions

Special suggestions could be populated from a list that a user provides to clap.
E.g.:

let args = App::new("foo")
    .subcommand(SubCommand::with_name("fetch")
        .arg(Arg::with_name("components")
          .completions(vec!["yajl", "zlib"])))

which would be nice, easy to implement (just dump the list straight into the completion script as a variable), but it's not very flexible.

Dynamic Completions

A dynamic option would be more useful if if the list is dependent on some outside state that needs IO to verify. E.g., if a command like fetch needs to look up against a networked registry, or some local database / yaml file / directory structure.

I can see two options from clap's side that could provide this:

1. Inline bash

Easy to implement (just dump the function straight into the completion script to be eval'd), but it's not a particularly elegant solution.

let args = App::new("foo")
    .subcommand(SubCommand::with_name("fetch")
        .arg(Arg::with_name("components")
            .completer("find ~/.foo/cache/ -maxdepth 1 -mindepth 1 -type d -printf \"%f \"")

2. Allow custom functions

Allow completer functions that do some computation or IO:

fn fetch_completer() -> Result<(), Vec<String>> {
    // provide a list from some IO operation
}

let args = App::new("foo")
    .subcommand(SubCommand::with_name("fetch")
        .arg(Arg::with_name("components")
            .completer(fetch_completer)

clap could then expose that function secretly as a hidden subcommand, say foo __fetch_suggestions, which could be eval'd in the generated completion script at the appropriate time in the algorithm.

----

In my current app, I am kind of using a combination of the two already:

• bash functions to do computation/io
• suggestion functions listed in main.rs as proper clap subcommands (but would like to hide these).

----

So.. Does this approach seem sensible? Anyone see better approaches?
I'd be willing to have a go trying to implement something like this in clap if there's interest.

[kbknapp]

Excellent! Thanks for taking the time to put all this together! I'm still on vacation for the next two days, but let me do some looking into this and I'll post back here with what I find. I'd be excited if we can get this into clap. Also, I've been working on the 3x branch, so this might get put in with those changes which would allow for a few changes in a breaking manner if need-be, although I'd try to err on the side of non-breaking changes.

[clux]

Cool. Enjoy your vacation :)

I have a rough prototype of something working here that maybe you could have a look at when you get back. A very hacky thing, but it gets some basics done.

[kbknapp]

I've finally began working on this issue! The method I'm going with uses a build.rs script and outputs a completion file which from initial testing works great even with subcommands and aliases. It's not perfect as it's "dumb" about what positional args are left but as an initial approach I'm happy. Once I do a little more testing and tweaking I'll put in the PR.

Edit, to be clear this also works by only completing on the valid options for each individual subcommand ;)

[clux]

Awesome. I had no idea how to actually generate the completion script without hooking into the arg chain at the very end when it was fully populated. Glad you have found a sensible way.

I'd be happy to test it out and look for some edge cases when you get it in a sensible state :smile:

[kbknapp]

I went ahead and put you 2.7.0 since I had limited time to work on this over the past few days. This issue is next in the hopper.

[kbknapp]

I just finished implementing this. Putting in the PR now. As a sample, I built rustup with this method and here's the generated completion script. I've tested it and it works through all subcommands and arguments. I'm quite happy with it :smile:

[clux]

Very nice. Just tested it on my app and got 400 line bash.sh that works pretty well :+1:

Is there a way to provide dynamic completion values to the script here? E.g. like completor fn above.

[clux]

I like how it suggests positional arguments as <argName>.. as you will never actually get that into your shell (provided you have at least one flag), but it provides helpful info in the completion list.

Found one issue:
If you type out a subcommand alias instead of the normal subcommand name, then you can't get completions for that subcommand.

[kbknapp]

I plan on adding aliases and possible values next prior to updating on crates.io. There isn't a way to use a completer fn yet, but it should be addable once I have this base system complete.

[kbknapp]

Aliases and possible values are now completable. Once I get home tonight I'll update the version on crates.io

[kbknapp]

For the curious, here's a demo for rustup

[nerdrew]

Are there plans for zsh completion?

[kbknapp]

@nerdrew Yes there are, I just haven't had time to implement it yet. #579 adds Fish support. I'm also open to people submitting a PR for ZSH :wink: