Nim: parseopt: high level helper

[FedericoCeratto]

Summary

parseopt provides low-level procs and requires for+case to extract parsed values.
Nim could provide a more practical helper.

Features

• Parse args in one line and return a configuration object
• Integrate with parsecfg: Scan for <name>.cfg in known locations, read it and populate the configuration object. CLI options override the contents of the files
• Generate help automatically
• Generate help compatible with help2man (or generate a manpage)
• Support nested actions ("verbs") and contextual help e.g. foo delete cache; foo help delete
• Print application name and version as part of help
• Support parsing integers and floats
• Handle correctly 1-letter options followed by value: -o foo

[kaushalmodi]

/cc @c-blake as he might be one of the best experts in this field.

May be "something in-between" Nim stdlib and his cligen package can be ported to Nim stdlib?

(I would be happy if the whole cligen is ported to Nim stdlib :D)

[c-blake]

I think cligen (https://github.com/c-blake/cligen) covers every single requested feature (and many more!) except printing the application version as part of the auto-generated help. For versions, CL authors must add a line or two to activate and users must request via --version or whatever short option was selected. At some point I tested help2man compatibility, but it's been years. I do think just informing @FedericoCeratto of cligen is probably enough to close out this issue.

I'd be fine with inclusion in the stdlib instead of a Nimble package, but I think this runs counter to the zeitgeist of trimming the stdlib. I had originally hoped to at least get parseopt3 into some stdlib parseopt, but never pushed that hard for it. For what it's worth, I did get his 1-letter options working in the stdlib parseopt, though, but it is not "by default". The CL author must activate it by using a non-empty shortNoVal. See the module documentation.

[FedericoCeratto]

I'm aware of cligen. The main reason to have a (simpler) subset of its features in the stdlib is that CLI and config file parsing is a very common requirement for most developers and it hardly warrants adding a dependency on an external library.

[dom96]

Why not start this as a Nimble package? I 100% agree that we need something more friendly than parseopt, but I don't think whatever is developed should be included immediately in the stdlib without at least some testing as a Nimble package.

[c-blake]

I may be biased, but I feel like cligen features (and the inferred approach in general) "scale down" pretty nicely. If you only want to do something simple then you needn't know very much. As you want to do fancier things, you need to learn more. Also, FWIW, it started as a simpler package but people requested a ton of features. But if you want to take a stab at your own package, I don't want to discourage you.

[genotrance]

I'm against the idea of a parseopt3 when cligen is successful and popular. I'd rather see it added to the stdlib. It has proved itself as a nimble package.

I understand there's been discussions around removing older cruft but I don't see why we cannot continue to add mature packages into the stdlib. I've not seen any in the last two years and it honestly makes no sense.

[c-blake]

Minor point of clarification - when I said parseopt3 above, I meant what is now cligen/parseopt3.nim, and I only meant equipping the stdlib parseopt with everything cligen needs.

[Araq]

We need an RFC if and how to grow Nim's stdlib.

[luav]

I would prefer to see in the stdlib something like argparse rather then too high-level cligen because the purpose of the help is not just the API listing but rather a _human-friendly explanation of the parameters_, of their usage.

dispatch() of cligen is designed to generate an API for functions, which IMHO duplicates functionality of docstrings and the respective API manual, where the argument parser should provide an easy and uniform way to specify help for the _console interface_. argparse resembles standard and widely used Python argparse and GNU Gengetopt.

[c-blake]

I think you may misunderstand cligen. It is basically just like Gengetopt except that it's "specification language" is regular old Nim code/syntax that any Nim user ought to already know instead of some new micro-language they have to learn like a set of API calls/DSL like argparse.

Also, the most minimal usage of dispatch is not all there is. If parameter/option names are not self-explanatory, the API doc comment insufficiently descriptive, or automatic short options not quite right, the CL author has to learn dispatch(foo, help={"myOption": "human friendly explanation"}, doc="override API doc comment for help message", short={"myOption": 'z'}). Chances are, those are the only three dispatch parameters you need to know until you get fairly picky. And, yes, being an API itself, dispatch parameters are a _little_ bit to learn, but I think less than the alternatives in the common case. Why, that single sentence above almost covers it if you already know Nim.

[luav]

@c-blake Probably, you are right, but there are some concerns related to the cligen approach. It extracts the function interface requiring to process all (mode) arguments in a single function rather than constructing the respective args object. Also, it seems to restrict some expected functionality including:

• mutually exclusive group(s) of arguments (in the same mode),
• hidden arguments (physically present arguments but excluded from the help),
• mutually exclusive modes (commands with groups of options), which seems do not support sub-commands in cligen,
• mandatory vs optional arguments (unspecified argument is not the same as the specified one but having an omitted default value),
• ...

cligen functionality is far beyond the actual CLI generation in many aspects but it also lacks some expected features and has the outlined restrictions. It provides API generation for _functions_ being more convenient but arguably less readable and more heavy alternative to docstrings.
I would prefer to see docstrings (i.e., API manual) automation with cligen (-> apigen) rather than using it for the CLI help generation having some restrictions.

[c-blake]

Readability, convenience, preferences are subjective. Mandatory vs. optional is handled by the implicitDefault parameter to dispatch. I could probably interpret help={"param":""} as a way to "hide" the help row for param. Raise an issue over at cligen if you like.

The only main restriction you mention which is true is exclusivity/groups. Note, this is also beyond the syntax of usual procedure calls. I would say that "beyondness" calls its value into question. I'm open to possibly supporting it, but again such feature requests belong over on my git repo not here. You can always write a custom wrapper around some existing API that does exclusivity checks (which Nim importing/calling users might also appreciate!), but grouping is more syntax/parseopt behavior.

cligen does not generate a Nim callable API - the API author does that, the CL author maybe wraps that directly or writes their own wrapper. So, some of your descriptions confuse me. Anyway, you're under no obligation to like or even fully understand cligen. More power to you if you want do your own, or add to the argparse you linked to, get that hardened/more developed and do some RFC for Nim library extension both in general and for a specific module. I agree that a systematic procedure/RFC for expanding the stdlib is wise. Otherwise you wind up with 3 arg parsers in the stdlib like Python, just for example. I think most would agree that 3 is at least 1 too many.

You've mentioned doc strings a couple times when I think you mean doc comments. This suggests you are new to Nim coming from Python. Welcome! I kind of suspect what you may really be after is a kind of "Python -> Nim porting library/system" - a full suite of many modules and procs whose names, structure, and behavior mimic as closely as possible the Python stdlib/stdecosystem..This would let people change their mental models of various task decompositions as little as possible and port code/scripts as quickly as possible. Maybe even with a py2nim translator that translates to "readable/hackable Nim" code. Those sound like good ideas to me especially because 95+% of the Python I've seen does not use most of its dynamic features and is often even written more like Java. Someone has probably even already started these projects somewhere and maybe someone besides me can point you to them.

This thread is roughly of the form "stdlib parseopt not _exactly_ what I want/expect, nor is cligen. Pick what commenter A,B,or C wants exactly and then force it on everyone via the stdlib". I think Araq is right to try to re-purpose the discussion into "how to best expand the stdlib" -- a discussion which will eventually need to be had over and over again. I don't know if he succeeded. :-)

[c-blake]

Well, I went ahead and added that requested feature of hidden/suppressed help rows over in https://github.com/c-blake/cligen/commit/0cd1c5eb285e350a2d7224d10011b12660baabfa . I agree that is common enough in other CLIs to support. One more feature to explain, of course.

[FedericoCeratto]

People are still complaining about the lack of better parseopt features in the stdlib: https://news.ycombinator.com/item?id=22401953

[c-blake]

I think the setup people were leaning toward is a distribution where certain popular packages might be "bundled" with Nim. For example, import cligen might work without people having to first learn nimble/be online/git clone& hack paths in nim.cfg/config.nims, but such packages also retain a life of their own on github/wherever. That seems to strike a good balance - making it easier for the Nim core devs to not be on the hook for maintaining these packages, retaining some package controlled primary documentation entry point, while also boosting visibility to newcomers and easing concerns about dependency availability.

If it matters, I'm planning on stamping cligen as 1.0 sometime in the next month or two unless some feature request derails that plan. It's been stable/backward compatible for a couple years now. Might still not be @FedericoCeratto's "ideal" amount of "high levelness", but it's also pretty hard to know how to please a very broad potential community about that { or most things, really :-) }.

[c-blake]

This issue thread should be cross-linked to the RFC discussion: https://github.com/nim-lang/RFCs/issues/173