Pydantic: Command-line argument parsing

[dmontagu]

Feature Request

I would really like a lightweight CLI-argument parsing class, similar to BaseSettings.

---

These days I always find myself using BaseSettings whenever I need to read something from the environment. Using this class is so convenient that I have started to eschew the use of arguments for lightweight scripts and instead just reading values from the environment (and using pydantic to parse them).

Taking a step back, though, it seems like it might be nice to just have a built-in class like BaseSettings for parsing command line arguments, maybe BaseCLIArguments? I always find myself struggling to remember the syntax for argparse; I personally think this is a great use case for pydantic.

I know there are lots of other libraries that handle command line argument parsing, but given how little effort I think it would take to integrate this with pydantic, I would much prefer to not need to learn (and remember) yet another tool.

I wouldn't mind if it were highly opinionated in the interest of simplicity; I just want an easy way to get parsed command line arguments using patterns I already use daily (namely, BaseModel).

---

(Either way I'll probably implement this for myself, but I will put more care into it if there is any interest in upstreaming it. Hence the issue.)

[samuelcolvin]

I'd never thought about this but it sounds like a great idea.

A few thoughts in no particular order:

• we can start small and expand, we don't need all the things I'll suggest here or to replicate click to make this useful.
• It would be very helpful if pydantic's CLI support could auto generate --help output as other libraries do.
• I think it could just be a Config flag on BaseSettings, most of the scenarios where you'd want to use BaseSettings you might also want CLI support. Or perhaps a new method eg. BaseSettings.parse_cli()?
• How do we deal with the distinction between arguments and options? Perhaps through a flag on Field() (currently aka Schema()) eg = Field(... arg=True), argument order would then be prescribed by field order. Is there are more elegant way than using Field(arg=...)?
• I guess List[...] fields can be populated with repeated arguments, eg. --foo green --foo red, lots of CLIs do this but I can't think of one at the moment, maybe not many do actually. Probably not required for the first release of this feature?
• I guess Dict[str, ...] fields can be populated in a similar way to docker run's -e. Probably not required for the first release of this feature?
• I think sub-commands might be cool (perhaps using multiple models?), but definitely not required for the first release of this feature?

Since this is just a new feature and shouldn't have any backwards compatibility problems, I think we should delay it until after v1?

[koxudaxi]

I want to use the feature. I think CLI argument parsing match pydantic absolutely.
Pydantic has a lot of functions for CLI parser. (eg. enum, parsing, type-check, required)

I write my thoughts.

• How do we define an alias? alias has been already used in the argument of Field ( alias is -h for --help)
• Where did we write descriptions for command and arguments? (eg. Field (..., help='a help message'), Field (..., description='a help message'))
• Should this feature support Color? I know the option is not required. And pydantic may need an extra dependency to support it. However, The Color make the feature great.

[samuelcolvin]

• How do we define an alias? alias has been already used in the argument of Field ( alias is -h for --help)

see https://github.com/samuelcolvin/pydantic/issues/721#issuecomment-521191005, aliases should use cli similar to how environment variables will use env

• Where did we write descriptions for command and arguments? (eg. Field (..., help='a help message'), Field (..., description='a help message'))

should use description on field, same as schema does now.

• Should this feature support Color? I know the option is not required. And pydantic may need an extra dependency to support it. However, The Color make the feature great.

You mean ansi colours? If so, I built this when developing python-devtools, it's not that hard at least for unix platforms. We could either make devtools an optional dependency or copy some of that logic. However I don't think it's required for the first release, also I'm not convinced coloured error messages or help would ever be that useful - please suggest a CLI that makes helpful use of colours in these scenarios?

[koxudaxi]

Sorry, I never have read the comment.

Yes, ANSI colors.
I agree that it's not required for the first release.
A few commands have sub-command, options, description, default in help command.
The help command makes a console screen is filled a lot of words.
If CLI draws words with colors, I think it's more readable for us.
But, I don't know another user need my suggestion.

serverless command looks good to me.

[euri10]

beets makes a useful use of colors as well

Le sam. 17 août 2019 à 6:58 PM, Koudai Aono [e-mail] a
écrit :

Sorry, I never have read the comment.

Yes, ANSI colors.
I agree that it's not required for the first release.
A few commands have sub-command, options, description, default as help
command.
The help command makes a console screen is filled a lot of words.
If CLI draws words with colors, I think it's more readable for us.
But, I don't know another user need my suggestion.

serverless command looks good to me.
[image: ss2019-08-18 2 52 03]
https://user-images.githubusercontent.com/630670/63215570-775cbc00-c163-11e9-8cee-df0658deecd4.png

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/samuelcolvin/pydantic/issues/756?email_source=notifications&email_token=AAINSPWL6B4MHLCWWSJRGWDQFA333A5CNFSM4IMLZCTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4QQQWQ#issuecomment-522258522,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAINSPUQYFVVRPYVQKFU3MTQFA333ANCNFSM4IMLZCTA
.

[dmontagu]

A few thoughts in no particular order:

• we can start small and expand, we don't need all the things I'll suggest here or to replicate click to make this useful.
• It would be very helpful if pydantic's CLI support could auto generate --help output as other libraries do.

Agreed with the above.

• I think it could just be a Config flag on BaseSettings, most of the scenarios where you'd want to use BaseSettings you might also want CLI support. Or perhaps a new method eg. BaseSettings.parse_cli()?

After some thought, I'm leaning toward a classmethod for this, rather than config. I think it could be nice to be able to choose to from either the environment or the command line (without needing to change the class definition), and I think a method would make it easier to do that flexibly. Also, I think it would be easier to have cli-parsing-specific settings as arguments to the method (rather than extra config settings), such as a bool fallback_to_env which could be used to allow/disallow falling back on values from the environment if not provided.

How do we deal with the distinction between arguments and options? Perhaps through a flag on Field() (currently aka Schema()) eg = Field(... arg=True), argument order would then be prescribed by field order. Is there are more elegant way than using Field(arg=...)?

Another option would be a custom type FlagArgument that parses to a bool in the same way as StrictBool (or the ill-fated RelaxedBool), but signals to the config that it should be read from command line as a flag rather than an argument with a value. Given how ubiquitous Field is, I'm less inclined to add another argument to Field for this very-specific use case, but my opinion isn't too strong on this point.

I guess List[...] fields can be populated with repeated arguments, eg. --foo green --foo red, lots of CLIs do this but I can't think of one at the moment, maybe not many do actually. Probably not required for the first release of this feature?

I think this makes sense.

I guess Dict[str, ...] fields can be populated in a similar way to docker run's -e. Probably not required for the first release of this feature?

I'd be fine with either --dict-arg key1=value1,key2=value2, or --dict-arg key1=value1 --dict-arg key2=value2

I think sub-commands might be cool (perhaps using multiple models?), but definitely not required for the first release of this feature?

Agreed this could be nice; I think there is a lot of room for cool APIs here. I think this could be supported by extra custom types later.

Since this is just a new feature and shouldn't have any backwards compatibility problems, I think we should delay it until after v1?

Agreed.

[enkoder]

I would love something like this as well! One request is I have is that the API should be able to take in a string or list of strings instead of assuming that this will be done only as a command line parser (sys.argv). My use case is for chat bots where a command handler would be wrapped by a decorator that parses and passes the given object to the handler.

Something like this !kick 123123 --silent

class KickUserArgs(BaseModel):
    id: int
    silent: FlagArgument()

@cli_parser(KickUserArgs) 
def kick_user_command(ctx, args):
    assert isinstance(args, KickUserArgs)

[dmontagu]

@samuelcolvin I've spent some more time thinking about this and I think I'm less interested in

I've actually been using @tiangolo's awesome typer package (which is basically just click with much better static typing, and, in my opinion at least, an improved API) and have found it to ergonomically capture the vast majority of my practical needs working with a CLI.

It doesn't offer the full power of pydantic for parsing, but to the extent that that is even necessary (basic types seem to get you pretty far with CLIs), I think we might be better served by adding pydantic integration there rather than adding CLI integration here.

I'm going to close this issue but I'm happy to reopen it if there is still interest.

[samuelcolvin]

Agreed.

I hadn't seen typer, looks great.

Perhaps worth creating an issue there and linking to this to at least discuss pydantic integration? Perhaps using #1179?

[tiangolo]

I'm glad you're liking Typer!

I came just to note that Typer's API is clearly inspired by Pydantic, as FastAPI was inspired (and powered) by Pydantic.

[aschreyer]

First of all thanks for the two great libraries! I have been starting to use both for CLI parsing and configuration management. I think it would be great to have Pydantic integration in Typer given that the former already has all the parsing and validation functionality needed (including JSON parsing). In fact, using

settings = Settings(**context.params)  # BaseSettings instance

already works with CLI examples I have been testing. However, there is redundancy in the type definitions of course.

[mpkocher]

@dmontagu I'm believe I had a similar need for a thin layer that enables Pydantic data models to be loaded from command line args or from JSON files.

This resulted in pydantic-cli with several examples here and is perhaps similar in spirit to Typer.

Here's a simple example:

from pydantic import BaseModel
from pydantic_cli import run_and_exit

class Options(BaseModel):
    input_file: str
    max_records: int


def example_runner(opts: Options) -> int:
    print(f"Mock example running with {opts}")
    return 0


if __name__ == "__main__":
    run_and_exit(Options, example_runner, description=__doc__, version="0.1.0")

[gabe-microsoft]

@mpkocher pydantic-cli is exactly what my group has been looking for!

We've been interested in using Pydantic, but have a common pattern of using a combination of JSON files and command line args to define settings for scripts (sometimes with the addition of environment vars). I looked into using Pydantic+Typer, but this still requires duplicating the model specification, has some bugs when using non-basic types, and it gets a bit messy to load settings from both JSON and command line (where each may be partial and only the merged results should be checked).

I really like the rich types defined in Pydantic and look forward to seeing pydantic-cli extended to include all types supported by pydantic. I think it would make the most sense for this CLI generation functionality, including JSON settings parsing to be included directly in pydantic.

@samuelcolvin is there any plan to add this feature? I was surprised that this feature request was closed since there seemed to be community interest in the feature and there still remains no complete solution...

[samuelcolvin]

I haven't tried pydantic-cli yet. I will.

I closed this because of typer, but having used typer I find it annoying - mostly because it's a wrapper of click and limitations caused by that.

I'd love a cli implementation based on pydantic but I'm struggling to provide support for pydantic as it is, another massive feature like CLI support doesn't seem sensible within this repo therefore.

[gabe-microsoft]

@samuelcolvin Thanks for the update. I empathize with the constraints on your time and really appreciate all of the time and effort that you've been putting into pydantic.

I tried out pydantic-cli yesterday and feel that it does a good job of adding CLI support to simple pydantic models. I'll think more about how to extend it further to support more complex pydantic models and use cases, and will work with @mpkocher to improve pydantic-cli to be a complete CLI solution for pydantic models.

[xkortex]

I started a thread over at https://github.com/mpkocher/pydantic-cli/issues/23 but I figured I'd mention it here as well.

It's a super rough PoC just to see if the concept - leveraging a bit of metaprogramming and introspection - has any traction. Theoretically, using a custom type hint that accepts the metadata needed to populate CLI options (argparse.Action in this case) works, I just need to massage the data flow a bit more and remove the redundancy which can be inferred from inspection or the Pydantic schema.

If people think this is a cool approach, I'll keep hacking on it. I actually don't think it would take much code to get CLI functionality into pydantic.