Runtime: Potential for conflict between runtime args and app args

[nguerrera]

When running an app with the muxer, you pass runtime args like --roll-forward before the dll like so:

dotnet --roll-forward major myapp.dll myapparg1 ... myappargN

The dll between the runtime args and the app args removed any ambiguity. However, when running an app via apphost, runtime args are passed liked so:

myapp.exe --roll-forward major myapparg1 .. myappargN

Arguments are consumed by host while recognized, and the rest go to the app. Unforunately, this is ambiguous: what if myapp's Main(string[] args) expects to handle --roll-forward with its own app-specific meaning?

It has been brought to my attention that we have already shipped this way, so it would be breaking to change the scheme for all arguments that are supported in past versions. If we can't break, I would still like this issue to cover the long-term case of adding new options. The current scheme means that every new runtime option allowed in this command line position is technically a breaking change.

When I raised this elsewhere, the analogy to the -- delimiter used in many places came up, but I still don't think this would solve it as the app may very well have its own interpretation of --. IMHO, what we use here should not occupy nice-looking, common real estate on the command line. To the extent that we try to make it look nice, we increase the risk of colliding with the apps.

If we were able to start from scratch, I would suggest a scheme like

myapp.exe --$runtime::roll-forward

And we say that the --$runtime:: prefix is reserved. I don't mean to have a strong opinion for this exact syntax, just trying to illustrate the idea of something that is obscure enough to not conflict with common patterns and provides a sort of namespace where more runtime options can go without risk.

[nguerrera]

From dotnet/cli#11852

grep has a scheme for separating one form of command-line input from another with --. That's what the man page doc says that I linked to. It is similar to the -- as supported by dotnet run. If we want to make the separation formal, we need a similar form of delimiter.

I think the chance of colliding with an application switch would be signficantly reduced if we called it --fx-roll-forward.

I agree in theory, however, that devolves into "make the runtime switches more obscure and less usable." That seems like a bad tradeoff and evidence of a missing capability.

Also, the runtime switches are intended to map directly to runtime config names, which doesn't require this kind of namespacing. It doesn't make sense to change our config naming scheme because of a weak commandline model.

I don't think a delimiter solves the problem in general. Suppose we choose our own delimiter. Say '@@` is the delimiter we choose.

T=1: I write app app that takes --potato arg
T=2: Host adds --potato arg
T=3: app --potato is broken and must be changed to app @@ --potato

So I think the scheme should apply to each arg and not be based on a delimiter.

[richlander]

I would expect it to work like this:

myapp --roll-forward major --potato @@ --potato --myapparg1 foo

The host would scan until it sees @@. If it doesn't, it doesn't look at any of arguments string. If it does, it processes/eats everything up until and including @@ and passes the rest to the app. This is similar to other schemes, right?

[nguerrera]

Oh, I see. Only if delimiter is present, then the args before it go to runtime. Yeah, I think that could work. The delimiter does have to be unique / obscure, though. I was thinking the other way for some reason.

[richlander]

My proposal is a breaking change on existing behavior. It can be added in a non-breaking way (only when '@@' is present), but that's kinda bad.

I can also see some apps saying "please don't scan the damn arguments every launch; I'm never going to tell you anything" so I'd want to way to tell the host to not scan arguments at all. I can imagine this for web apps as opposed to tools. In fact, we might make this whole behavior opt-in, which would avoid it being breaking. It would also make the delimiter choice less of a hard choice in that case.

[nguerrera]

This is similar to other schemes, right?

I think it's slightly different in a way that explains how I misinterpreted the proposal.

A common way -- is used is to indicate that all arguments after it are file arguments, not switches.

$ touch -foo
touch: invalid option -- 'o'
Try 'touch --help' for more information.

$ touch -- -foo

So it changes the meaning of things after it, not things before it in some sense. Anyway, I don't think this matters, just trying to see how I missed this.

[richlander]

I think I understood the existing semantics of the typical -- scheme less well then you, so I was less confused by it ;)

[KathleenDollard]

@jonsequitor looked at the issue of orthogonal information in the context of directives for System.CommandLine.

A lot of characters we might want to se already have meaning in some context or another.

While the @@ syntax above looks like it would work, I'm concerned about readability. I like the namespace idea early on:

myapp.exe --$runtime::roll-forward

if there is no issue with it. But I also keep coming back to the idea of orthogonal switches being set off with clear orthogonal surrounding delimiters (square brackets only because Jon found them available):

myapp [roll-forward major] [potato] @@ --potato --myapparg1 foo

or

myapp [--roll-forward major --potato] @@ --potato --myapparg1 foo

However, I think this whole conversation would go a lot better if you had adopted the standard unknown switch name of banana rather than switching the switch to potato!

[richlander]

One of the challenges I have with this is that there are multiple levels of this. It's a bit like the old saying "one man's garbage is another's treasure" ...

We're talking about the host/apphost having its own private switches. It's possible that someone could build some kind of managed host for hosting plugins that has the same need. In fact, the CLI itself is basically that.

I don't see how the square brackets help with this problem, particularly if they are paired with an additional delimiter (the @@). To my eyes, the square brackets just makes the syntax more complicated and foreign. We also don't need any of this in the dotnet foo.dll case, such that analogous syntax for that scenario would either be different (because it wasn't needed) or overkill (because we made it match the apphost case).

I'd like to turn this around and ask "what's the simplest workable solution"? Here are my requirements:

• Enables a consistent model for the host, apphost, the CLI, and apps/tools.
• Works the same for local (I still don't understand how this is going to work for that case, since they are dotnet hosted) and global tools.
• The CLI already has a model with -- so we need to rationalize about that.
• Enables existing tools with similar concepts to be written with no change in syntax (like grep and touch). This is the same as the CLI case.
• And last but not least, passes usability testing. This use case is pretty critical.

[vitek-karas]

I personally like the proposal from @nguerrera - with the namespaces... The exact syntax is TBD, but I think this would work nicely. In next version we would support all the options via the new syntax, and the existing ones via the old one as well - there's probably no good way to retire the old syntax (muxer is for all versions, so we can't say that in 6 we won't support it)...

[vitek-karas]

/cc @lpereira who might have some experience with Linux based tools and how they handle similar situations.

[KathleenDollard]

@jonsequitur for Posix/Linux experience

If we can find a syntax that works with namespaces, I could be OK with that. I do not like it because. I do not believe the users should have to understand who needs their meta request, I believe many of these orthogonal issues should be available to all meta code. By meta code, I mean all code that is not the code of the thing being called.

I feel the namespace idea takes things a step further than necessary and puts a burden on users to understand how this necessary trivia will be piped.

Is there any prior art on namespace like approaches?

My understanding of Posix is that

myapp.exe --$runtime::roll-forward

parses as a --$runtime switch with an argument of :roll-forward

[nguerrera]

IIRC, posix only defines getopt() which only supports single character options. The long-form derivatives like GNU getopt_long don’t seem to conform to a single standard.

I’m more accustomed to seeing --name=value than --name:value in the long form.

[nguerrera]

Like I said, I didn’t mean to propose a precise syntax, just the general idea of something that is namespace-like. I arbitrarily borrowed from C++ namespace syntax and threw in the $ to try to make it more likely to be unique. I realize now that $ does not work as it will get interpreted by shells as a variable reference.

But I think we should decide if we want the namespace concept before worrying about its precise syntax in terms of conflicts like that.

[nguerrera]

I was also deliberately trying to avoid “prior art” as that increases the likelihood of conflict with the app.

[jonsequitur]

This issue is one example among a set of similar cases so I'd like to provide some background on the square bracket convention.

The use case (but not the syntax, initially) emerged from seeing many examples of command line gestures that were orthogonal to a tool's core functionality, including:

• Specifying versioning if you want a tool whose syntax or behavior has changed over time
• Setting output formats (VT-100, non-VT-100, JSON, etc.)
• Meta-commands for debugging, parse diagramming, completions

These are typically different from the tool's core behavior and we wanted to signal the difference visually, so users could understand that the behavior was not only different from the tool's main functionality, but also that it might apply across tools. A good analogy is HTTP headers.

From there, the square bracket syntax is what passed through various filters, avoiding characters with shell-specific meanings (!, @, $) and seeming to work well across different shells (bash, PowerShell) and command line styles (POSIX vs Windows).

[nguerrera]

Our goals are very similar here. One challenge is layering two levels of these. Suppose you have one set of things for the host, another as meta commands for System.CommandLine, followed by the app args. You need for the three sets to be disambiguated from each other. So while we would be in search of similar characteristics, it seems that we can't just use what you chose. This is why I was drawn to some form of namespacing over delimiters.

[nguerrera]

Perhaps we can combine namespacing and square brackets. We say that in the square bracket world of meta options, there are namespaces. This is our new real estate to specify, right?

One thing to consider is that we should keep the host parsing and stripping of its options simple, though. It starts to get a lot more complicated if it's strip everything from square brackets with our namespace, and remove square brackets entirely if nothing remains, etc.

[jonsequitur]

I think both namespacing and a generalized directive syntax are valuable and complementary.

Exact string match is the simplest thing for the host as I understand it, versus needing a full parser. The square-bracketed directives syntax is deliberately constrained in a way that I think works well here. It is (per the current System.CommandLine implementation, but there's room to change this if it makes namespacing easier):

• Directives must precede all other command line tokens.
• A key and a value within the directive can be delimited by a :.
• A value is optional; no : is required.
• No spaces can occur between the key and value; only a : is allowed. This avoids escaping.
• If there's a second : it's part of the value.