Powershell: DashDash token causing remaining parameter tokens to be flagged as CommandName

[msftrncs]

Apparently there is some special meaning to the DashDash token when it is used in command mode, but I couldn't seem to uncover it. While trying to understand the tokenizer and parser I ran in to these two issues:

1. A DashDash token causes further Parameter tokens to be flagged as CommandName, which can be confusing with PSReadLine if PSReadLine if changed to prioritize the highlighting of CommandName over Parameter (https://github.com/PowerShell/PSReadLine/pull/989).
2. A DashDash token can be accepted as a command, when used with the invoke operators, but it does not get flagged with CommandName, and so causes PSReadLine to highlight it as a operator. Even when it is accepted as a command, the behavior in point 1 still occurs.

Steps to reproduce

& -- -NowACommandNameFlaggedToken

[bpayette]

-- signals the end of parameter processing for a command. Any remaining tokens are parsed as arguments. PowerShell adopted this from the Unix shells we used as our models: What does “--” double-dash mean? There is another form of this --% that turns off all processing for the rest of the command: About Parsing

[vexx32]

Right, but then shouldn't all the arguments be flagged as _arguments_ and not as additional command names during tokenizing?

[BrucePay]

@vexx32 Since @msftrncs seemed unclear, I just wanted to point out that there is, in fact, a very specific meaning to -- (and --%).

[msftrncs]

I could tell from the code that the -- (DashDash) caused any further 'parameters' to be considered arguments (in the AST), but still find it surprising that they are flagged as 'CommandName'. Without modification, PSReadLine will continue to highlight them as parameters, but if changed to prioritize CommandName flagged tokens highlighting as command, it would be alarming to a user.

I also think its an error, but maybe the 'nix shells do the same thing, that the -- (DashDash) can be accepted as a command name at the same time it triggers further parameters to be ignored. This would be similar to a redirect being accepted both as a command name, and redirection at the same time.

I did test changes to the parser to change this behavior. I had to add a couple when clauses to a couple of switch case statements to cause the 'command' -- to not trigger this behavior and to be properly marked as a command name, when the context of the parser was 'CommandName' or 'CommandNameAfterInvocationOperator'. I haven't determined what the correct change would be for the parameter's being marked as command names though.

There also still seems to be no documentation on the topic, as it relates to PowerShell. Is it a recent addition, or in there from day one? Looking at the code, I'd be willing to bet its relatively recent addition and it was added hastily on the end of a section that normally found parameters that were meant to be command names … unfortunately I cannot trace the blame (it was a new file when the code first appeared, so its been here since 6.0, which I can prove it was in Windows PowerShell 5.1).

Only showing the section of importance:
https://github.com/PowerShell/PowerShell/blob/5974afaa8daf9ed6fc45077eff899088a650e1e2/src/System.Management.Automation/engine/parser/Parser.cs#L6281-L6289

I am thinking the fix here it to use a when clause for !sawDashDash on the case statement, it seems to do the trick.

[msftrncs]

There happens to be some inconvienence from the -- (DashDash) token stopping parameter processing. While one might assume that all remaining arguments will now be processed according to standard argument rules, they are not. Parameter tokens are never expandable, even when they are treated as arguments past the -- (DashDash) token. (this also becomes true of a parameter token that is consumed as a command name by an invocation operator)

(echo used for example only)

# these two are not equivelent
echo -- -hello$a
echo "-hello$a"

# however this quickly produces desired result
echo `-hello$a

[msftrncs]

I've investigated Bash on Ubuntu. Bash accepts -- as a command, and does not treat it as the 'stop processing parameters' for itself. However, Bash's dot-source operator . does allow parameters to itself before a filename is specified, so it allows the -- token to 'stop processing parameters' but it only applies to the parameters being applied to the . operator, the command itself that is specified must be further specified to stop processing parameters. Since PowerShell's . operator doesn't accept any parameters, the behavior for -- as a command name should not trigger the 'stop processing parameters' logic.

In Bash

. -- -- -- --help
#< dot source operator
# ^^ stop processing parameters for dot source
#    ^^ command name
#       ^^ stop processing parameters for command
#           ^^^^^^ argument value

Because I originally brought this issue up for the purpose of PSReadLine and highlighting of commands, I could commit a demonstration of changes for this issue in https://github.com/PowerShell/PowerShell/pull/10295.

[msftrncs]

I also see this happening for the --% verbatim argument syntax.

& --% this is now one large argument verbatim

where --% is actually a command name, but the rest of the command line is one single verbatim argument.