Powershell: Operators tokens remain flagged as operators after TokenKind is changed to Generic in Command Mode

[msftrncs]

While trying to understand why PSReadLine highlights operator tokens in command mode as operators (though they are not in that context) , I determined that the parser/tokenizer (Tokens Class) has a method that specifically resets the TokenKind of a token, to 'Generic'. However it fails to remove the flags originally set from the original TokenKind. This most commonly occurs to the operators, which in command mode, are not operators and instead are treated as unquoted arguments. PSReadLine uses the TokenFlags UnaryOperator, BinaryOperator and AssignmentOperator to highlight operators, and so that is what it sees

Steps to reproduce

echo 1 + 2 + 3 (1 + 2 + 3)
#      ^   ^ not operators
#                 ^   ^ operators!

[lzybkr]

I removed the Area-PSReadLine tag. The issue can only be worked around in PSReadLine, a proper fix in the PowerShell parser might help other tools as well.

[vexx32]

Is there any behaviour that depends on the retention of the original TokenFlags values? It seems odd not to reset the flags when changing the TokenKind. 🤔

If not, it doesn't seem like a particularly complex fix.

[lzybkr]

Clearing the "operator" token flags when changing the token kind to Generic feels safe to me.