Powershell: PSParser Tokenize with Switch Default fails

[LAltinell]

Trouble when trying to Tokenize a script containing a Switch-statement with a Default case in Powershell 7.1.0-preview.5. This works as expected in Powershell 7.0.3 (stable). If one does comment-out the default case, it works as expected. This has been reproduced on more than one computer.

Steps to reproduce

Create a file containing a simple function that has a switch-statement with a default case (see below). Save that file. Then try to tokenize that file (also see below) from a console running Powershell 7.1.0-preview.5.

# Save this as a file called 'TEST.ps1'
Function Test {
    Param (
        $String
    )

    Switch ($String) {
        "test" {
            "sfdsf"
        }

        "test2" {
            "sfdsf"
        }

        default {
            "sdfsdf"
        }
    }
}

# From a Powershell 7.1.0-preview.5 prompt execute the following:
$scripts = Get-ChildItem -Path "C:\__TEMP\TEST.ps1"

Foreach ($file in $scripts) {
    $contents = Get-Content -Path $file.fullname -ErrorAction Stop
    $errors = $null
    $null = [System.Management.Automation.PSParser]::Tokenize($contents, [ref]$errors)
    $errors.Count -eq 0
}

Expected behavior

True

Actual behavior

MethodInvocationException:
Line |
   4 |      $null = [System.Management.Automation.PSParser]::Tokenize($conten …
     |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Exception calling "Tokenize" with "2" argument(s): "Index was outside the bounds of the array."
True

Environment data

Name                           Value
----                           -----
PSVersion                      7.1.0-preview.5
PSEdition                      Core
GitCommitId                    7.1.0-preview.5
OS                             Microsoft Windows 10.0.19041
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[vexx32]

PSParser is legacy and I'm not sure it's actually supported in a meaningful way.

Instead, you can use:

$tokens = $parseErrors = $null
$ast = [System.Management.Automation.Language.Parser]::ParseInput($script, [ref]$tokens, [ref]$parseErrors)

My guess is the PSParser code path was probably broken with #10487 cc @SteveL-MSFT @rjmholt not sure PSParser is really supported / what would need to be done to "fix" it following that change, or whether it's worth looking into as there are fully functional and supported alternatives available.

[LAltinell]

Thank you for your prompt reply!

Perfect, that works for me. Hopefully this will help someone else as well.

[rjmholt]

This sounds like something we should still fix though -- the PSParser API needs to be maintained for broad module compatibility

[rjmholt]

Full stack trace:

Exception             :
    Type           : System.Management.Automation.MethodInvocationException
    ErrorRecord    :
        Exception             :
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Exception calling "Tokenize" with "2" argument(s): "Index was outside the
bounds of the array."
            HResult : -2146233087
        CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
        FullyQualifiedErrorId : IndexOutOfRangeException
        InvocationInfo        :
            ScriptLineNumber : 1
            OffsetInLine     : 1
            HistoryId        : -1
            Line             : [System.Management.Automation.PSParser]::Tokenize($s, [ref]$errs)
            PositionMessage  : At line:1 char:1
                               + [System.Management.Automation.PSParser]::Tokenize($s, [ref]$errs)
                               + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            CommandOrigin    : Internal
        ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
    TargetSite     :
        Name          : ConvertToMethodInvocationException
        DeclaringType : System.Management.Automation.ExceptionHandlingOps,
System.Management.Automation, Version=7.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
        MemberType    : Method
        Module        : System.Management.Automation.dll
    StackTrace     :
   at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception
exception, Type typeToThrow, String methodName, Int32 numArgs, MemberInfo memberInfo)
   at CallSite.Target(Closure , CallSite , Type , Object , PSReference )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1
arg1, T2 arg2)
   at System.Management.Automation.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame
frame)
    Message        : Exception calling "Tokenize" with "2" argument(s): "Index was outside the
bounds of the array."
    Data           : System.Collections.ListDictionaryInternal
    InnerException :
        Type       : System.IndexOutOfRangeException
        TargetSite :
            Name          : GetPSTokenType
            DeclaringType : System.Management.Automation.PSToken
            MemberType    : Method
            Module        : System.Management.Automation.dll
        StackTrace :
   at System.Management.Automation.PSToken.GetPSTokenType(Token token)
   at System.Management.Automation.PSToken..ctor(Token token)
   at System.Management.Automation.PSParser.get_Tokens()
   at System.Management.Automation.PSParser.Tokenize(String script, Collection`1& errors)
   at CallSite.Target(Closure , CallSite , Type , Object , PSReference )
        Message    : Index was outside the bounds of the array.
        Source     : System.Management.Automation
        HResult    : -2146233080
    Source         : System.Management.Automation
    HResult        : -2146233087
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : IndexOutOfRangeException
InvocationInfo        :
    ScriptLineNumber : 1
    OffsetInLine     : 1
    HistoryId        : -1
    Line             : [System.Management.Automation.PSParser]::Tokenize($s, [ref]$errs)
    PositionMessage  : At line:1 char:1
                       + [System.Management.Automation.PSParser]::Tokenize($s, [ref]$errs)
                       + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1

[rjmholt]

The problem is that this mapping hasn't been maintained:

https://github.com/PowerShell/PowerShell/blob/4b9b0788ed28ea6d463ce857d1ed81bd4a977a59/src/System.Management.Automation/engine/lang/interface/PSToken.cs#L110-L307

That's ostensibly easy to fix, but:

• We need testing for it
• We need to work out what else might be missing. I notice for example that base doesn't seem to be in there
• There may be tokens that we could return but we can't for compatibility reasons -- something I'm not sure how to handle (haven't thought about this properly). For example, how should a caller built to handle PS v3 code deal with the class keyword?

[rjmholt]

/cc @daxian-dbw

[daxian-dbw]

@rjmholt Is this specific regression caused by https://github.com/PowerShell/PowerShell/pull/10487?

[rjmholt]

@rjmholt Is this specific regression caused by #10487?

I believe so

[daxian-dbw]

base seems to be working fine:

PS:162> $es = $null; [System.Management.Automation.PSParser]::Tokenize("class foo { foo() : base() }", [ref]$es)
...
Content     : base
Type        : Unknown
Start       : 20
Length      : 4
StartLine   : 1
StartColumn : 21
EndLine     : 1
EndColumn   : 25
...

[Update] I took a look at the code, and the keyword base works just by accident.

[rjmholt]

Type : Unknown

That looks like this:

https://github.com/PowerShell/PowerShell/blob/4b9b0788ed28ea6d463ce857d1ed81bd4a977a59/src/System.Management.Automation/engine/lang/interface/PSToken.cs#L306

[daxian-dbw]

Yeah. Let's first fix this regression by simply updating s_tokenKindMapping. And also open a new issue to discuss how to deal with PSParser in the long term to keep it up-to-date.

[msftbot[bot]]

:tada:This issue was addressed in #13779, which has now been successfully released as v7.1.0-rc.2.:tada:

Handy links:

• Release Notes