Powershell: Enhancement: Ability to invoke generic methods with explicit type parameters when PowerShell cannot figure out <T> from the context

[0xfeeddeadbeef]

Please provide ability to invoke generic methods with explicit type parameters.

Steps to demonstrate

[Array]::Empty[string]()

or using a class:

class A {
    static [string] GetTypeName[T]() { return [T].Name }
}

[A]::GetTypeName[string]()

(Array.Empty was chosen as a simplest possible example. In scripts Iʼd use @())

Current behavior

At line:1 char:16
+ [Array]::Empty[string]()
+                ~
Array index expression is missing or not valid.
At line:1 char:16
+ [Array]::Empty[string]()
+                ~~~~~~~
Unexpected token 'string]' in expression or statement.
At line:1 char:24
+ [Array]::Empty[string]()
+                        ~
An expression was expected after '('.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingArrayIndexExpression

Ruminations

I understand that itʼs not important to have full support of generics in a scripting language or to have 100% feature parity with C# (I might be wrong, — now we have classes in PowerShell after all :thinking:)

Here are some _not-very-elegant_ workarounds from the Interwebs:

• Workaround to call to a generic method OfType() in PowerShell, when PowerShell cannot figure out from the context by @vors
• PSGenericMethods module by @dlwyatt
• Invoking Generic Methods on Non-Generic Classes in PowerShell by @LeeHolmes
• How do I call a parameterless generic method from Powershell v3?

Here are my questions:

• Is it possible to implement this at all? How hard it could be?
• Anyone tried to implement this before?
• Is it worthwhile?

Just posting this here for discussion. Didnʼt find any mentions of this problem in other issues.

[iSazonov]

At first glance, this is not worthwhile because the main asset of PowerShell is to hide complex details - Powershell automatically calls the correct methods.

As for implementation it seems is a lot of work but not so difficult.

[0xfeeddeadbeef]

@iSazonov Problem is that PowerShell can not automatically call correct methods when there is no way to deduce generic type parameter automatically (as seen in "Steps to demonstrate"). It must be specified explicitly. This is true for other CLR languages too.
We are forced to dig through all the complex details using reflection (see list of workarounds).

[iSazonov]

I mean that the implementation of the general case is likely to be redundant. On the other hand, if there are specific popular scenarios, maybe we could implement them natively in PowerShell.
We also have to take into account that PowerShell performs many implicit type conversions - we may encounter insurmountable hurdles.

[lzybkr]

This is totally a reasonable thing to add, and a perfect example of why I'm excited PowerShell is open source.

People do want this feature, but as you point out, it's not critical for a shell. With limited bandwidth, this feature was never quite important enough for a small team to implement, but that's not an issue anymore - there are plenty of motivated and capable people who might add this feature now.

For anyone willing to tackle this:

• I think the proposed (and obvious) syntax should work, but you must be careful to not break array references, e.g.

$t = [System.Collections.Generic.Dictionary[string,int]]
$t.GenericTypeArguments[0]

When you first see the [, you can't know if you are expecting an expression or a type argument list.

• We will need a new property in MemberAst. It might not be obvious why here and not InvokeMemberAst, but sometimes you refer to a method without arguments so you can call it later, e.g.

$op = "hello".PadLeft
$op.Invoke(15)

It's probably not important to support the equivalent for generic methods in the initial implementation, but it is important that the representation support it should we want that in the future.

• This type probably needs a new property to hold the specified type arguments, which would get added from the Ast here.

Hopefully this is enough to get someone started, if more help is needed, just ask.

[dlwyatt]

I'll take a look when I have some time. It'll certainly be easier than writing PSGenericMethods was, since I can just use the type conversion / binding code / overload identification that's already there instead of having to reinvent it.

[dlwyatt]

@lzybkr , how would you feel about this syntax for the PSMethod reference option:

$op = "Hello".SomeGenericMethod
$op.InvokeGeneric([someType], $arg1, $arg2)

This makes it easier to see what's happening at parse-time, and we only need to modify InvokeMemberExpressionAst to support it. If we modify MemberExpressionAst instead (allowing for $op = "Hello".SomeGenericMethod[someType]), it's tricker to distinguish between a property followed by an array index versus a method with a generic type. By looking for MethodName[type](args), it's not ambiguous (parser currently will complain about the open parenthesis).

[lzybkr]

Because a pipeline isn't allowed as the index expression, I don't think there is any ambiguity - one token of look ahead is sufficient - if could be a type argument list, it is, otherwise it must be an expression.

If a pipeline was allowed, a bare word is ambiguous - is it a type or an command invocation.

Note that one token look ahead doesn't mean there is only one token to look for, there is at least a bare word and an '[', e.g.

[Array]::Empty[string]()
[Array]::Empty[[string]]()

The extra square brace is there for a corner case where you want to specify the assembly name along with multiple type arguments, e.g.

[System.Collections.Generic.Dictionary[[string, mscorlib], string]]

[dlwyatt]

Hmm... playing devil's advocate for a second, do we want to support variables in the type expression?

$something.MethodName[$typeInAVariable]()

If so, then this could be ambiguous:

$something.MethodName[$typeInAVariable] # Array index or method with generic type params?

[lzybkr]

That seemed useful in V2, but it's been that long since I've seriously considered it.

And honestly, if we wanted to support that, I'd look for a different syntax for explicit type parameters.

[dlwyatt]

ok, cool. I'll be on a plane for 4-5 hours tomorrow morning; if I'm not feeling too groggy (waking up at 4am, bleh), I'll plug away at it a bit.

[dlwyatt]

I might be biting off more than I can chew on this one. I've tweaked the parser and AST to support the syntax, but I'm having trouble wrapping my head around the compiler and binders to see where to make use of the new information from the AST. Will sleep on it.

[dlwyatt]

@lzybkr

Can I pick your brain a bit about this one? Updating the parser was fairly straightforward, and when I look at the Adapter / DotNetAdapter / PSMethod classes, it's very similar to what I wrote in PSGenericMethods (easy to add some new overloads that accept a list of generic type arguments).

Between those, however, I'm getting lost in the Binders / DLR expression tree stuff, trying to figure out the best way to get the new information from the AST over to either DotNetAdapter or PSMethod. Should there be new fields on binder class(es)? Do we need to uniquely identify a binder based on the generic type arguments, or just use one binder instance per method / property name (the way it works now) and somehow pass on the generic type arguments so eventually PSMethod.Invoke can identify the right overload?

While looking into this, I came across another possiblity for syntax:

# Imaginary generic method that takes one T and two other arguments
$psmethod = [SomeClass]::SomeMethod
$psmethod.InvokeGeneric(@([string]), 'Argument 1', 'Argument 2')

(Names don't matter here, just saying that we'll probably be adding new input to PSMethod's Invoke* method(s) anyway, and may not need to support the $method = [SomeClass]::SomeMethod[string] syntax as a result. $emptyStringArray = [array]::Empty[string]() could still be a convenient shortcut for calling it, though. )

[lzybkr]

Look at PSMethodInvocationConstraints - I think that's the right place - just be sure to update the hashing so you don't reuse the same binder in different dynamic sites.

And FYI - I'll be slow to respond over the next week.

[mikeTWC1984]

Was there any change? Or maybe some better option rather than using "MakeGenericMethods"? Generics are widely used in .net assemblies that deal with data, like entity framework or ML.NET, where classes are used to defined data schema. It's extremely tedious to port such methods to PS. I think it would be great to have <> operator in PS too.

[vexx32]

Some allowances for casting operations would be nice as well. For example, if you're casting to [List[int]], the following does not currently work without an intermediate cast:

using namespace System.Collections.Generic

[List[int]]@(1..10)

Currently you must have an intermediary cast of [int[]] before that will go through. It would be nice if we could have the parser / conversion methods check if the target type is, e.g., IEnumerable<T> (where T is the same generic type parameter given to the target type we're casting to), and then automatically attempt to first convert anything that is [object[]] to [T[]] first to ensure the cast will succeed if it is able to.

In short, when given:

[List[int]] 1..10

The parser should infer that [List[int]] requires a cast to [int[]] first and automatically apply the intermediary conversion.

Given #11768, we may also want to special case certain types, since Memory<T> itself isn't IEnumerable<T> but it does contain Span<T> which is, and you can already cast T[] to Memory<T> (I think?).

[mklement0]

@vexx32, [System.Collections.Generic.List[int]] @(1..10) currently works fine, or are you talking about optimizations behind the scenes? Also note that (...) is preferable to @(...), as the latter would enumerate and reassemble an already strongly typed array as an [object[]] array.

Note that enclosure of the input array in () in something like [System.Collections.Generic.List[int]] (1..10) is a _syntactic_ necessity, because the cast operator has higher precedence than the range operator.

you can already cast T[] to Memory (I think?).

You can, but just to spell it out: only if it is the _exact_ target type:

# OK - exact type match; omitting [int[]] or using [long[]] wouldn't work.
[Memory[int]] [int[]] (1..3)

[vexx32]

Huh, I didn't recall casting directly to List working. Odd. Good to know that does work, though.

But yeah, the last one there is the one I'd look to improve specifically, then, perhaps.

[mikeTWC1984]

To continue discussion on #12341, just realized same idea was suggested here. I had this issue in mind, but somehow thought it's about new syntax only.
So speaking of $obj.Method.Invoke - at a glance it's not obvious how it's implemented, but I guess it should use TypeInfo.GetMethod behind the scene. If so, shouldn't we be a step away form getting InvokeGeneric ( basically adding MethodInfo.MakeGenericMethod(types) in the chain)? @vexx32 you mentioned adding such method might need some change in .net core itself, does it mean that PSMethodInfo is using some different mechanism?

[vexx32]

I was thinking the MethodInfo was coming from .NET Core directly, but it looks like we have a PSMethodInfo class I was forgetting about which we're already using. So adding something like your suggestion to it would probably be fairly easy, if you could figure out the necessary code to handle the generic method invocation directly. 🙂

It'd be a public API so at least probably require committee review, but beyond that I don't see anything too difficult about getting that added.

I would _prefer_ personally just having generic methods work correctly in PS and be able to provide type args with a more c#-like syntax directly, but that's still a step in the right direction; when we eventually get to that point, we'll then have an available method to use to make it work directly that we can just reuse.

[dlwyatt]

I never did figure out how to properly get the info from the AST through the binders / etc. The code updates I did for the parser are still in my fork though, if someone wants to pick up where I left off: https://github.com/dlwyatt/PowerShell/commit/74809e0080106926907085d80adea59d6844f070

[lzybkr]

Binders are passed what they need and in general they don't need the AST. PSMethodInvocationConstraints encapsulates this extra information (beyond the types of arguments) needed to nudge overload resolution in the desired way.

[vexx32]

Would it make sense to create a new PSGenericMethodInvocationConstraints to store the additional type arguments needed for a generic method as a separate value to the types of the method parameters?

[lzybkr]

@vexx32 - probably not - just add a new field to the existing type.

[mklement0]

@lzybkr, @dlwyatt, I'm unclear on what the consensus ended up being regarding the syntax:

[Array]::Empty[string](...)

vs.

[Array]::Empty.InvokeGeneric([string], ...)

or _both_?

If it's either / or, the first seems much preferable; if we only support type _literals_, then confusion with numeric indexing shouldn't be a concern ([Array]::Empty[0]); also, even multiple overloads are reported as a _single_ PSMethod`1 object, so indexing isn't useful anyway.

[vexx32]

I think that the latter is simpler to implement as it doesn't require parser changes. It could be implemented as a halfway measure, handling the actual logic of invoking the generic method first, and then we can tackle the parser challenge afterwards.

@lzybkr summarised the possible challenges better than I can in his comment: https://github.com/PowerShell/PowerShell/issues/5146#issuecomment-338268805

[lzybkr]

@vexx32 - the sharing of logic between invoking methods on a PSMethod and invoking methods the normal way is less than ideal. That's not to say you shouldn't bother implementing the former, just don't get your hopes up that the problem is half solved if you go that route.

[dlwyatt]

The first syntax is what's in the branch I linked earlier

[vexx32]

@dlwyatt thanks for that! After resolving a few merge conflicts I was able to get that code rebased on top of the current master branch, and with a few nudges from @SeeminglyScience here and there on some of the more confusing bits, I've got a working implementation.

Tidying it up and figuring out some tests to write, and then I'll throw up a PR.