Powershell: Inside parentheses, adding a redirection to an expression causes enumeration, unlike without parentheses

[mklement0]

---

_Update, based on the discussion below:_

What it comes down to is the following _inconsistency_:

• In <expression> <redirection> the redirection has _no effect_ on the expression (except if the success output stream is suppressed); e.g.:
  $result = [int[]] (1, 2) 2>$null; $result.GetType().Name yields Int32[])

• whereas enclosing the same expression _in parentheses_ applies _pipeline logic_ and therefore _enumeration_ of the expression result (and subsequent collection in a regular PS array); e.g.:
  ([int[]] (1, 2) 2>$null).GetType().Name yields Object[]

Give how exotic this scenario is, no resolution of this inconsistency is planned.

---

This is an edge case, given that applying a redirection such as *>&1 to a pure _expression_ (one not containing embedded _commands_) is is conceptually pointless, given that it is only ever stream 1 output that is produced.

Still, the fact that adding such a redirection makes empty-collection expressions turn into [System.Management.Automation.Internal.AutomationNull]::Value is puzzling.

Steps to reproduce

# OK: an empty array is not the same as $null
$null -eq (@()) | Should -BeFalse

# !! Unexpectedly fails; the *>&1 redirection makes the expression return $null.
# !! Using specific streams in the redirection - e.g., 3>&1 - fails too.
$null -eq (@() *>&1) | Should -BeFalse

Expected behavior

Both tests should succeed.

Actual behavior

The 2nd test fails, because the RHS unexpectedly becomes $null

Expected $false, but got $true.

The behavior is related to the combination of the redirection with (...), which is a syntactic requirement here, however.

$foo = @() *>&1 works fine, for instance - the *>&1 has no effect.
By contrast, $foo = (@() *>&1) exhibits the problem (but removing the redirection doesn't).

Environment data

PowerShell Core 7.0.0-rc.1

[jhoneill]

It's the same if you do

@() | out-default

$foo = @() *>&1 works fine, for instance - the *>&1 has no effect.

Yes it does.

>$foo.GetType() 
InvalidOperation: You cannot call a method on a null-valued expression.

c.f.

>$foo = @()    
>$foo.GetType()     
IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Array

What you're saying is "send all members of the empty array to the default output." There are no members so the result is null.

[mklement0]

Out-Default only comes into play if the output is _not_ captured - it is unrelated to this issue.

Yes it does.

It doesn't for me, neither in WinPS nor in PS Core, and it is the behavior that I expect:

PS> $foo = @() *>&1; $foo.GetType().Name
Object[]

That is, the *>&1 should be a no-op and indeed is in this case.
No enumeration takes place nor should it.

[jhoneill]

I copied what you _Just_ did

#509 >$test1 = @() *>&1
#510 >$test1.GetType()

IsPublic IsSerial Name                                     BaseType                                                                                                                          
-------- -------- ----                                     --------                                                                                                                          
True     True     Object[]                                 System.Array                                                                                                                      

Not the result I got before ... what ? ... different machine ... then I copied what you _first_ gave

#511 >$test2 = (@() *>&1)
#512 >$test2.GetType()
You cannot call a method on a null-valued expression.

Then the penny dropped; in this:
>$test1 = @() *>&1
the redirection is processed after the assignment - you'd expect *>&1 to be processed last. Wrapping it in brackets caused it to be processed first.

What I meant about out-default was empty sent through anything is null.

#513 >$test3 = (@() | out-default)
#514 >$test3.GetType()
You cannot call a method on a null-valued expression.

Maybe this is better.

#515 >$test4 = (@() | Write-Output)
#516 >$test4.GetType()
You cannot call a method on a null-valued expression.

or

#517 >$test3 = (@() | % {$_} )
#518 >$test3.GetType()
You cannot call a method on a null-valued expression.

These don't need the () because the pipeline is evaluated before the assignment.

[SeeminglyScience]

Then the penny dropped; in this:
>$test1 = @() *>&1
the redirection is processed after the assignment - you'd expect *>&1 to be processed last. Wrapping it in brackets caused it to be processed first.

You'd think based on the results, but non-pipeline statements can't be redirected (language rule).

{ $a = @() *>&1 }.Ast.EndBlock.Statements[0].Right
# Expression Redirections Extent   Parent
# ---------- ------------ ------   ------
# @()        {*>&1}       @() *>&1 $a = @() *>&1

Unless the compiler has some special logic to capture only part of it's child expression, that's not what's happening here.

[mklement0]

Thanks, @SeeminglyScience.

That redirections are processed first actually makes sense, otherwise you wouldn't be able to write statements such as $out = ls / nosuch 2>&1 and have the merged streams captured in $out.

@jhoneill:

then I copied what you first gave

The initial post states that (...) is somehow involved and contrasts that with the parentheses-free $foo = @() *>&1, which preserves the empty array as expected; you previously claimed that the latter also turns to $null, which we now agree is not true.

empty sent through anything is null.

If you assign an _expression_ to a variable, no enumeration is performed. An empty array stays an empty array.

The question is whether something like *>&1 should change that.

Indeed, > - if wrapped in (...) - seems to apply pipeline enumeration semantics:

# No (): no enumeration
PS> ( $foo = [int[]] (1, 2) *>&1 ).GetType().Name
Int32

# With (): enumeration, and collection in [object[]] array:
PS> ( $foo = ([int[]] (1, 2) *>&1) ).GetType().Name
Object[]

Whatever behavior you favor (I see no reason for the enumeration), it is clear that we have an inconsistency on our hands.

[jhoneill]

The initial post states that (...) is somehow involved and contrasts that with the parentheses-free $foo = @() *>&1, which preserves the empty array as expected; you previously claimed that the latter also turns to $null, which we now agree is not true.

My missreading - I missed that you'd contracted the title at the end.
The title should be "using () causes a change in the order of evaluation" :-)

Something like this
>$x = & {@(1) *>&1}
shows the array gets enumerated not passed through by *>&1
Without the () assignment happens first. Nothing complicated or surprising about either of those.

@SeeminglyScience

You'd think, but non-pipeline statements can't be redirected (language rule).

But assignment expressions function as pipeline statement

#542 >$a = 42 > delete.me
#543 >cat delete.me      
42

What's weird is there the value of assignment appears in that redirection but not in $a = 42 *>&1 I've always understood that > was actually doing | out-file underneath and that would need everything before the > to be wrapped in () *>&1 is doing something different

[SeeminglyScience]

@SeeminglyScience

You'd think, but non-pipeline statements can't be redirected (language rule).

But assignment expressions function as pipeline statement

Assignments can't be expressions, they're just statements. Everything to the right of the = is then assigned to the variable.

#542 >$a = 42 > delete.me
#543 >cat delete.me      
42

Yeah, just like 42 > delete.me would do by itself. And $a is null, because the output was redirected to the file, so there was nothing to assign.

If it was redirecting the assignment the file would be null and the variable would be populated.

What's weird is there the value of assignment appears in that redirection but not in $a = 42 *>&1 I've always understood that > was actually doing | out-file underneath and that would need everything before the > to be wrapped in () *>&1 is doing something different

It's not really that cut and dry, there's more that goes into it than just translating to a command call. That said, $a = 42 | Out-File delete.me acts the same.

[mklement0]

Something like this
$x = & {@(1) *>&1}
shows the array gets enumerated

& { ... } _always_ acts this way (it enumerates), whether a redirection is involved or not:
(& { [int[]] (1,2) }).GetType().Name -> Object[].

---

I've updated the issue title to mention the parentheses.

[mklement0]

Without the () assignment happens first.

It's implied by @SeeminglyScience's explanation, but to spell it out:

The assignment does _not_ happen first; if it did, $a = 42 > delete.me would result in $a containing 42, but it actually contains "nothing" ([System.Management.Automation.Internal.AutomationNull]::Value).

[jhoneill]

Assignments can't be expressions, they're just statements. Everything to the right of the = is then assigned to the variable.

Something I've assumed for a decade or more is showing cracks
I've always fitted = into being an operator, and += etc belonging with + as arithmetic operators.

To support this thinking: Brackets don't turn a statement into a value

 >(foreach ($x in 1,2) {$x})
ParserError:
Line |
   1 | (foreach ($x in 1,2) {$x})
     |              ^ Unexpected token 'in' in expression or statement.

It needs a leading $

>$(foreach ($x in 1,2) {$x})
1
2

But assignment does turn into a value with just brackets

>($j =1)
1

This works
if ($s = [datetime]::Now.Second %2) {"odd $s"}
This fails
if (if ([datetime]::Now.Second %2) {$true}) {"odd"}

So assignment doesn't belong in the same box as for or while or if

>$j ++ > delete.me
>cat delete.me
1
>$j
2

Increment is a form of assignment, right ? This should be increment $j and write the result
>$j += 1 > delete.me

$j
2
cat delete.me
1

So that has done $j += (1 > delete.me)
which is logical enough because > is actually out-file, which you can see with a broken file name

$j ++ > delete:me
Out-File: Cannot find drive. A drive with the name 'delete' does not exist.

So really it $j += 1 > delete.me is $j += 1 | out-file delete.me and we'd expect $x = something | somethingElse | etcto put the result of the whole pipeline into $x, ($x = something) | somethingelse would be crazy.

Of course the > operator is crazy because you can write it in the middle of a command
>dir \foo,'C:\Program Files\p*','C:\Programs\' > foo.txt -include p*

If that wasn't enough.

dir \foo,'C:\Program Files\p*','C:\Programs\' > foo.txt -include p* >bar.txt
ParserError:
Line |
   1 | dir \foo,'C:\Program Files\p*','C:\Programs\' > foo.txt -include p* >bar.txt
     |                                                                     ^ The output stream for this command is already redirected.

Well obviously. So I shouldn't be able to pipe the output but this :
dir \foo,'C:\Program Files\p*','C:\Programs\' > foo.txt -include p* | format-table
Runs

To the matter in hand.
Are wer surprised that $x *>&1 enumerates X if it is an array ?

$y = (@(1) *>&1 )
gives the same result as
$y = $(@(1) )
or
$y = (@(1) | % {$_} )

It's not the enumeration which is odd at all. I think @mklement0 has it backwards. What is odd is that
$y = @(1) *>&1 assigns the unenumerated array to $y

But it has been like that since Windows PowerShell 3 released so I think that boat has sailed, run aground and all aboard have been taken off. So I'm not sure what the goal is here, go back and change some very old behavior to remove the need for some brackets in a very odd use case ?

[SeeminglyScience]

Assignments can't be expressions, they're just statements. Everything to the right of the = is then assigned to the variable.

Something I've assumed for a decade or more is showing cracks
I've always fitted = into being an operator, and += etc belonging with + as arithmetic operators.

To support this thinking: Brackets don't turn a statement into a value

Whoops I misspoke, AssignmentStatementAst is a pipeline... sorta. It's a subclass of PipelineBaseAst, so it can go in most of the places where a pipeline can, however PipelineBaseAst does not store redirections. That's PipelineAst, which can only store a command or a command expression.

It's more of a technicality to allow assignments to be stored in a paren expression, not much more than that though.

But assignment does turn into a value with just brackets

>($j =1)
1

Yeah that's some special casing by the compiler. They wanted to match C# where an assignment is typically a void statement, but if wrapped in parenthesis it returns the value that was assigned. (Side note, iirc someone wanted assignments to return by default. That would have been nuts)

It's not the enumeration which is odd at all. I think @mklement0 has it backwards. What is odd is that
$y = @(1) *>&1 assigns the unenumerated array to $y

But it has been like that since Windows PowerShell 3 released so I think that boat has sailed, run aground and all aboard have been taken off. So I'm not sure what the goal is here, go back and change some very old behavior to remove the need for some brackets in a very odd use case ?

I just want to clarify that I don't really have an opinion on this. I want to make sure that it's understood what is happening, but I don't use redirection and typically advise other folks not to either.

[jhoneill]

Whoops I misspoke, AssignmentStatementAst _is_ a pipeline... sorta. It's a subclass of PipelineBaseAst, so it can go in most of the places where a pipeline can, _however_ PipelineBaseAst does not store redirections. That's PipelineAst, which can only store a command or a command expression.

Phew. I'm not totally nuts then. And it's nice to see someone other than me doing the mis-speaking

It's not the enumeration which is odd at all. ....
But it has been like that since Windows PowerShell 3 ... I'm not sure what the goal is here,

I just want to clarify that I don't really have an opinion on this. I want to make sure that it's understood _what_ is happening, but I don't use redirection and typically advise other folks not to either.

I know (from discussing ansi sequences going into standard output) that you're not a big fan of redirection but "Send what you printed on the screen to a file" / "Send it to std-in of another executable" is still a common (if less than ideal) scenario. I don't doubt that mk has found something odd, but I think the oddity is not that $x = ($array *>&1) assigns the result of enumerating the array but that $x = $array *>&1 assigns the array.
But then what ? Do we just congratulate him for finding something so obscure that it's been there for 10 years and no-one has noticed, and leave it alone, on the grounds that changing it has infinitesimally small value , but entails effort and the risk of breaking other things ?

[vexx32]

I think the point of filing the issue is mainly to note that a) it is a thing that happens, though it's a bit of an unusual case to stumble across, and b) to prompt folx like ourselves and also the PS team to think about whether it's something we should be fixing. 🙂

I think the argument that both cases should behave the same is sensible -- the fact that adding parentheses changes how the expression behaves does seem like the kind of thing that shouldn't happen. Whether it's worth fixing or might break something else to fix it, is something that we should try to determine, I think. 😁

[SeeminglyScience]

Whoops I misspoke, AssignmentStatementAst _is_ a pipeline... sorta. It's a subclass of PipelineBaseAst, so it can go in most of the places where a pipeline can, _however_ PipelineBaseAst does not store redirections. That's PipelineAst, which can only store a command or a command expression.

Phew. I'm not totally nuts then. And it's nice to see someone other than me doing the mis-speaking

😄

I know (from discussing ansi sequences going into standard output) that you're not a big fan of redirection but "Send what you printed on the screen to a file" / "Send it to std-in of another executable" is still a common (if less than ideal) scenario.

What I mean is that I don't recommend using the redirection language syntax. There's plenty of reasons to use Out-File and what not. I don't recommend sending formatted data to it, but it's still a useful cmdlet for sure.

I don't doubt that mk has found something odd, but I think the oddity is not that $x = ($array *>&1) assigns the result of enumerating the array but that $x = $array *>&1 assigns the array.
But then what ? Do we just congratulate him for finding something so obscure that it's been there for 10 years and no-one has noticed, and leave it alone, on the grounds that changing it has infinitesimally small value , but entails effort and the risk of breaking other things ?

🤷‍♂ I don't have an opinion on it. If I chime in to explain something, some folks could take that to mean that I'm advocating for one side or the other. In this case I'm not, I'm just making sure that those who do care have all the information (basically what @vexx32 said 🙂).

[jhoneill]

I think the point of filing the issue is mainly to note that a) it is a thing that happens, though it's a bit of an unusual case to stumble across, and b) to prompt folx like ourselves and also the PS team to think about whether it's something we should be fixing. 🙂

Maybe... but there is a level of obscurity where it is really not worth fixing. Take a couple of things I mentioned along the way
Get-ChildItem > dir.txt -force
Syntactically legal and the force applies to GCI , not to the redirection. Or
Get-ChildItem > dir.txt -force | format-table
You can ask for something to be redirected twice. Are those wrong ? Should they be fixed ? or are the just part of life's rich tapestry ?

I think the argument that both cases should behave the same is sensible -- the fact that adding parentheses changes how the expression behaves does seem like the kind of thing that shouldn't happen. Whether it's worth fixing or might break something else to fix it, is something that we should try to determine, I think. 😁

Worth fixing ...Security risk ? no. Problems in the field ? No ?
I think the change in behaviour with and without brackets is odd. But normally $x = something-with-redirection , has the redirection fire before the assignment happens and what ever emerges at the end of the redirection goes to the assignment. Here you have to enclose the redirection to get it to happen, otherwise the assignment happens and nothing gets redirected.

But there is other wierdness What should this do ?
$y = (1/0) *>&1
The divide by zero error should end up in y, right ? Try it because that isn't what happens.
OK let's wrap it and force it to evaluate
$y = ((1/0) *>&1)
That's got to work right ? No.

This works.
$y = & {(1/0)} *>&1

Conclusion I get to is redirecting (or merging the streams of) an expression instead of a command means normal rules might not apply. How urgent is it to fix this ... wash your dog first :-)

[vexx32]

Aye, I'll definitely agree it's not an important issue for everyone. But that doesn't mean it's not an important issue for anyone. 🙂

[mklement0]

Fully agreed, @vexx32, but I think the point from your earlier comment is a more important one that bears repeating:

An anomaly / inconsistency is _always_ worth investigating, _wherever it may lead_.

Once the behavior and, ideally, the underlying cause is _understood_, appropriate action - if any - can be taken, _whatever it may be_.

---

Clearly, in the case at hand discussion was needed to get conceptual clarity on the behavior: we've now narrowed down the inconsistency (though we still don't know its precise cause).

The stated premise of this issue was that this is an exotic and conceptually pointless use case, but, if you build [the ability to do] it, they will come, if you will - a since-edited answer on Stack Overflow prompted the question.

Plus, before fully understanding the issue you never know what other, potentially more serious manifestations it may have.

Personally, I'm happy with leaving this be, now that I understand the issues involved.

If a fix were to be made, I'd be happy with making >&1 consistently _enumerate_ as well - what matters, in the end, is whether the behavior is _consistent_, not (too) surprising, and easily conceptualized.

Even if no action is taken, the discussion was worthwhile (most of it), at least to me.

Finally, in the spirit of furthering a shared understanding, I will address a few more points below.

---

You can ask for something to be redirected twice. Are those wrong ? Should they be fixed ?

No. There are no inconsistencies there. > in the middle of a command is defensible, though perhaps ill-advised; Get-ChildItem > dir.txt -force | format-table is just user error - you can't fully guard against users combining language elements in nonsensical ways.

Here you have to enclose the redirection to get it to happen, otherwise the assignment happens and nothing gets redirected.

For the primary use case for redirections - calling an external program - _not_ using (...) works just fine.

$out = ls / nosuch 2>&1   # captures both stdout and stderr in $out

What should this do ? $y = (1/0) *>&1

1/0 is a _statement-terminating_ error, so the entire statement is aborted - $y is not created - this example is therefore unrelated to our discussion.

$y = (@(1) *>&1 )
gives the same result as
$y = $(@(1) )
or
$y = (@(1) | % {$_} )

$(...) and @(...) _always_ enumerate, whereas (...) doesn't (except if you use it as the 1st segment of a pipeline).

[mklement0]

I've updated the initial post with a summary to help future readers, should they stumble across this, and I'm closing the issue.

@SeeminglyScience, if you're familiar with the technical why of the inconsistency and you can think of a real-world use case (as opposed to the contrived one shown here) where it could manifest, please let us know.