Powershell: Accessing the $input variable in Start-Job/Start-ThreadJob causes the $input value not to be output to the pipeline.

[rayzorben]

Steps to reproduce

Note: the issue happens with Start-Job as well but Start-ThreadJob will show you the issue 8x faster! :)

The following outputs 1..10 to the pipeline. It writes 'foo' to the console, and 'out 1..10'

$jobs = 1..10 | %{ $_ | Start-ThreadJob -ScriptBlock { Write-Host "foo"; $input } }; $jobs | Receive-Job -Wait -AutoRemoveJob | ForEach-Object { Write-Host "out $_" }

The following does not output anything to the pipeline. It prints 'input 1..10' to the screen, but no 'out 1..10' is printed.

$jobs = 1..10 | %{ $_ | Start-ThreadJob -ScriptBlock { Write-Host "input $input"; $input } }; $jobs | Receive-Job -Wait -AutoRemoveJob | ForEach-Object { Write-Host "out $_" }

Even just $var = $input.Item causes the same issue. Once $input is accessed it can no longer be written out to the pipeline.

Even assigning to another variable, that variable can't be output to the pipeline. Example from my code:

{
    $item = $input
    $test = $item.Entity
    $item
}

Expected behavior

I would expect the first one to output $input e.g., 1..10 to the pipeline.

Actual behavior

1..10 is not output to the pipeline.

Environment data

Name                           Value
----                           -----
PSVersion                      7.0.0-rc.1
PSEdition                      Core
GitCommitId                    7.0.0-rc.1
OS                             Linux 5.4.7-zen1-1-zen #1 ZEN SMP PREEMPT Tue, 31 Dec 2019 17:20:2…
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[vexx32]

$input is not the same as $_; $input is an _enumerator_ variable containing all the as-yet-unused pipeline values. As soon as you read a value from $input, that value is no longer available from $input. It's better to treat $input as a function that outputs a value once rather than something you can continually pull the current value from.

Not sure why $input needs to be used here, I would think $_ is a more natural choice, especially since it doesn't have this rather annoying pitfall. Perhaps @PaulHigin can speak to the design decision there.

In my opinion, using $input for threadjobs is probably not the best idea, but if that's how it's presently designed... 🤷‍♂

[rayzorben]

I found a workaround but I wasted 3 days dealing with inconsistent behavior because this isn't documented anywhere I can find, nor is it expected behavior in any programming language i've used that once you simply access a variable, you can no longer use it. For now I do

-ScriptBlock {
    switch ($input) {
        { $true } {
             $foo = $_
             $_
        }
    }
}

And now I can use $_ just like I would expect to be able to.

I would love for $_ to be how you refer to the pipeline input, but that option doesn't seem to be available. For example this does nothing

$jobs = 1..10 | %{ $_ | Start-ThreadJob -ScriptBlock { $_ } }; $jobs | Receive-Job -Wait -AutoRemoveJob | ForEach-Object { Write-Host "out $_" }

Honestly I am considering writing a wrapper around Start-ThreadJob to dynamically modify the ScriptBlock and add in a switch statement automatically, if I could only now figure out how to parse the Ast :)

[SeeminglyScience]

I found a workaround but I wasted 3 days dealing with inconsistent behavior because this isn't documented anywhere I can find, nor is it expected behavior in any programming language i've used that once you simply access a variable, you can no longer use it.

That's not really what's happening. You can still access it, it'll even still be the same exact object, it's state is just different. It's sort of like if you've ever used adsisearcher. If you've already ran a Find method you wouldn't expect it to automatically reset itself so you can get the same object back when you run the method again. Same thing here, $input is an enumerator; after you enumerate, it's not going to reset itself.

All that said, I'd recommend just pretending $input doesn't exist. It's basically the only place in PowerShell you need to actually know what an enumerator is and how it works for anything to be consistent. There's not much that can be done about it now since changing it would be a breaking change, but I really wish that was never an enumerator.

Instead, use a process block:

Get-ChildItem | Start-ThreadJob { process { $_ } } | Receive-Job -Wait -AutoRemoveJob

(Also note that this doesn't really have anything to do with jobs, gci | & { $input; $input } has the same problems)

[iSazonov]

/cc @bergmeister Perhaps PSSA could address the problem.

[bergmeister]

PSSA has the AvoidAssignmentToAutomaticVariable for that but at the moment only read-only automatic variables are part of it as some automatic variables can be assigned to by design. I agree though that $input should be added to the list, with a lower severity though.

[SeeminglyScience]

@bergmeister I think the request is for a new rule that warns against enumerating $input multiple times.

# First enumeration
$allInput = $input | ForEach-Object { $_ }

# Warn
$allInput = foreach ($obj in $input) { $obj }

# Warn
$asString = "$input"

# Don't warn
$input.Reset()

# Don't warn
$enumerator = $input

# Don't warn
gci | & { $input }

Tbh it seems incredibly difficult to create such a rule without either missing most cases or creating a whole lot of false positives. It would be really cool though.

I could also see a (opt-in) rule that just straight up recommends against using $input period.

[vexx32]

I think the issue here is partially design decision (-InputObject is _only_ assigned to $input, not $_, in a scriptblock sent to a Start-Job or Start-ThreadJob cmdlet) and part documentation -- the _only_ example in the documentation using -InputObject also recommends using $input rather than a process{} block as you illustrated.

We've kinda sent folx directly into the pitfall here.

[SeeminglyScience]

I think the issue here is partially design decision (-InputObject is _only_ assigned to $input, not $_, in a scriptblock sent to a Start-Job or Start-ThreadJob cmdlet)

Part of the complication there is that if you are going to set $_, then you can't really have named blocks. Otherwise actually adding process block is confusing. Also if you're going to make it an implicit process block (or rather, act like one) similar to ForEach-Object then that (to me) implies that it's creating a job per pipeline object.

Afaik there's no way to directly set $_ without using one of the ScriptBlock.Invoke* methods, none of which will allow explicit named blocks. Even if there is another way, I'm not sure what it would set it to. For example, in a begin block, there wouldn't be anything to set it to.

and part documentation -- the _only_ example in the documentation using -InputObject also recommends using $input rather than a process{} block as you illustrated.

We've kinda sent folx directly into the pitfall here.

I'd like to see that changed, but I don't really blame whoever wrote that.

1. At a glance, $input seems to do what you want. That's why it's an easy trap to fall into, even for experienced folks

2. Folks with a CS background don't think twice about it (I assume anyway). I imagine enumerators are a concept that's taught pretty early on, so it's probably routine

3. I can't recall seeing anyone else talk about why it's confusing, nor can I remember seeing other folks recommend against it's use. I'm not sure that it is a popular sentiment

[bergmeister]

Ok, I see. Yes, I agree writing a useful PSSA rule is too hard with too little benefit.
This sounds more like something that should be better documented. Otherwise we'd end up creating rules for all possible combinations of how to shoot yourself into the foot.

[iSazonov]

Please anybody open the doc issue.