Powershell: Make it easier to download and execute scripts as a single operation, by enhancing Invoke-Command

[mklement0]

Summary of the new feature/enhancement

A longstanding, convenient idiom in the Unix world is to download and execute installation / bootstrapping shell scripts with a single, low-complexity shell command:

For instance, you can install the .NET Core SDK with the following command, by downloading the script from GitHub and piping to bash for execution:

curl https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain/dotnet-install.sh | bash

As an aside:
There is also a *.ps1 script, but as of this writing is _Windows-only_, which is unfortunate. I have a PR open to make it _cross-platform_, but it is languishing due the .NET Core CLI team giving no indication as to whether they're even _prepared_ to accept such a PR and thereby future maintenance of the script.
@SteveL-MSFT, if you'd like to see a cross-platform *.ps1 .NET Core SDK installation script happen, I encourage you to make your voice heard at https://github.com/dotnet/cli/issues/8278 - which is an issue that @vors originally opened in December 2017.
The utility of such a script would be greatly enhanced if this issue's requested feature were implemented.

It would be nice if PowerShell supported something similar, which is currently not the case:

• Providing the code via stdin along the following lines results in extremely slow, pseudo-interactive execution that echoes each script line (irm (Invoke-RestMethod) is used to download the script):

# !! Even though it ultimately works, the UX is unacceptable.
irm https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain/dotnet-install.ps1 | pwsh

This is a general problem with stdin input, originally reported in #3223

• Using Invoke-Expression is an improvement, but has caveats and only works in limited circumstances:

# ALMOST works, but has a number of problems - see below.
iex (irm https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain/dotnet-install.ps1)

Limitations / bugs:

• The downloaded script-content must not terminate with exit, lest the calling session _as a whole_ be exited.

• A couple of _bugs_ currently prevent correct script execution:

  • #8778
  • #8815

• You cannot pass _arguments_ to an Invoke-Expression call.

  • This SO answer shows a workaround via [scriptblock]::Create(), but that is obviously cumbersome and obscure.

• Invoke-Expression invariably "dot-sources" the code (executes directly in the caller's scope), thereby polluting the caller's scope. (While you could wrap the iex call in & { ... } that is both cumbersome and easy to forget.)

_Update_: @SteveL-MSFT's clever approach below actually _bypasses_ the bugs and the scoping problem, but it is too obscure (and it doesn't address the exit problem).

Note that Invoke-Expression has one advantage over the curl ... | sh idiom for POSIX-like shells: due to running _in-process_, the script has the ability to update the caller's _environment variables_.

In short: Currently, the only way to robustly handle download-and-invoke scenario is to download to a _temporary script file_ first, execute that, and then clean up - which is obviously cumbersome.

Proposed technical implementation details (optional)

Based on the above, the following limitations must be overcome (in addition to fixing the bugs mentioned above):

• exit in a script (a scriptblock created from a script's downloaded content) must only exit the script itself.

• the script must execute in a _child scope_

• it must be possible to pass _arguments_ to the script.

Additionally, it would be nice not to have to use a separate command for downloading the script.

To that end, Invoke-Expression Invoke-Command (icm) / & (.) could be enhanced as follows:

^{Note: Invoke-Expression is ultimately not the right cmdlet to use for two reasons: (a) it doesn't quite convey the intent of the operation (that is, invoking a whole _script_) and (b) we want to discourage Invoke-Expression use in general.}

Note: The hypothetical example command line below use invocation of the PowerShell installation script https://aka.ms/install-powershell.ps1 with argument -Preview

• Add a -FromPipeline switch - with a short alias -s to parallel the from-stdin option in POSIX-like shells such as bash (mandated by POSIX)- to accept a script's _content_ via the pipeline:

Invoke-RestMethod https://aka.ms/install-powershell.ps1 | Invoke-Command -FromPipeline -Preview

• As a more convenient (additional) alternative, add a -FileUri parameter to parallel -FilePath that directly accepts a URI to download the script text from, which would essentially perform the Invoke-RestMethod (irm) call _internally_, which aside from being shorter and more convenient:

  • allows validating that the URI points to a resource of the expected time ('text/plain`).
  • allows enforcing the effective execution policy if the script is unsigned, with a new -Force switch allowing intentional overriding
    
    
    
    • Note: the execution policy could also be enforced with pipeline-supplied input text, though that could be considered too restrictive, given that what is sent through the pipeline needn't have a _remote_ origin.
    
    
  • Note: If more control over the download aspect is needed (e.g., passing credentials), the -FromPipeline approach with a separate download command (such as Invoke-RestMethod / Invoke-WebRequest) must be used.

Invoke-Command -FileUri https://aka.ms/install-powershell.ps1 -Preview

Finally, enhance & (and ., though that is less likely to be useful) so that the command above can more concisely be written as:

& https://aka.ms/install-powershell.ps1 -Preview

To be safe, the execution policy should be invariably enforced in this case. Overriding it would then require the more deliberate act of using Invoke-Command with -Force.

---

_Obsolete_ part of this proposal, which suggested enhancing Invoke-Expression:

• Generally, add a -UseChildScope switch to the cmdlet to opt into execution in a child scope.

• Generally, add a -ArgumentList (-Args) parameter to allow passing arguments to the script, as with Invoke-Command.

  • As with Invoke-Command defining the parameter as ValueFromRemainingArguments allows pass-thru arguments to be specified more naturally (-foo bar instead of -Args '-foo', bar).

• Add a -FromUri <uri> parameter that allows implicit downloading of script files to execute; this parameter would _imply_ -UseChildScope.

  • If there are security concerns, a confirmation prompt / -Force switch could be added.

• When running in a child scope, make exit exit the script only.

  • _Update_: By limiting this behavior change to when the _new_ (implied via -FromUri) -UseChildScope is in effect, there are no backward-compatibility concerns (though it's hard to imagine anyone _relying_ on something like iex '"hi"; exit' exiting the entire session).

With the above, downloading and executing the .NET Core SDK installation script with argument
-DryRun would then look like this:

$uri = 'https://raw.githubusercontent.com/dotnet/cli/master/scripts/obtain/dotnet-install.ps1'
iex -FromUri $uri -DryRun

Written as of:

PowerShell Core 6.2.0-preview.4

[SteveL-MSFT]

@mklement0 since your PR in dotnetcli repo is marked as WIP, I suspect they will wait until you deem it not a WIP before they review

It's certainly possible to pass args with saving to file first with this hard to remember syntax :)

iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI -Preview"

I don't think we want invoke-expression to overlap with invoke-restmethod, but agree that having a way to pass args would greatly improve the experience. Making it easy to verify the signature within the pipeline would help keep it secure. In general, exit should not be used in PowerShell scripts. Would be breaking change to change it.

[mklement0]

@SteveL-MSFT:

Re PR:

I suspect they will wait until you deem it not a WIP before they review

Fair enough, but I want _assurance that there's fundamental willingness to accept such a PR_, given that it means that _they_ will have to maintain this cross-platform script _in parallel_ with the Bash version going forward - that is what I've been asking for, with no response.

Obviously, _my_ vote is for them to take this on, but I can also see why they'd be hesitant - and that's where it helps if others voice interest.

The PR is _functionally_ ready for review, but it is _lacking tests_ - adding those would mean substantial additional effort, which I'd like not to expend without knowing if the PR will even be accepted.

---

Re syntax:

with this hard to remember syntax :)

That's clever, but, as you say, obscure and hard to remember - and it doesn't address the exit issue (see below).

---

Re exit:

In general, exit should not be used in PowerShell scripts.

Not only does the documentation state "Causes PowerShell to exit a script or a PowerShell instance.", using exit <n> in a script is the only way to set the _exit code_ for callers that expect success / failure to be communicated via exit codes - a vital feature for robust automation.

To address potential backward-compatibility concerns, making exit behave _the way it already does in scripts_ could be limited to the - previously nonexistent - -UseChildScope parameter (which the new -FromUri parameter would imply).

---

Re overlap with Invoke-RestMethod:

The overlap is only for the very specific use case discussed here - which ideally should have a _single-command_ implementation.

Internalizing Invoke-RestMethod-like functionality to Invoke-Expression -FromUri would have two advantages:

• being able to validate that the targeted URI is a plain-text resource
• being able to enforce the effective execution policy with respect to unsigned scripts, with a -Force switch allowing intentional overriding.

Thus, your clever-but-obscure command:

iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI -Preview"

would turn into:

_Update_: Note that proposal is now to enhance Invoke-Command rather than Invoke-Expression - see updated initial post.

iex -FromUri https://aka.ms/install-powershell.ps1 -UseMSI -Preview

• with -Force allowing execution of unsigned scripts prevented by the execution policy.
• with -ArgumentList (-Args) defined as ValueFromRemainingArguments to allow passing arguments through in a more natural fashion.

[mklement0]

@SteveL-MSFT:

Taking a step back, I've realized that adding the proposed functionality to Invoke-Expression is ill-advised, for two reasons:

• it doesn't quite convey the intent of the operation (that is, invoking a whole _script_)
• we want to discourage Invoke-Expression use in general.

By contrast, Invoke-Command is a more natural fit - I've updated the initial post accordingly.

This also makes concerns about Invoke-Expression's current behavior irrelevant.

Based on the updated proposal, your command would become:

icm -FileUri https://aka.ms/install-powershell.ps1 -UseMSI -Preview
# or, shorter:
& https://aka.ms/install-powershell.ps1 -UseMSI -Preview

[SteveL-MSFT]

@mklement0 I like using Invoke-Command vs Invoke-Expression. I like the terseness and conciseness. However, still not a fan of adding web capability to another cmdlet. Perhaps an alternative would be to have a HTTP: and HTTPS: drive that would allow any cmdlet accept a file path to GET and POST to a URL? This would mean Get-Content https://aka.ms/install-powershell.ps1 would work.

[vexx32]

@SteveL-MSFT implementing such a provider would be interesting to say the least. Love the idea, though!

@markekraus do you think this idea is feasible?

If it can utilise the web cmdlets' base functionality, we can hopefully avoid having two disparate sets of features for the same thing here.

[markekraus]

This is a duplicate of #5909

[mklement0]

Thanks, @SteveL-MSFT and @vexx32.

However, still not a fan of adding web capability to another cmdlet.

I can see why, and I'd personally be happy with limiting the Invoke-Command change to -FromPipeline (and thereby forgoing the convenience of & <url> ...):

iwr https://aka.ms/install-powershell.ps1 | icm -FromPipeline ...

That way, download functionality remains the purview of Invoke-WebRequest / Invoke-RestMethod.

That said, I like @lzybkr's comment here (thanks for pointing out the duplicate, @markekraus), which, adapted to this proposal, would allow us to still let Invoke-Command enforce the execution policy:

icm -FromWebRequest (iwr https://aka.ms/install-powershell.ps1) ...

# or, *more naturally, via the pipeline*:
iwr https://aka.ms/install-powershell.ps1 | icm ...

That is, a by-value pipeline-binding [Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject]-typed parameter named something like -FromWebRequest could directly bind Invoke-WebRequest output (_not_ Invoke-RestMethod output, whose only advantage would be direct _text_ output) and perform all the necessary validation and execution-policy enforcement (overridable with -Force).

If we implement (just) this, I personally don't see the need for a http: drive anymore.

[SteveL-MSFT]

Enabling pipelining would certainly solve your problem and is a much simpler effort than a http: drive. I think (outside of this issue), a http: drive might still be interesting :)

[SteveL-MSFT]

Since we've agreed that pipelining is a fine solution, resolving this as dupe of #5909

[iSazonov]

With http/https drive we could do

icm https://aka.ms/install-powershell.ps1

without FromWebRequest parameter.

[mklement0]

@iSazonov:

On a _minor_ note: That would require tweaks to the parameter sets, because you can't pass a file path positionally to icm, and even with an explicit -FilePath (which would be a tad awkward with a URL) you cannot currently execute _locally_ - you need the -ComputerName parameter too, from what I can tell (not sure if that's by design).

Overall, however, I can definitely see the appeal, and it would again also enable use with & and .

& https://aka.ms/install-powershell.ps1   # or: . https:.//...

This new provider would have to report all its "files" as downloaded-from-the-net for the purposes of enforcing execution policy.

My saying that we may not need this was primarily based on the perceived substantial implementation effort, but I encourage you or @SteveL-MSFT to open a new feature request.

[iSazonov]

It should be RFC for new WWW provider/drive/namespace.

Update: #8835