Azure-pipelines-tasks: pwsh (PowerShellV2) tries to spawn powershell.exe process inside container with PowerShell Core

[Robot1que]

Type: Bug
Task Name: pwsh (PowerShellV2)

Environment

Server

Azure DevOps Server
Version 17.143.28912.1

Agent

Type: Private
OS: Windows Server 1809 with Docker
Version: vsts-agent-win-x64-2.144.2

Issue Description

I run pwsh task inside a mcr.microsoft.com/powershell:6.2.3-nanoserver-1809 container and get failure mentioning that node.js cannot spawn C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe process.
I wonder why pwsh task needs to run powershell.exe process?

YAML definition excerpt

container:
  image: mcr.microsoft.com/powershell:6.2.3-nanoserver-1809

steps:
- pwsh: |
    # Inline script

Task logs

2019-09-27T11:17:12.7725567Z ==============================================================================
2019-09-27T11:17:12.7725614Z Task         : PowerShell
2019-09-27T11:17:12.7725642Z Description  : Run a PowerShell script on Windows, macOS, or Linux.
2019-09-27T11:17:12.7725676Z Version      : 2.140.2
2019-09-27T11:17:12.7725702Z Author       : Microsoft Corporation
2019-09-27T11:17:12.7725730Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613736)
2019-09-27T11:17:12.7725767Z ==============================================================================
/* ... */
2019-09-27T11:17:13.0712078Z events.js:160
2019-09-27T11:17:13.0712509Z       throw er; // Unhandled 'error' event
2019-09-27T11:17:13.0712639Z       ^
2019-09-27T11:17:13.0714281Z 
2019-09-27T11:17:13.0720062Z Error: spawn C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe ENOENT
2019-09-27T11:17:13.0723425Z     at exports._errnoException (util.js:1018:11)
2019-09-27T11:17:13.0723712Z     at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
2019-09-27T11:17:13.0723870Z     at onErrorNT (internal/child_process.js:367:16)
2019-09-27T11:17:13.0723990Z     at _combinedTickCallback (internal/process/next_tick.js:80:11)
2019-09-27T11:17:13.0724113Z     at process._tickCallback (internal/process/next_tick.js:104:9)
2019-09-27T11:17:13.1880286Z ##[error]Exit code 1 returned from process: file name 'C:\Program Files\Docker\docker.EXE', arguments 'exec -i 75c31673282e4f2396b2b1c44e37e07a683a7ad0e248aacbc0eaaf084541377d C:\__a\externals\node\bin\node.exe C:\__w\_temp\containerHandlerInvoker.js'.

[mjvankampen]

Same here except that it is a script task that given the same error:

==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.163.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: spawn powershell.exe ENOENT
    at exports._errnoException (util.js:1020:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:197:32)
    at onErrorNT (internal/child_process.js:376:16)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
##[error]Exit code 1 returned from process: file name 'C:\Program Files\Docker\docker.EXE', arguments 'exec -i  fe5dd5bf5899302bcd6120d6350f363c9b6b4c54d00c006ca8134ed63e5b519a C:\__a\externals\node\bin\node.exe C:\__w\_temp\containerHandlerInvoker.js'.

[JustinGrote]

I had the same problem on nanoserver image but it works in windowsservercore image: mcr.microsoft.com/powershell:windowsservercore-1809

but not nano image

May have something to do with the WindowsCompatability module that spawns a powershell process for 5.1 module compatibility?

[JustinGrote]

This looks like the culprit:
https://github.com/microsoft/azure-pipelines-tasks/blob/b414a7ab7798515079dba9479dc49c040c742221/Tasks/PowerShellV2/task.json#L129-L134

In both the cmdline and pwsh tasks, on windows they actually call a .ps1 file directly. I'm guessing in the nano container, .ps1 files are still registered in cmd to start with Windows Powershell which doesn't exist. Perhaps a .js pre-step or something needs to happen to detect if powershell.exe is available and use pwsh.exe instead if not available on windows systems?

Workaround is to use the windowsservercore container instead.

@jpetty thoughts?

[jtpetty]

I actually have a PR to fix this in the agent itself. https://github.com/microsoft/azure-pipelines-agent/pull/2767

The problem is that our task model uses either a Node or PowerShell handler to run the task, not PowerShell Core. On Windows, we always prefer the PowerShell handler, but in this case, there is no PowerShell in the container, only PowerShell Core.

[JustinGrote]

@jtpetty I saw your change was merged, is this in the new image and can be closed?

[mjvankampen]

I get a different error now. I tried mcr.microsoft.com/windows/nanoserver:1809 and mcr.microsoft.com/powershell:lts-nanoserver-1809. It seems somehow the task tries to use bash, pretty sure that's not in Windows images :-).

This seems a separate issue, ref https://github.com/microsoft/azure-pipelines-agent/issues/2870

##[error]Unable to locate executable file: 'bash'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

[jtpetty]

@damccorm - Here is another issue around the same problem with needed bash to run the Node task handler.

[damccorm]

Hey all, we've decided to not make the change that fixed this issue and hide it behind a flag instead. If you would like to continue using the current behavior, set a pipeline variable agent.preferPowerShellOnContainers to false in any pipelines that require this behavior.

I'm going to try to summarize the issue and explain why we landed where we did. There's probably room for improvement going forward, so this isn't necessarily where this issue will end up, but is our current outlook.

Original Behavior

Originally, we assumed if you were on a windows container that you had PowerShell installed. This has been a requirement for windows machines as long as we've had the agent, and we extended that principle to containers. Unfortunately, however, PowerShell Core is not compatible with our current PowerShell handler logic. This means that when we try to launch PowerShell on newer Windows containers with only PowerShell Core installed, any tasks that use our PowerShell handler fail. That includes tasks like CommandLine and PowerShell.

Updated behavior

To fix this, we added a setting that prefers the Node handler for tasks by default in containers. This means that we try to execute any tasks that have a Node handler using that handler. That breaks an assumption that many tasks implicitly have: that if they are executing the PowerShell handler they are on Windows, if they are executing the Node handler they are on Linux. That's why the CommandLine task is trying to find bash, on Linux that's what it executes!

Going forward

Going forward, we will revert to the original behavior, with the option to use the updated behavior behind a flag.

We're doing that to avoid breaking existing scenarios that are working - before this change, Windows containers with only PowerShell Core didn't work. We are looking into enhancements to enable those scenarios without breaking existing ones

[JustinGrote]

Thank you @damccorm.

So, to interpret, if we set agent.preferPowerShellOnContainers to false on a nanoserver container, it should now work for pwsh scripts?

[damccorm]

So, to interpret, if we set agent.preferPowerShellOnContainers to false on a nanoserver container, it should now work for pwsh scripts?

The powershell task itself should work. I would expect the commandline task to fail, though, since it would run its Node Handler Linux version which looks for bash.

[damccorm]

I'm going to close here since the issue should be resolved. If that's not the case, please make sure you've upgraded to the most recent agent version and comment here and we can reopen