Azure-pipelines-tasks: The pwsh task breaks with multi-repo checkout

[mitchdenny]

Question, Bug, or Feature?
Type: Bug

Enter Task Name: PowerShell

Environment

• Server - Azure Pipelines
• Agent - Hosted

Issue Description

When using the pwsh task on with the multi-repo checkout feature I experience the following issues:

1. On Windows 2019 image, I get the error "##[error]Container path not found: 'd:\a\1\s'"
2. On Ubuntu 16.04 image, I get the error "##[error]No such file or directory"

For #1, the issue can be worked around by setting the workingDirectory. I haven't found a workaround for #2 yet.

[vtbassmatt]

@jtpetty @stephenmichaelf @jpricket FYI. Not sure who wants to take a crack at this one.

[jtpetty]

@mitchdenny - This is quite possibly fixed in the latest agent, 2.164.8. Can you try that agent version? We are in the process of rolling this version out to all customers so you may not have gotten it automatically yet.

[mitchdenny]

Sorry the .8 build doesn't fix it:

Agent for this job:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=257527&view=logs&j=f80576d6-31b7-560d-8b9f-83fe28ca8786&t=88947367-a263-4838-a380-bd518b2c8198&l=4

Failure:
https://dev.azure.com/azure-sdk/internal/_build/results?buildId=257527&view=logs&j=f80576d6-31b7-560d-8b9f-83fe28ca8786&t=7ed5dfc6-5bd8-5180-ee3c-c917ed9782c4&l=9

Let me know if you don't have access, since we are in the MS tenant I can give you access.

[jtpetty]

@mitchdenny - Ok, for multi-repo checkouts, I think that it is expected that you have to explicitly
set the working directory since we do not create an "s" directory in this scenario. @jpricket - Please correct me if I am wrong with that statement.

I am curious why setting the working directory to either one of the repos you are populating or to the work directory itself does not fix the problem for both Windows and Ubuntu.

Can you set the work directory and rerun the Ubuntu pipeline with debug enabled and send me a link?

[mitchdenny]

Here is the repro on Ubuntu where even if I have a workingDirectory defined it still bombs out:

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=260112&view=logs&j=2e98f527-e589-587d-26d2-07597fe3ed5e&t=0f9c5414-019b-525a-288c-48b8db156974&l=9

FWIW, I don't think I should have to specify a working directory. There should always be some reasonable default. In the case of multi-checkout I would expect that to be Pipeline.Workspace/Agent.BuildDirectory. I thought that was the direction things where heading fro a consistency perspective anyway.

[mitchdenny]

THe snippet for the above build failure is:

  - stage: AgentProblemRepro
    dependsOn: ${{parameters.DependsOn}}
    jobs:
      - job: PwshOnUbuntu
        steps:
          - checkout: azure-sdk-build-tools
            path: azure-sdk-build-tools
          - pwsh: |
              Write-Host Hello world
            workingDirectory: $(Agent.BuildDirectory)

[jpricket]

I will try to repro this today and take a look.
Just to be clear, the above example is not a multi-repo scenario, since there is only a single checkout task.

[jpricket]

@mitchdenny I can't seem to repro the either error with the latest agent. Perhaps there's something missing for the repro. Can you share more of your yaml? I tried to mimic what you show in the snippet but I don't know what the other stages are doing.

[jpricket]

@mitchdenny Actually, I was able to repo it after a few more builds. I am not sure what changed. Perhaps, it started happening after I stopped changing things and it started re-using the same build folders. In any case, I think I have a repro now and will try to find a fix.
Thanks for reporting this issue!

[mitchdenny]

There are actually two issues. The first occurs with multi-repo where pwsh needs to have the workingDirectory set otherwise it blows up (this occurs on Windows). But there is another issue that happens on Ubuntu where it just blows up even if you do set the working directory. I suspect that they are related (the way using the named checkouts impacts variables etc).

[jpricket]

@mitchdenny - I think I have found the problem and I have a fix. It needs some additional testing, but should get in this week. I will work with @jtpetty to get the fix rolled out as soon as possible.

For others viewing this thread...
The problem shows up when you are checking out just one repo other than the "self" repo and you set the path on the checkout step. The reason for the problem is that we don't get the path from the correct repo when we are setting up the environment for other tasks. We get it from the self repo everytime which has the path pointing to a directory that doesn't exist.

[jpricket]

FYI the PR can be found here https://github.com/microsoft/azure-pipelines-agent/pull/2815

[mitchdenny]

Thanks for looking into the issue and getting a resolution. So given this will I need workingDirectory set all the time moving forward?

[jpricket]

@mitchdenny No, setting working directory was just a workaround for windows. The PR should fix the problem without any changes needed.

[jpricket]

The fix is in agent release 2.165.1 - https://github.com/microsoft/azure-pipelines-agent/releases/tag/v2.165.1