Azure-pipelines-tasks: Build.StagingDirectory and Build.ArtifactsStagingDirectory are the same directory

[zooba]

When I run my build with the PublishBuildArtifacts 1.0.4 task, I get errors like:

Executing the powershell script: C:\VSOAgent\tasks\PublishBuildArtifacts\1.0.4\PublishBuildArtifacts.ps1
Preparing artifact content in staging folder C:\VSOAgent\_work\1\a...
Copying all files from C:\VSOAgent\_work\1\a\layout\Microsoft.PythonTools to C:\VSOAgent\_work\1\a\layout\Microsoft.PythonTools
The process cannot access the file 'C:\VSOAgent\_work\1\a\layout\Microsoft.PythonTools\BuiltinScraper.py' because it is being used by another process.

Previously, the task would prepare artifacts in a staging folder that was different from Build.StagingDirectory, which the docs suggest using for temporary build artifacts.

I've worked around it by inserting an extra directory level into my build outputs, so they don't conflict with where PublishBuildArtifacts tries to copy them, but ISTM that Build.StagingDirectory and Build.ArtifactsStagingDirectory should refer to different folders.

[JoelKunze]

I have noticed a change that may be related...
On the build agents I run, the Build.StagingDirectory changed from the documented value of "$(Agent.BuildDirectory)\Staging" to "$(Agent.BuildDirectory)\a"

This appears to coincide with an agent update that occurred about October 7th.
There is an additional folder named "b" created in the $(Agent.BuildDirectory) now as well, but it appears empty at least at the point our builds are failing due to the staging folder name change.

[chrispat]

You shouldn’t rely on the name of any given folder, that is what the variables are for. We shortened the folder names to take up less path space and try to mitigate long path issues on windows. Further we added the Build.BinariesDirectory back for customers who were using the old build system and wanted to have a single place to redirect binaries to during their build. By default we let MSbuild put binaries where you have defined them to go. Trying to override that blindly causes unexpected results and hard to debug issues.

From: JoelKunze [mailto:[email protected]]
Sent: Wednesday, October 14, 2015 10:03 AM
To: Microsoft/vso-agent-tasks [email protected]
Subject: Re: [vso-agent-tasks] Build.StagingDirectory and Build.ArtifactsStagingDirectory are the same directory (#614)

I have noticed a change that may be related...
On the build agents I run, the Build.StagingDirectory changed from the documented value of "$(Agent.BuildDirectory)\Staging" to "$(Agent.BuildDirectory)\a"

This appears to coincide with an agent update that occurred about October 7th.
There is an additional folder named "b" created in the $(Agent.BuildDirectory) now as well, but it appears empty at least at the point our builds are failing due to the staging folder name change.

—
Reply to this email directly or view it on GitHubhttps://github.com/Microsoft/vso-agent-tasks/issues/614#issuecomment-148059306.

[bryanmacfarlane]

https://github.com/Microsoft/vso-agent/blob/master/docs/workfolder.md

[zooba]

So Build.StagingDirectory was removed? That's fine, but we were just following the MSDN docs and not hardcoding paths "blindly".

It looks from the MD doc that we should change from Build.StagingDirectory to Build.BinariesDirectory.

[dmccaffery]

I just hit this issue as well. I was previously publishing build artifacts to BUILD_STAGINGDIRECTORY and then using the "Publish Build Artifacts" task to upload said artifacts.

Now, all is still fine and dandy with the shortened paths (I don't care what the paths actually are), but the "Publish Build Artifacts" task tries to copy files from BUILD_STAGINGDIRECTORY to BUILD_ARTIFACTSSTAGINGDIRECTORY in preparation for uploading. The trouble is, these two paths resolve to the same thing; so it tries to copy a file onto itself, which is why the error occurs.

If BUILD_STAGINGDIRECTORY is deprecated, thats perfectly fine. I think the real issue here is that the VSO agent isn't smart enough to realize that files are already in the expected place, and shouldn't try to copy them onto themselves.

[ericsciple]

@dmccaffery Thanks! Can you confirm this PR addresses your issue: https://github.com/Microsoft/vso-agent-tasks/pull/677

I tested it by putting $(Build.StagingDirectory) in the copy root input and verified it effectively skips the preparation copy portion but still performs the rest (e.g. upload if server drop).

[dmccaffery]

Confirmed; thx!

[zooba]

That wouldn't have helped me, as I don't just blindly publish the entire StagingDirectory, so files need to be move around.

(That said, I've already fixed it by switching to BinariesDirectory instead of StagingDirectory, so no need to worry about keeping me happy here.)

[ericsciple]

@dmccaffery sorry I had to revert my fix. It did not take into account some things. I talked this over further with some folks, and we've got another plan altogether. I'll circle back when we get closer to finishing this. For now the workaround might be to use the binaries as the intermediate staging directory (as zooba pointed out).

@zooba fyi as a result of this issue, it was realized the binaries directory is being deleted by the agent before each build. For incremental build it shouldn't be deleted... that's going to be fixed this sprint. Wanted to give you a heads up in case you're using incremental build (i.e. clean flag is set to false on the repo tab of the build definition) and you're expecting the agent to cleanup that folder before a build.

[dmccaffery]

@ericsciple I am already using that work around, so its no bother, really. take your time :)

[zooba]

@ericsciple I'd been assuming it wasn't being deleted, so I guess that explains the longer-than-expected incremental builds :) (not that I mind how long it takes when it's not blocking my dev machine)

[ericsciple]

The clean settings have been updated. Setting clean on the repo tab performs git clean -fdx if git is in your PATH (otherwise drops the repo folder). On TFVC it performs scorch. If any issues performing clean/scorch, then it falls back to dropping the repo folder and redownloading.

Additionally there's other options exposed by setting a variable build.clean to specific values. @TingluoHuang to elaborate. I don't see the variable documented so I'm not sure offhand the values.

[bryanmacfarlane]

We should add a help icon next to it to explain this

Sent from Outlookhttp://taps.io/outlookmobile

On Mon, Dec 21, 2015 at 9:28 AM -0800, "ericsciple" <[email protected][e-mail]> wrote:

The clean settings have been updated. Setting clean on the repo tab performs git clean -fdx if git is in your PATH (otherwise drops the repo folder). On TFVC it performs scorch. If any issues performing clean/scorch, then it falls back to dropping the repo folder and redownloading.

Additionally there's other options exposed by setting a variable build.clean to specific values. @TingluoHuanghttps://github.com/TingluoHuang to elaborate. I don't see the variable documented so I'm not sure offhand the values.

Reply to this email directly or view it on GitHubhttps://github.com/Microsoft/vso-agent-tasks/issues/614#issuecomment-166365770.

[ericsciple]

The build.clean options are defined here: https://msdn.microsoft.com/en-us/library/vs/alm/build/define/repository

The plan is to build a better experience in the UI rather than expose via a variable.