Arcade: Publish Assets failed with "PublishArtifactsInManifest.proj(51,5): error : File does not exist"

[vatsan-madhavan]

https://dev.azure.com/dnceng/internal/_build/results?buildId=299900

 Decompressing https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/sleet.packageindex.json
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19406.7\tools\SdkTasks\PublishArtifactsInManifest.proj(51,5): error : File does not exist. Remote: https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/flatcontainer/runtime.win-x64.microsoft.dotnet.wpf.dnceng/4.8.0-preview9.19407.26/runtime.win-x64.microsoft.dotnet.wpf.dnceng.4.8.0-preview9.19407.26.nupkg Local: C:\Users\VssAdministrator\AppData\Local\Temp\16f1c0ad-a75c-4f1e-955b-e9a4c1aab432\ea3c7c4f-9909-4f63-a57d-b01141049190.tmp
##[error].packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19406.7\tools\SdkTasks\PublishArtifactsInManifest.proj(51,5): error : File does not exist. Remote: https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/flatcontainer/runtime.win-x64.microsoft.dotnet.wpf.dnceng/4.8.0-preview9.19407.26/runtime.win-x64.microsoft.dotnet.wpf.dnceng.4.8.0-preview9.19407.26.nupkg Local: C:\Users\VssAdministrator\AppData\Local\Temp\16f1c0ad-a75c-4f1e-955b-e9a4c1aab432\ea3c7c4f-9909-4f63-a57d-b01141049190.tmp
  Uploading 1 items:
  Async uploading d:\a\1\a/BlobArtifacts\runtime.win-x64.Microsoft.DotNet.Wpf.DncEng.4.8.0-preview9.19407.26.symbols.nupkg

Can someone please take a look at this failure?

/cc @riarenas, @JohnTortugo
/cc @MattGal in case this is a build-infrastructure problem
/cc @dotnet/wpf-developers

[vatsan-madhavan]

Another rolling build failure in ".NET Core 5 Dev Channel Publish Assets" - slightly different failure-mode.

https://dev.azure.com/dnceng/internal/_build/results?buildId=299940

  GET https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/sleet.packageindex.json
  Decompressing https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/sleet.packageindex.json
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19406.7\tools\SdkTasks\PublishArtifactsInManifest.proj(51,5): error : Package already exists: runtime.win-x64.Microsoft.DotNet.Wpf.DncEng 4.8.0-preview9.19407.29.
##[error].packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19406.7\tools\SdkTasks\PublishArtifactsInManifest.proj(51,5): error : Package already exists: runtime.win-x64.Microsoft.DotNet.Wpf.DncEng 4.8.0-preview9.19407.29.
  Uploading 1 items:
  Async uploading d:\a\1\a/BlobArtifacts\runtime.win-x64.Microsoft.DotNet.Wpf.DncEng.4.8.0-preview9.19407.29.symbols.nupkg

[riarenas]

Looking

[vatsan-madhavan]

Also some of these failures could be related: https://github.com/dotnet/core-eng/issues/7310

[riarenas]

Still trying to understand the first failure mode. The second resulted from the two default channels that are set for the wpf-int repo.

The developer channel publish ran first, and published the packages.
The .NET core 5 channel ran second, and failed due to the package already being published to the same feed.

Going to look into the configuration of the sleet publish we are using. Finding an identical package already in the feed should not be a failure mode.

[vatsan-madhavan]

We will remove .NET 5 channel then until fixes can be made to the publishing logic. Resilient/reliable builds takes precedence over that.

[vatsan-madhavan]

I've removed .NET Core 5 Dev for now.

λ darc get-default-channels --source-repo dotnet-wpf-int
(130)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/master -> .NET Core 3 Dev
(140)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/release/3.0 -> .NET Core 3 Release
(298)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/master -> .NET Core 5 Dev
(382)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/dev/vatsan/yamlstages -> .NET Tools - Validation

λ darc delete-default-channel --channel ".NET Core 5 Dev" --branch refs/heads/master --repo https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int

λ darc get-default-channels --source-repo dotnet-wpf-int
(130)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/master -> .NET Core 3 Dev
(140)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/release/3.0 -> .NET Core 3 Release
(382)  https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int @ refs/heads/dev/vatsan/yamlstages -> .NET Tools - Validation

We can add it back once this has been resolved and the publishing tools have been separately validated to be able to publish simultaneously to 2 or more channels from the same branch.

/cc a few folks who tend to be interested in channel associations and flow of wpf into .NET 5: @mmitche, @zsd4yr, @AdamYoblick , @dagood, @rladuca, @ericstj

[vatsan-madhavan]

/cc @wtgodbe

[riarenas]

@JohnTortugo this is the issue with publishing to multiple channels only working for the first channel that runs.

I believe it's a matter of just passing the properties for "override if package already exists" and "skip if it's the exact same package"

[JohnTortugo]

@vatsan-madhavan is it OK if I copy the master branch and spin some test builds -- I intend to publish only to some test feeds?

[vatsan-madhavan]

What feeds & channels are we talking about?

Can you fork master off to another branch, associate with a new channel created for this purpose and publish to a feed that’s not WINDOWSDESKTOP or any of the well known feeds (make a new one; will require changes to wpf’s yml- you can’t use the default yml in master since it’s set up to publish to WindowsDesktop feed)? In short, would you take off the effects of the experimentation to an isolated test-bed ?

/cc @rladuca

[riarenas]

MSBuild is also seeing problems with publishing to more than one channel as part of the same build.

https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=2964450&view=logs&j=4ce5c9fb-70c0-5595-8a89-b54c663f40f4&t=53a7ba22-9b14-5b3a-0906-3b79578817b1&l=84

@johnTortugo have you been able to make any progress here?

[riarenas]

Do we perhaps need to change the settings here?

https://github.com/dotnet/arcade/blob/27a6926f6b1d4b36d86a236d2b05cdee3669589e/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifest.cs#L358-L362

[vatsan-madhavan]

Now that everyone has branched for .NET 5 and .NET Core 3.0 separately, is this really an urgent concern?

At least WPF won't need to map a single branch to 2 or more channels in the short-term... (at least I hope that's the case).

[JohnTortugo]

@riarenas I did some investigation on this in the afternoon and the conclusions/observations I've are:

• The check for existing artifact location doesn't prevent the task from trying to re-upload the artifact, only of trying to add the location again to the asset. Logic is here.

• Given the values AllowOverride = false and PassIfExistingItemIdentical = true I think we wouldn't see the reported error given this logic and since we are uploading the same file.

• As weird as it sound the points above make me thing that IsPackageIdenticalOnFeedAsync returned false for the asset..

• The version of Tasks.Feed used is pretty old.

Here are some infos that might help if you want to debug more:

WPF BAR Build ID is 22012
Target feed is: https://dotnetfeed.blob.core.windows.net/dotnet-windowsdesktop/index.json
SDK version used is: 1.0.0-beta.19406.7
Tasks.Feed version in use (judging by unzip of SDK): 2.2.0-beta.19367.5
Produced by BAR Build ID: 18758
Produced by Commit: e0adeeae15412035c1f68cec14ef9e6151356ca2
Name of task called: PublishArtifactsInManifest
Maestro.Tasks version: 1.1.0-beta.19401.2

[JohnTortugo]

Issue moved to dotnet/core-eng #7551 via ZenHub