Arcade: Packages are disappearing from dotnet-core feed after being published

[vatsan-madhavan]

Not a dup of #2936

Can someone take a look please?

https://dev.azure.com/dnceng/internal/_build/results?buildId=300660

##[error].packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19407.9\tools\Tools.proj(0,0): error NU1603: Tools depends on Microsoft.DotNet.SignTool (>= 1.0.0-beta.19407.9) but Microsoft.DotNet.SignTool 1.0.0-beta.19407.9 was not found. An approximate best match of Microsoft.DotNet.SignTool 1.0.0-beta.19408.2 was resolved.

[riarenas]

The packages for that Arcade build were published here: https://dev.azure.com/dnceng/internal/_releaseProgress?_a=release-environment-logs&releaseId=14159&environmentId=45623

Some repos were able to restore this version of the sdk successfully, but it's missing from the feed now.

@jcagme do you know if your work to delete some packages might have mistakenly taken this one out?

I'm going to re-add this build to the channel, and that should take care of re-publishing the SDK and all related packages, but I'll toss this into FR for investigation into what happened with the package originally

[riarenas]

https://dev.azure.com/dnceng/internal/_releaseProgress?_a=release-pipeline-progress&releaseId=14235 is running and should publish the packages again.

[vatsan-madhavan]

I restarted the build assuming you had already republished them - my error. Feel free to abandon the build - I'll let you retry the build when you are ready.

[dagood]

Does this also track the same happening with Microsoft.DotNet.Arcade.Sdk? Hit it on my dev box, and @rainersigwald hit similar on macOS (see Teams channel):

error : Unable to find package Microsoft.DotNet.Arcade.Sdk with version (= 1.0.0-beta.19408.13)
error :   - Found 1747 version(s) in dotnet-core [ Nearest version: 1.0.0-beta.19409.1 ]
error :   - Found 0 version(s) in nuget.org
error :   - Found 0 version(s) in dotnet-coreclr
error :   - Found 0 version(s) in dotnet-windowsdesktop
error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found.

[riarenas]

Yeah. Both signtool and the arcade sdk are both published the same way. And also seem to be mysteriously becoming unpublished somehow.

[dagood]

Core-Setup CI/official build is blocked by this.

It looks like it's happened to CoreFX's packages as well, from this dotnet/standard update PR: https://dev.azure.com/dnceng/public/_build/results?buildId=302530&view=logs

...Microsoft.Extensions.DependencyModel.csproj(0,0): error NU1603: Microsoft.Extensions.DependencyModel depends on System.Text.Json (>= 5.0.0-alpha1.19409.1) but System.Text.Json 5.0.0-alpha1.19409.1 was not found. An approximate best match of System.Text.Json 5.0.0-alpha1.19409.2 was resolved.

[riarenas]

https://dev.azure.com/dnceng/public/_build/results?buildId=302500&view=results succeeeded,

Restore completed in 849.89 ms for F:\workspace.1\_work\1\s\.packages\microsoft.dotnet.arcade.sdk\1.0.0-beta.19408.13\tools\Tools.proj.

but then 20 minutes later https://dev.azure.com/dnceng/public/_build/results?buildId=302544&view=results failed because it couldn't find the arcade SDK version:

error : Unable to find package Microsoft.DotNet.Arcade.Sdk with version (= 1.0.0-beta.19408.13)

@MattGal Could FR take a look at this soon?

[dagood]

@dotnet/dnceng for awareness--some Arcade and CoreFX packages appear to be removed from the dotnet-core feed some time delay after they're published, breaking builds.

[vatsan-madhavan]

More failures

https://dev.azure.com/dnceng/internal/_build/results?buildId=302540

/cc @dotnet/wpf-developers

[mmitche]

@jcagme Is this somehow related to your deletion?

[riarenas]

@jcagme Is this somehow related to your deletion?

That is my current theory too. I don't think his scripts are actively deleting the packages, but it's messing with the registrations in the nuget feed, and that causes it to not be available.

The bits are still in the storage account:

and the registration for the version is also there:

But maybe the main feed index is getting reverted somehow?

[dagood]

That would make sense to me: if Sleet is failing to keep the feed locked (or respect the lock), maybe some builds are being published while the delete operation isn't complete yet, and when the delete completes, maybe that ends up reverting by applying changes onto the index as it was when the delete began.

[borgdylan]

This is also breaking builds of MSBuild. I experienced it on my Linux box and so did one of the MSBuild team members.

[jcagme]

@emgarten it seems that deleting and publishing is messing the registration indexes causing some of the published packages to not have an index entry. We are on 2.2.139 of DotNet.SleetLib. Worth mentioning we don't normally delete stuff, we are just trying to remove some packages from the feed since they are causing some noise. Is there a way to un-register a package? That would also mess with the registration indexes so I don't know what's better

[nguerrera]

A bunch of core-sdk PRs are failing too:

https://dev.azure.com/dnceng/public/_build/results?buildId=301957

FYI @livarcocc

[nguerrera]

we are just trying to remove some packages from the feed since they are causing some noise.

What noise exactly?

Can we restore all deleted packages until we figure out a better approach?

[jcagme]

we are just trying to remove some packages from the feed since they are causing some noise.

What noise exactly?

Can we restore all deleted packages until we figure out a better approach?

Packages been deleted should not have anything to do with the current state. https://github.com/dotnet/core-eng/issues/7204 for reference

[dagood]

The packages that were intentionally deleted shouldn't do this, but I think @nguerrera is suggesting reuploading (maybe just re-indexing?) the unintentionally deleted packages. (And if they can't be distinguished, all of them.)

[jcagme]

Yes, I think @riarenas is already (looking at) doing that and I'll also jump in to get this sorted.

[nguerrera]

Thanks!

[mmitche]

I wonder if you push a single package from each of the 1900-ish we have on the feed whether that will resolve it.

What is the plan for still deleting the packages though? Do we not delete them then?

[dagood]

https://github.com/dotnet/core-eng/issues/7204 is not marked critical, I suggest we treat figuring out how to safely delete things as part of that issue (or split it into its own as a prereq) and don't proceed. I'm not aware of any other ongoing deletes.

[jcagme]

I wonder if you push a single package from each of the 1900-ish we have on the feed whether that will resolve it.

We know when this started happening so I'll write a thing to download the packages that were published after the start time and re-publish them. This will only update the indexes given that the bits exist

What is the plan for still deleting the packages though? Do we not delete them then?

Yes, this will need to happen but after this I'll spend some time in figuring out if there is a better/safer way of doing it to avoid breaking the world

[emgarten]

I'll go through the code to see where this could be happening. As long as the feed is locked there shouldn't be any conflicts between deletes and publishing.

[markwilkie]

Thanks @emgarten !

[emgarten]

I went through the delete script with @jcagme and we found that the cache was being reused between lock operations, which lead to delete working against older files, hence new packages were getting dropped.

I've added a helper to sleet to run delete operations in a single batch to avoid this and will also add some checks to that the virtual file system to help prevent this happening in the future if the library is used this way.
https://github.com/emgarten/Sleet/pull/84

The problem here is limited to the current delete script that was being used, everything else is working correctly.

[markwilkie]

Thanks @emgarten ! cc/ @mmitche

[jcagme]

Thanks for the quick turnaround @emgarten! I'll use the updated Sleet to continue with the cleanup.

At this point all repos should be unblocked since @riarenas re-published Arcade packages and I updated the registration indexes that were in a bad state.

Please, either reopen this issue or create a new one in case you still hit this error