Arcade: Intermittent NU3004 errors (fixed?)

[tmat]

Time to time I see errors like this one in the Arcade CI build:

error NU3004: Package 'vswhere 2.6.7' from source 'https://api.nuget.org/v3/index.json': This repository indicated that all its packages are repository signed; however, this package is unsigned.

Doesn't make much sense to me as it happens only sometimes.. The package is definitely signed.

Failure example:
https://dev.azure.com/dnceng/public/_build/results?buildId=172551

[tmat]

@rrelyea @nkolev92 Any ideas?

[nkolev92]

Can you point to which step has this warning?

What's the version of the tooling.
I'm not sure if there's any way for this to be hit beyond an actual server inconsistency.

@loic-sharma @dtivel might have better ideas.

[nkolev92]

Chatted with @loic-sharma.

Even though I can't find the logs I'd imagine it's possible that you are hitting this bug.
https://github.com/NuGet/Home/issues/7997

Basically, if on the same machine, multiple client version of dotnet.exe are being used in a span of 30 minutes you can run into it.
Basically I think you need one of 2.1.500/2.2.100 (not latest patch) to run after a 3.0.0 P4-P5/2.2.300/2.1.700.

@loic-sharma also pointed out that we should probably have a better description of this scenario somewhere. Ideally a nuget tracking issue and/or a docs update for the NU3004 warning.
//cc @rrelyea

[dtivel]

@tmat, there are 2 known scenarios where you might get this error erroneously:

1. You're running a NuGet 5.0 client on the same machine as a NuGet 4.7-4.9 client. A bug in HTTP caching would enable a 4.7-4.9 client to incorrectly use a cached 5.0 resource. A fix should be available in NuGet 5.0.2 and 5.1. I worked with @MattGal on this, and I wonder if this is another report of the same issue. One workaround is to clear just the HTTP cache before calling NuGet (e.g.: `nuget.exe locals http-cache -clear).
2. You're using NuGet.exe on Mono to restore signed packages. This was a bug; signed package verification should have been disabled when NuGet.exe runs on Mono. A fix should be available in NuGet 5.0.2 and 5.2.

[brianrob]

Will a subsequent Arcade update take a build of nuget that will fix this? Or is there a good way to run the workaround in an Arcade enabled repo to fix this? I'm seeing this in the optimization repo in our official runs in AzDO and am not sure how to modify the build to implement the workaround proposed above.

@adiaaida

[markwilkie]

Yes - as soon as preview 5 goes out the door, we'll update in Arcade.

cc/ @riarenas @MattGal

[brianrob]

Perfect. Will wait for that. Thanks.

[riarenas]

After discussing with @mmitche and @markwilkie, we decided it's worth updating now to help out with this. Once #2663 is merged, the next dependency update from arcade will update to that version if the repo is using anything older.

[riarenas]

Haven't seen reports of this popping up by repos that got their sdk versions updated by Arcade, or moved manually.

[rainersigwald]

I think I just hit this in a PR build for the Arcade update that allegedly contains the fix: https://dev.azure.com/dnceng/public/_build/results?buildId=182476&view=logs&j=6772b272-ab79-59eb-8cca-405ab5c4d2b5&t=77aa7d44-8ac6-58f9-07fa-1bd408749035&l=104

[riarenas]

Clearly I jinxed it.

Reopening to investigate

[rrelyea]

Is arcade machine wide? or side by side?
if it is side by side, did this machine also have old-arcarde builds?

[MattGal]

Arcade dependencies are themselves NuPkgs and branches which have not been updated can bring along the old stuff whenever they want, so the answer is "almost certainly yes"

[MattGal]

@rrelyea I take that back; looking at Rainer's build, one thing to note is that since this is the Hosted 16.04 pool, and given the logs show it using workspace "1", odds are this machine had nothing else cached anywhere.

[rrelyea]

i don't understand workspaces that you refer to.
realize, the problem that causes a new problematic version to impact older versions that might be fine -- is transfered through the nuget http cache. run dotnet nuget locals all --list
(see where it says the http-cache is)

clearing that location between builds would be one way to reduce this problem.

(sorry)

[MattGal]

@rrelyea what I'm saying is my understanding of the Hosted pools is that machines are only used once then recycled, so "some older build polluted the cache" is nigh impossible, unless the VSTS agent restored some nuget-related Nupkgs as part of its own setup. "Workspaces" are just the directories sources go into on these agents (monotonically increasing), so every one of them being 1 indicates starting from a "clean" machine.

I'm verifying this understanding separately and will update if I am more certain. (edit: looking at a long history of "agent 2" it's clear, there's never been a build run before on any of the hosted 16.04 agents)

[rainersigwald]

In my case, it should be a fresh VM running just my build, with the latest Arcade, since I'm using the Azure Pipelines Hosted pool.

[riarenas]

@MattGal Can someone from FR take a look? I believe Rainer's report is the first instance we've seen since moving the rest of the repos to a newer SDK.

[MattGal]

@riarenas yes I'll get someone on this this morning.

[MattGal]

@rainersigwald after monitoring this for some days with no repro, I'm closing this out. All recent failures in https://dev.azure.com/dnceng/public/_build?definitionId=86&_a=summary do not show this symptom.

Feel free to reactivate if I am mistaken or simply ping me on Teams