Nugetgallery: Packages should be immediately available for download even before indexing completes

[anangaur]

There is a gap w.r.t. npm in this space. Something that can be improved:

Action | NuGet.org | Npmjs.org
-- | -- | --
Publish a package from CLI | Takes ~5 mins for validation | Immediately published
Package page availability | Immediate | Immediate
Does it show on owner's profile page | No* | Yes
Available in search results immediately? | No* | No
Installation possible with full ID | No* | Yes

[skofman1]

The intention is to make the package available for download in VS immediately / nuget.org download link?

[matthid]

I have run into this today and like to add this feedback regarding ~5 mins:

Yes this is a problem

[JohanLarsson]

I think my record is 17 hours. IIRC when publishing packages with dependencies the packages must be validated before each other, then we could be looking at days.

[anangaur]

Thanks for reporting this issue. While most of the times the validation is done within 5 mins and so is the case for indexing, there are times when we see a spurt in package submissions and hence the delays. We investigate each such occurrence and make our service resilient. /cc: @skofman1 for this particular issue.

@JohanLarsson can you also send the package details and the time when this happened to your package? Helps in RCA.

[anangaur]

@skofman1 I am referring to nuget being able to install/restore when the package ID and version is known.

[forki]

@anangaur 5min? Are you sure? I personally have never seen it processing in that time. Unfortunately it's always much much longer. Also many people report it on Twitter. Latest: https://twitter.com/theimowski/status/957741946809847808

Regarding installation: nuget broke paket's feature to install freshly uploaded packages recently. That's a bit unfortunate

[matthid]

I am referring to nuget being able to install/restore when the package ID and version is known.

Yes exactly, This would be a workaround when such scenarios happen. That's why I reported this here as it is just another good reason to do it.

Regarding the current delay, should we report this in a new issue? (Note: The package above is still validating....)

[anangaur]

@forki I am talking about validation only. Indexing take its usual time as before in addition to the validation time.

The feature you talk about, is this because of validation delay? From V2 feed?

[forki]

paket was able to see packages and download them directly after upload. Some changes in nuget.org V2 and V3 rendered that broken.

[anangaur]

@matthid the current delay issue is already being looked into. /cc: @cristinamanum @skofman1

[anangaur]

@forki I think previously the download was possible through the V2 feed. This issue plans to address the ability to download before indexing is complete for both the feeds.
However the validation time will still be blocking the download. To clarify, currently after the package is uploaded the delay is a combination of validation + indexing. Validation will continue to block the availability of package. But once validation is complete, download of package with ID and version should bot block on indexing.

[matthid]

By the way: What does validation actually do? Is this official information?

[anangaur]

@matthid As the name suggests it looks for the sanity of package. Among other things, this step scans for security issues in the package.

[matthid]

This does also mean that if the verification fails the package-id will be released and can be pushed again?

[forki]

How can that possibly take longer than let's say a minute on azure
functions with unlimited scalability?

Am 29.01.2018 17:07 schrieb "Matthias Dittrich" [e-mail]:

This does also mean that if the verification fails the package-id will be
released and can be pushed again?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/5287#issuecomment-361294712,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNMuySNvfGJ6uhzewtyuXxNdtGMfWks5tPezZgaJpZM4RiGVo
.

[jameswrightrg]

This is still a problem. The main problem is that new versions of packages are available in the search - so Paket will detect the new version and try and fail to download it. The effect of that is that after new packages are released, our build process is unusable until it finishes indexing.

If it's too hard to make packages immediately available, how about excluding them from the search until they are available?

[forki]

The problem is a long delay between packages are listed and packages are downloadable. This is not only a problem with Paket but also with dotnet tools and probably even dotnet add.

[skofman1]

@jameswrightrg , @forki , packages show up in nuget.org V3 protocol search only after they were indexed are available for download. What feed are you using? Is it https://api.nuget.org/v3/index.json ?

[forki]

I'm almost certain that's not actually the case. With v2 protocol there is definitely something leaking and with dotnet tool there is also something wrong.

[agr]

Please note, that V2 and V3 use different storages: if you are using V2 to discover the existence of a package, you should use links provided in V2 responses to download that package. V3 lags behind V2 because by design it essentially ingests packages from V2.

[forki]

Of course those things use the links they discover. The problem is that those links don't have the downloads available.

[loic-sharma]

The problem is that those links don't have the downloads available.

I'm not sure I follow. The V2 API (like https://www.nuget.org/api/v2/Packages(Id='newtonsoft.json',Version='12.0.3')) has the download link on the content element's src attribute.

✔ V2 download: https://www.nuget.org/api/v2/package/Newtonsoft.Json/12.0.3
❌ V3 download: https://api.nuget.org/v3-flatcontainer/newtonsoft.json/12.0.3/newtonsoft.json.12.0.3.nupkg

Note: if possible, you should prefer the V3 API as V2 has performance issues.

[forki]

Yes the links just 404.

[loic-sharma]

Which link 404s? The V3 download during ingestion? If so, please refer to @agr's comment:

Please note, that V2 and V3 use different storages: if you are using V2 to discover the existence of a package, you should use links provided in V2 responses to download that package. V3 lags behind V2 because by design it essentially ingests packages from V2.

In other words, the ingestion flow is:

1. Validate package
2. Ingest in V2
   
   
   
   1. Upload package to V2
   
   
   2. Mark package as listed
   
   
3. Ingest in V3 (upload to V3, add to search, etc..)

Therefore, using V3 APIs to download a package you discovered using V2 APIs may result in a 404. You should download a package using V2 APIs if you discovered it using V2 APIs.

[forki]

No I don't have a repro since stuff is only 404 for short timespan (couple of minutes). And no we don't switch from v2 to v3 midflight.