Nugetgallery: [Discussion] Deprecate NuGet.org OData queries

[anangaur]

Discussion issue for https://github.com/NuGet/Announcements/issues/37: Using OData to query NuGet.org repository is being deprecated

[forki]

"This does not impact the official legacy clients (nuget.exe 2.x or Visual Studio 2013) that rely on the V2 endpoints (https://www.nuget.org/api/v2)"

what does this mean? the old endpoints are still available after that date?

[skofman1]

@forki , currently nuget.org APIs support custom OData queries that are not part of the V2 protocol. For example:

https://www.nuget.org/api/v2/Packages?$orderby=LastUpdated&$skip=123456

https://www.nuget.org/api/v2/Packages()?$filter=(((Id ne null) and substringof('sometext',tolower(Id))) or ((Description ne null) and substringof('sometext',tolower(Description)))) or ((Tags ne null) and substringof('sometext ',tolower(Tags)))&$skip=4300

Queries of this sort will be deprecated.

[maartenba]

Would like to see https://github.com/NuGet/NuGetGallery/issues/3532 as an alternative to querying all packages on the OData feed.

[joelverhagen]

Here are some more examples of queries that will be blocked:
```
/api/v2/Packages()?$filter=(tolower(Id) wq 'PLACEHOLDER') and IsLatestVersion&includePrerelease=false&$top=80&includeAllVersions=false

Not supported due to the use of a function tolower.

/api/v2/FindPackagesById()?$orderby=Version desc&includePrerelease=false&$top=242&id='PLACEHOLDER'&includeAllVersions=false

Sorting by Version is not supported.

/api/v2/Packages()?$filter=substringof('PLACEHOLDER',Tags) eq true or startswith(Tags,'PLACEHOLDER') eq true&$orderby=Published desc&includePrerelease=false&$top=40&includeAllVersions=false

The substring function is explicitly blocked and startswith is not supported.

/api/v2/Packages()/$count?$filter=IsAbsoluteLatestVersion&includePrerelease=false&includeAllVersions=false&semVerLevel=2.0.0

No Id or Version filter is provided.

/api/v2/FindPackagesById()?$top=1&$orderby=LastUpdated desc&id='PLACEHOLDER'&semVerLevel=2.0.0

Sorting by LastUpdated is not supported.

[mpiggott]

@joelverhagen Is there a definitive list of what will be blocked? Or alternatively the list of allowed queries?

[joelverhagen]

@mpiggott, the list of queries that will be supported are those used by the NuGet client. Since V2 was never officially documented, it's hard to point to a list of well described endpoints and give you the full picture. I'll try my best here 😄. These are the endpoints used by the official NuGet codebase (and 3rd party implementations as well). They will not be disabled.

/Packages(Id='{id}',Version='{version}')
-> get metadata about a specific version

/FindPackagesById()?id='{id}'&semVerLevel={1.0.0 | 2.0.0}
-> get metadata about all versions of a package

/Search()?searchTerm='{search term}'&includePrerelease={true | false}&semVerLevel={1.0.0 | 2.0.0}&$skip={skip}&$take={take}
-> search for package metadata

There's also a specific pattern used by nuget.exe list as well as an old /GetUpdates endpoint no longer used by NuGet 3.x+ but these are used MUCH less commonly used than these three queries (on the order of < 0.1% of the total).

You can get a feel for what the official NuGet code is doing in this file:
https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.Protocol/LegacyFeed/V2FeedQueryBuilder.cs

Note there are some quirks in this code for package feeds that have some functions missing from the OData $metadata document as well as some other oddities for history reasons but these do not apply to nuget.org.

[jlstephens89]

@joelverhagen I'm a Product Manager for the Nexus Repository Manager product. I'd like to understand the impact this is going to have on our users. Is there a Nuget Gallery instance with these endpoints disabled that we can test against? If not are the changes on master so that we can stand our own instance up?

[joelverhagen]

@jlstephens89 our INT (integration/pre-production) environment has these endpoints disabled.

You can access INT here:
https://int.nugettest.org/

The API V2 URL is:
https://int.nugettest.org/api/v2

[jlstephens89]

The language in https://github.com/NuGet/Announcements/issues/37 suggests that all OData queries are going to be deprecated but that is not reflected in the responses in this issue (gave me a bit of a scare). Is this just the first step towards full deprecation? If so is there a more detailed plan anywhere?

[joelverhagen]

Yes, the language there is a bit confusing. Sorry about that. There is no plan to deprecate the V2 queries I mentioned above (https://github.com/NuGet/NuGetGallery/issues/7423#issuecomment-692206642). In other words, there is no plan to deprecate V2 queries currently in use by official clients. These "official" queries are well optimized and the surrounding scenarios are well understood so thinks like caching can be implemented safely.

Apologies on the scare and the lack of clarity. There are several compounding factors in this space that lead to this confusion:

1. The V2 API was never documented officially
2. OData is by definition a nearly unlimited API surface area, leading to countless query variants and massive DB load
3. Official client implementations drifted over time
4. Sparse or missing user agent information, making investigations very difficult (I do want to say Artifactory, Nexus, ProGet, and several other clients do well in this area).

If you wonder about a particular query pattern, please try it on our INT environment (https://int.nugettest.org/api/v2) or post a question here.

[jlstephens89]

Thanks @joelverhagen that's a helpful clarification.

[isaacabraham]

@joelverhagen is there any way we can see stats on those queries which might be deprecated (that are popular / being used today)? I'm not sure if we provide the user-agent in Paket to identify the client but it would be useful - especially if those queries being deprecated means we need to find a new way to achieve the same thing.

Thanks!

[joelverhagen]

Roughly 2% of V2 queries by request count are deprecated. By inspecting the user agent strings, we were able to identify 70% of unsupported traffic and have reached out to these parties proactively. The other 30% of unsupported traffic has no user agent at all or has a very generic value that does not allow up to reach out to the customer or know the context or user scenario requiring the query. We are hoping the tweet, brown-out, GitHub announcement, and future blog post will allow us reach this anonymous minority.

Paket itself accounts for a very small slice of V2 traffic -- 0.8% of all V2 traffic. Of the Paket traffic itself, only 0.56% of it is using unsupported query patterns. Unfortunately, our data about Paket user agents does not seem to include Paket client version so it's hard to tell whether these are just old client versions or if this is impacting new Paket versions. In my own tests with Paket, I was unable to reproduce any unsupported V2 queries. I posted an issue on the Paket repository providing notice of the deprecation process and specific Paket details (e.g. what the specific unsupported queries were). You can see it here: https://github.com/fsprojects/Paket/issues/3904.

@isaacabraham, please let me know if you have any additional questions! I hope you're doing well 😃.

[isaacabraham]

Thanks for doing that - that's very helpful. Really appreciated.

Likewise :-)

[joelverhagen]

Just so everyone on this thread is aware, we are currently conducting the announced brown-out period described here: https://github.com/NuGet/Announcements/issues/48. We are seeing about 1% of traffic being blocked (which is approximately what we expected) and database load down significantly.

[dotMorten]

I got hit hard by this tonight. Totally missed the announcement. I guess brown out was a good idea after all ;-)

[joelverhagen]

@dotMorten, sorry to hear you're impacted. The best alternative would be to switch to the V3 endpoint which returns JSON:
https://api.nuget.org/v3-flatcontainer/nuget.commandline/index.json
Docs: https://docs.microsoft.com/en-us/nuget/api/package-base-address-resource#enumerate-package-versions

Note that it's best to lookup the base URL in the service index:
https://docs.microsoft.com/en-us/nuget/api/overview#service-index
https://docs.microsoft.com/en-us/nuget/api/package-base-address-resource#versioning

The last version in the versions array is what you want. This is going to contain SemVer 2.0.0 versions (the V1 feed does not contain SemVer 2.0.0 packages) and be more reliable. The main problem with the query you had there is the $orderby=Version (this is unsupported in the official API surface area).

Also note that nuget.exe has an update -self option.

[dotMorten]

@joelverhagen Thanks. It's alright. Better a brown-out than a complete surprise. I opted for a much simpler solution that doesn't really require odata. I think the solution I had was totally overkill and way too generic really.

[joelverhagen]

Just so everyone on this thread is aware, we are currently conducting the second phase of the announced brown-out period described here: NuGet/Announcements#48. The first phase, consisting of three 4-hour windows was completed last week with only a single issue reported: https://github.com/NuGet/NuGetGallery/issues/7423#issuecomment-725188546.

For the next 48 hours, deprecated OData requests will be blocked on nuget.org.

[joelverhagen]

Note that we have halted the schedule brown-out 4 hours and 10 minutes early due to unexpected impact on a 3rd party partner. There was no other known impact, aside from https://github.com/strongbox/strongbox/issues/1960 which I have commented on.

If you were impacted by this 2 day window of temporarily disabled V2 queries, please feel free to comment here and I will try to help or answer your questions.

[joelverhagen]

This was completed on March 9th, 2021. We have not heard any complaints since then. Good night, sweet prince!