Nugetgallery: V2 Search endpoint does not return all versions when no filter is provided

[joelverhagen]

nuget.exe list -AllVersions expects the Search() OData endpoint to return all versions per ID (while still observing the includePrerelease=true parameter.

Reasoning:

1. nuget.exe list -AllVersions -source http://www.nuget.org/api/v2 does not return all versions.
2. MyGet does 😉
3. It is at least partially plumbed through from the gallery to the V2 search endpoint.
4. The V2 search endpoint (JSON) supports it with ignoreFilter=true:
   
   
   
   • https://api-v2v3search-0.nuget.org/search/query?q=packageid:newtonsoft.json&ignoreFilter=true&prerelease=true&take=100
   
   

I think there the Gallery should pass ignoreFilter=true to the V2 search service when there is no $filter provided to the OData Search() endpoint.

[skofman1]

https://github.com/NuGet/Home/issues/3441

[joelverhagen]

If it's a client bug, what Search() parameters on V2 should the client hit use to get all versions?

[skofman1]

@yishaigalatzer is this behavior intentional?
https://github.com/NuGet/NuGet.Services.Metadata/blob/master/src/NuGet.Indexing/NuGetSearcherManager.cs#L259

[yishaigalatzer]

I don't think it's intentional, Joel worked on this last he should know if this was intentional

[joelverhagen]

😓 I do not know if this was intentional. But again, I don't think this problem is in the search service. Search service supports returning all versions just fine:
https://api-v2v3search-0.nuget.org/search/query?q=packageid:newtonsoft.json&ignoreFilter=true&prerelease=true&take=100

It's the gallery that is not sending the right parameters (presumably during the Search() hijack).

[splatteredbits]

This is a pretty glaring, annoying bug. We can't discover different package versions with nuget's list command. We have an internal package management tool that allows users to specify a wildcard in the version they want, e.g. 4.* to get the latest 4.x.x.x version of a package. Please bump this up in priority and fix ASAP.

[splatteredbits]

Can someone point me to the place in the nuget.org source code where this bug is present? I can fix it and send a pull request, but I'd prefer not to spend a couple hours spelunking for the correct spot.

[joelverhagen]

This is probably the general that needs the fix:
https://github.com/NuGet/NuGetGallery/blob/7eed4f25888e8fb88794d4e2a2942d6fed115c7d/src/NuGetGallery/OData/SearchService/SearchAdaptor.cs#L201

I think the IncludeAllVersions needs to be true in this particular case:
https://github.com/NuGet/NuGetGallery/blob/7eed4f25888e8fb88794d4e2a2942d6fed115c7d/src/NuGetGallery/Services/SearchFilter.cs#L36

[splatteredbits]

Thanks for the pointers. Those do indeed seem like the right spots.

I've got a local nuget.org running on my developer computer, but am unsure how I registers so I can add packages to test my fix. Are there docs anywhere for this?

[splatteredbits]

I figured out how to get registered. I've uploaded three versions of a package. When I point nuget.exe at my local NuGet Gallery, it returns all three versions.

Is there a way to point my local NuGet Gallery at https://api-v2v3search-0.nuget.org/search/query ? I tried setting the Gallery.ServiceDiscoveryUri app setting, but that didn't work.

[joelverhagen]

Take a look at your web.config in NuGetGallery. You basically need to set these things:

<add key="Gallery.ServiceDiscoveryUri" value="https://api.nuget.org/v3/index.json" />
<add key="Gallery.SearchServiceResourceType" value="SearchGalleryQueryService/3.0.0-rc" />
<add key="Gallery.AutocompleteServiceResourceType" value="SearchAutocompleteService/3.0.0-rc" />

[splatteredbits]

Thanks. I've got the search working so it returns all versions, but it is also returning all pre-release versions even if I don't supply the -prerelease flag. It seems like https://api.nuget.org/v3/index.json always returns prerelease versions if ignoreFilter is true. I've tried always adding a prerelease=false|true query parameter (it usually only passed prerelease=true).

[joelverhagen]

This is expected behavior for ignoreFilter.

[splatteredbits]

Is there documentation for https://api.nuget.org/v3/index.json? I'd like to find a way I can return all versions without using the ignoreFilter parameter.

It seems that sometimes the search results are getting cached by the gallery. Is there anyway to turn this off?

Is there a better place to get help than in this thread?

[joelverhagen]

Our HTTP protocol has almost no documentation. This is not good, as you are seeing now! The particular V3 service you are interacting with is the search service, which is denoted by the SearchGalleryQueryService type in that index.json. The URLs for PROD are:

• https://api-v2v3search-0.nuget.org/
• https://api-v2v3search-1.nuget.org/

You can play with these endpoints directly although the base URLs should always be looked up in the v3/index.json for production systems.

This is the code in the search service that handles the gallery "hijack":
https://github.com/NuGet/NuGet.Services.Metadata/blob/42fe7dd040c7d08e4a2fd5d4e15489ff9272dca9/src/NuGet.Services.BasicSearch/ServiceEndpoints.cs#L53-L72

Looking at the code, it looks like it's not possible to ignoreFilter=true _and_ prerelease=false. There are two options then:

1. Just get all versions and allow the gallery to filter out the results (bad since paging will be nasty)
2. Enhance search service to support returning all versions and still support the prerelease=true flag.

[SurajGupta]

ETA on fix for this? It's not possible to automate the delisting of a package if you can't list all of them. Unfortunately the nuget.org UI makes you delist each version of a package separately. Anyways, seems bad form to be unable to list all versions of a package. Seems like one of the basic operations one would expect.