Nugetgallery: Surface PackageType query parameter in the NuGet.org search API

[karann-msft]

Users can specify a package type - https://docs.microsoft.com/en-us/nuget/create-packages/set-package-type

It should be possible to search for all DotnetCliTool packages on NuGet.org

Proposal

• Expose a new query parameter called packageType
• This will map to the packageType in the nuspec
• For example, the following query would return all dotnet CLI tools on NuGet.org that match the search term
  https://azuresearch-usnc.nuget.org/query?q=codegenerator&packageType=DotnetCliTool

[wli3]

• Looks like a new end point, what is the schema? it is the same as
  https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource ?

• Can I query call DotnetCliTool, basically without any "q"

• Is the API throttled?

• Is the result paged?

• Is there an easy way to list all latest version (we don't to display the whole list of versions)

[wli3]

cc @kathleendollard

[loic-sharma]

Keep in mind that all my answers may change as we haven't done a proper design yet.

• Looks like a new end point, what is the schema? it is the same as
  https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource ?

Current plan is that this will be an addition to that resource, yup.

• Can I query call DotnetCliTool, basically without any "q"

Yes. This would likely get you the most "popular" .NET CLI tools.

• Is the API throttled?

Not at this time, but this may change. Why?

FYI, we document our rate limits here: https://docs.microsoft.com/en-us/nuget/api/rate-limits#package-search

• Is the result paged?

Yes. Each result is the metadata for the latest version of a package (plus a list of all its versions). We return 20 results by default, but you can increase this using the take query string. We don't recommend paging through the search endpoint to get a complete list of results though (we use the equivalent to sharded Elasticsearch, which does not support stable sorting). Instead, you should use the catalog resource.

• Is there an easy way to list all latest version (we don't to display the whole list of versions)

I'm not sure I follow. If you mean, "how can I find a the latest version for ALL .NET tools?", use the catalog resource. If you mean, "what is the latest version for a given package?", then:

1. The search results' version field is the latest version for that package. NOTE: Make sure you properly set the semVerLevel and prerelease query string parameters, otherwise you may have incomplete results.
2. If all you have is a .NET CLI tool's package id, you can either use use the package content resource to find the latest version for the package. NOTE: This API includes unlisted versions. If this is unacceptable, consider using the package metadata resource instead.

Please let us know if you have any other questions or suggestions! Also, I'm curious, what is the experience you plan to build?

[wli3]

I will later write up a more concrete plan.

The current draft is to have a _dotnet tool search blah_ and we will display all tools where packageid/summary/description/tags contain blah. They will be displayed as a form.

and we will also have a "detailed" mode to list out more details like all available versions since it is hard add too much information in a form.

One additional question is, is there an easy to search multiple fields(packageid/summary/description/tags)?

[loic-sharma]

The current draft is to have a _dotnet tool search blah_ and we will display all tools where packageid/summary/description/tags contain blah. They will be displayed as a form.

How will this work if the user's nuget.config file has multiple sources?

Have you considered something like dotnet search --tools blah instead? I'd expect users want to search for packages and templates too.

One additional question is, is there an easy to search multiple fields(packageid/summary/description/tags)?

The search endpoint searches across multiple fields by default on NuGet.org. On NuGet.org you have to craft a special query like packageid:newtonsoft.json to scope the search to a single field. Note that the query syntax is not part of the documented NuGet protocol, so different NuGet servers may have different query syntax. I would suggest passing the user's query directly to the search service and letting the service interpret it.

I will later write up a more concrete plan.

Sounds good! Please let us know if you have any more questions, the NuGet protocol can be a little tricky 😊

[wli3]

How will this work if the user's nuget.config file has multiple sources?

We will focus on nuget.org for now, considering search by package type is not a standard

Have you considered something like dotnet search --tools blah instead? I'd expect users want to search for packages and templates too.

Just my opinion, I think we should make it generic, since VS has package manager to discover package.

I would suggest passing the user's query directly to the search service and letting the service interpret it.

We discussed that. If the user has the knowledge of crafting queries. It is better and faster for them to just use the web browser. We want to solve tools discoverability problem first.

At the same time, it is possible for SDK to craft a query to search multiple fields? Could you give me an example?

cc @KathleenDollard @livarcocc

[KathleenDollard]

@loic-sharma Do you have a list of the fields that are searched by default? That may be a great solution for us. Would the results be the same as if the user typed "Blah" in the search field of the website?

We aren't currently planning any source other than NuGet.org because we understood this API to be NuGet.org specific where another package source could easily use a different API, or not implement it. Hopefully implementing this on NuGet,org will help start a conversation that can lead to a reliable approach across the ecosystem. If you have a different perspective on alternate sources, I'd love to discuss.

As for syntax, the current CLI syntax guidelines, result in

dotnet tools search blah

Designing this to allow other searches may be a good idea. We would probably use the syntax:

dotnet package search

Currently new breaks the syntax guidelines in a few ways, so we'll have to work with them if we include a template search in this work.

[loic-sharma]

At the same time, it is possible for SDK to craft a query to search multiple fields? Could you give me an example?

Yes, we search on multiple fields by default. For example, searching for "unittesting" matches the package "Moq" because it has the tag "unittesting": https://azuresearch-usnc.nuget.org/query?q=unittesting

Do you have a list of the fields that are searched by default?

This list changes as we evolve our search service. You can get a rough sense by looking at what is labelled IsSearchable on our repo. Today, this includes:

• Package authors
• Package description
• Package version
• Package id
• Package summary
• Package tags
• Package title
• Package owners

We will focus on nuget.org for now, considering search by package type is not a standard

We aren't currently planning any source other than NuGet.org because we understood this API to be NuGet.org specific where another package source could easily use a different API, or not implement it.

Considering that you can install tools from any NuGet source, I think it'd be ideal if you can search for tools on any source.

At this time, the NuGet team does not intend to make this API NuGet.org specific. The NuGet protocol includes a "service index" that lets clients discover what capabilities the server has. I would imagine that package type filtering support should declared using the service index. Please keep in mind that the NuGet team hasn't formally discussed the new protocol yet, so this may all change.

As for syntax, the current CLI syntax guidelines, result in...

I did not know we had CLI syntax guidelines, thanks for sharing!

Maybe we should setup a meeting to discuss this?

[joelverhagen]

I agree with @loic-sharma. Seems like we (nuget.org) need some help understanding the big picture that the CLI has. I think you have been working with @karann-msft up till now. Could we get in a room and discuss before we start diving into how the protocol should be shaped ?

/cc @karann-msft - PM
/cc @skofman1 - server lead
/cc @ryuyu - he will be working on the code changes

[KathleenDollard]

Great! Let's get together on this! Meeting notice sent.

[wli3]

Today's meeting note

• Query sample https://azuresearch-usnc.nuget.org/query?q=codegenerator&packageType=DotnetCliTool is good. Note, packagetype will be a separate query parameter.
• SDK will directly hit the web REST API. Although SDK would use NuGet.Protocol if packagetype API is accessible from the client package.
• The search term (after q=) will search fields like nuget.org (package id, description etc) and SDK will not do any customization on top it.
• The return schema will be the same as https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource#search-result with the addition of package type and package type version. And the return fields will not change according to the query (do not need to worry about missing Required fields when parsing).
• If SDK will pass through the query is TDB
• Package Type
  
  
  
  • Should be able to search PackageType version. The package type version is a System.Version. Search query detail TBD.
  
  
  • Will search all PackageType version when not specified.
  
  
  • PackageType and PackageType version will be in the search result.
    
    Side notes:
  
  
• compatibility search (TFM - Runtime search) is in backlog
• Verified field will not be able to filter
• Obsolete and vulnerability field are in discussion, need a separate meeting
• The metadata will only show the latest package version’s

[KathleenDollard]

In addition to @wli3 's notes:

• When Deprecated and Vulnerability are available in output, we suggest boolean values and other ("normal") channels used for additional information.
• Search will be based on the latest versions package type

[joelverhagen]

/cc @zkat this is the epic for the server-side change to support package type filtering.