Nugetgallery: Users should be able to see package repository metadata

[joelverhagen]

Overview

Since August 2016, the NuGet package manifest (.nuspec) has supported the <repository> XML element. This is an optional metadata element with two optional attributes: type and url. Presumably, the purpose of this element is to communicate to package consumers the version control type and location of the source code used to produce the package.

Currently, I see no NuGet documentation about this property and no support for it in the various NuGet UIs (Visual Studio and the nuget.org web gallery).

This metadata adds additional valuable information about a package that augments the existing <projectUrl> package metadata field. Project URL seems to me like a "product" or "company" specific website and repository URL is where the source code (and maybe even bug tracker) lives. If a project has project site that is different than their GitHub repository, this differentiation is helpful to package consumers.

This property is already widely adopted on nuget.org but is not surfaced in any UI or used for any scenario. Let's give an easy win to our package consumers!

History

As far as I can tell, the repository metadata has the following history:

• 2015-07-15: Repository type is added to the project.json schema (PR)
• 2015-07-25: the repository node is added to project.json for dnu sources (announcement)
• 2015-09-14: dnu sources and the repository node are removed (announcement)
• 2016-01-17: Repository url is added to the project.json schema (PR)
• 2016-04-27: Repository URL and type are added to .NET CLI under packOptions (PR)
• 2016-05-02: Repository fields' move to packOptions is announced (accouncement)
• 2016-08-12: NuGet adds <repository> to the .nuspec and to the .csproj (commit)
• 2016-09-22: .NET CLI migrate (.xproj to .csproj) adds support for packOptions (PR)
• 2016-10-03: NuGet merged MSBuild new support to dev branch (PR)
• 2016-06-20: NuGet gallery added schema support for repository URL (PR)

This is an odd history that indicates decent support and documentation when it was initially implemented by the ASP.NET but has since fallen to the wayside when it was adopted by the NuGet team.

Usage on nuget.org

I walked the entire nuget.org catalog (source) and gathered the following information concerning the real life usage of <repository>. This data is up-to-date as of November 1, 2017.

Some high level points about packages with <repository> metadata:

• A total of 35,231 packages have been published with <repository> metadata.
• There are 6,539 distinct package IDs that use repository metadata.
• The first package on nuget.org to use this feature was contentful.csharp 0.5.0-alpha 🥇. This package was published on November 18, 2016.
• In October of 2017, 6,381 packages were published. That's up from 6,173 in September and 5,877 in August. Usage has gone up every month since it's release!
• 73% of packages (25,677) have the type attribute.
• 99% of packages (35,006) have the url attribute. 225 don't have a url attribute.
• 99% of packages (34,817) have a URL value that is parsable as an absolute System.Uri. 189 are not parsable.
• 91% of the packages (23,362) with a type attribute have git (case insensitive).
• The next most common type attribute balue is GitHub (1,977 packages).
• 95% of the packages (33,275) with a url attribute have "https://" URLs.
• 1% of the packages (331) with a url attribute have "http://" URLs.
• 3% of the packages (1,206) with a url attribute have "git://" URLs.

The top 10 packages that have <repository> are:

ID | Version | Downloads
-- | -- | --
Newtonsoft.Json | 10.0.3 | 3,816,231
Castle.Core | 4.1.1 | 918,532
Castle.Core | 4.1.0 | 497,535
Microsoft.Rest.ClientRuntime | 2.3.8 | 460,118
Microsoft.Extensions.Primitives | 2.0.0 | 368,589
Microsoft.VisualStudio.Web.CodeGeneration.Tools | 1.0.0 | 368,363
Autofac | 4.6.0 | 359,598
Microsoft.Extensions.DependencyInjection.Abstractions | 2.0.0 | 351,539
Microsoft.Extensions.Options | 2.0.0 | 341,419

Proposal

Document the thing!

There is a huge variety of type and url values. We should document:

1. What this metadata is intended for
2. What values are acceptable

In my opinion we should strongly suggest http:// or https:// URLs that are intended for viewing in web browsers, as opposed to, say, .git URLs: https://github.com/NuGet/NuGetGallery.git. This allows us to make clickable links in VS or in the browser, after all.

git:// links are nice because you can probably run git clone directly on them, but this is still achievable with http:// or https:// links by just navigating there in your browser and find the inevitable "copy clone command" button (e.g. on GitHub).

Show the thing!

Let's make the repository URL clickable in the NuGet UIs!

If the repository URL is http:// or https://, show this link, just like the project URL.

An analogous change should be made by the VS client.

Thoughts?

[joelverhagen]

/cc @anangaur @karann-msft @skofman1 @rrelyea

[jainaashish]

I liked this idea to showing repository urls in NuGet.org and at some point in time in VS client too. This will also help consumers to know more about the package and if anything is wrong, then directly log an issue on their respective repository instead of filling a NuGet support request.

[emgarten]

Not to rain on this proposal, but here is an argument against adding a repository link.

In my opinion repository url was added to eventually get to a Switch PackageReference -> Project from Source feature, which is a common ask.

The repository has type=git not type=website, it just happens that for github the url works has both, but in my opinion it is wrong to open these urls in the web browser.

It should be possible to write a tool to read the nuspec, and if it has type git, it runs git clone on the url. If nuget.org were to treat this property as projectUrl2 then my guess is that a lot of packages would end up using this incorrectly and it would hard to create a tool like that.

I also think that there is minimal value in having 2 project urls, if the repository web site is important, the project url should link to it, or the site it goes to should link to it.

I'm not sure how useful it would be, but I would rather see nuget.org read the actual git respository and surface information about whether or not the project is still active.

[joelverhagen]

The repository has type=git not type=website, it just happens that for github the url works has both, but in my opinion it is wrong to open these urls in the web browser.

I don't have a strong opinion here, but there is definitely needs to be some documentation about what the value means. As it stands today, users are guessing.

[natemcmaster]

Hey @rrelyea @skofman1, is this still on your radar? In 2.1, aspnet packages include repository url and commit metadata. We'd love to see the gallery display this somehow. Is this proposal complete enough that you would consider accepting a PR if we could do the work?

[anangaur]

@natemcmaster On our radar for sure. Not up in the backlog yet. We will factor this into NuGet Package Debugging & Symbols Improvements

/cc: @diverdan92

[loic-sharma]

A customer created a PR: https://github.com/NuGet/docs.microsoft.com-nuget/pull/900

[diverdan92]

Documentation proposed: https://github.com/NuGet/docs.microsoft.com-nuget/pull/930

[MaximRouiller]

So we would need to push this forward. Here's how the landscape look in terms of package management.

NPM does it on the first page and even allow to split between the Homepage and the Repository URL.

If we look at Newtonsoft, even thought they had the attribute for a while now...

It still does not show up in the project description.

It doesn't look like something that is hard to implement since the nuspec doesn't need to be changed.

Is it something that can be PR'd?

/cc @spboyer

[natemcmaster]

@MaximRouiller good timing. I was thinking about this last night too and drafted this PR: https://github.com/NuGet/NuGetGallery/pull/6151

[MaximRouiller]

Pinging this to keep this alive.

We have a PR, we don't have any spec... I'd love an estimated timeframe on when we plan on getting this solved/merged.

[natemcmaster]

I'd also love to get the PR merged. There is an internal email thread going on this one, and it seems like there are two things holding this up:

• scope creep.
• it's unclear how to fix the database for already-uploaded packages. A RepositoryUrl column was added to the table a while ago, but it was never populated with data, so the column is always null.

[skofman1]

The work we decided to do here:

1. Accept @natemcmaster 's PR: https://github.com/NuGet/NuGetGallery/pull/6151
2. Backfill the data from existing packages

[karann-msft]

Where is the spec for this?

cc: @anangaur, @diverdan92

[anangaur]

Spec announced at https://github.com/NuGet/Announcements/issues/30

[knocte]

...get to a Switch PackageReference -> Project from Source feature, which is a common ask.

Hey @emgarten, just wondering if there's a github issue tracking this? I'm also interested in this feature. Thanks and sorry for hijacking the thread for this.

[skofman1]

Hey @emgarten, just wondering if there's a github issue tracking this? I'm also interested in this feature. Thanks and sorry for hijacking the thread for this.

@rrelyea , perhaps you know?

[natemcmaster]

On the .NET Core side, we have this issue: https://github.com/dotnet/sdk/issues/1151

[knocte]

On the .NET Core side, we have this issue:

Mmm, not sure I misunderstood the feature then; I was thinking it's more to do with NuGet rather than .NET Core. (More info: https://github.com/ctaggart/SourceLink/issues/320)

[natemcmaster]

I think what @emgarten was suggesting was that you could use repository url and commit data from the package nuspec to replace a PackageReference with a source code (i.e a ProjectReference). Implementing this would likely require an SDK change (https://github.com/dotnet/sdk/issues/1151 is about making it possible to swap a PackageRef for a ProjectRef) and some NuGet features to fetch source code based on package metadata. This feature outside the scope of what this particular issue is about. Not sure if an issue exists on the NuGet side to actually build this feature.