Nugetgallery: Provide Status page for nuget.org indexing

[abdullin]

Folks, thank you for the awesome work you are doing for .NET community. You are the central hub for all our package needs in the entire .NET community. Keep it up!

There is a tiny issue, though.

I published a package today (https://www.nuget.org/packages/MessageVault.Client), version 2.1.0. Build server was unable to pull this version from the nuget.org, saying that the package is not found. After the investigation, it looks like that this url wan't updated: https://www.nuget.org/api/v2/Packages(Id='MessageVault.Client',Version='2.1.0'). It returns:

<m:error xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<m:code/>
<m:message xml:lang="en-US">Resource not found for the segment 'Packages'.</m:message>
</m:error>

Older versions of the same package return meaningful XML.

When I asked for advice on the twitter, I got this feedback: "@abdullin recently uploaded? NuGet.org has indexing issues today" and "@JakubKonecki @abdullin apparently the 6 hours ago on the packages page is an indicator."

If indexing issues are indeed the root cause here, please provide some feedback on their status. It would be awesome to have them visible on status.nuget.org.

I'd love to have visibility instead of following another twitter advice regarding nuget.org downtime "Recommend patience...".

[abdullin]

Hi all, so what's the decision on that ticket?

[csharpfritz]

We do need to be more transparent about how long the package indexing process takes when you upload a package. A simple, "package index is x minutes behind" to indicate the upload time of the earliest package awaiting indexing could help clarify the situation.

[maartenba]

http://www.nuget.org/packages/ shows the last index update time ("Index last updated X time ago")

[abdullin]

@maartenba, unfortunately this doesn't come up as an issue on status.nuget page :(

[forki]

as written above the broken indexing prevents user from installing/updating, which is one of the major reasons people use a package manager.
I think it's fair to say that "Search Index last updated about 7 hours ago" is an outage for install/update of newly released packages.

/cc @ashic

[abdullin]

I'd say that a delay of a few minutes is already an outage. Imagine a workflow.

1. Developer in Russia creates a new nuget package, uses it another solution to fix an urgent bug.
2. He commits a change to the repository
3. One minute later, a build server in USA picks up the changes and tries to build the solution.

And the solution at that point breaks with something like that:

The name specified has already been added to the list of available package sources. Please provide a unique name.
All packages listed in packages.config are already installed.
Unable to find version '0.6.24' of package 'AmazonAccess'.

Boom, you have a world-wide team of 20 people that are very frustrated.

[forki]

of course the root course is something else. It's that two API functions give different information. Reported years ago somewhere on codeplex and also here on github.

https://github.com/NuGet/NuGetGallery/issues/2126#issuecomment-43786202 <- closed without solution

https://github.com/NuGet/NuGetGallery/issues/2149 <- nothing happens

[csharpfritz]

I would consider this behavior an outage, but rather a tolerable service
degradation.

All read operations against the NuGet gallery contents that were present
from before the point that the index process slowed down were serviced with
no failures.

On Mon, Jun 1, 2015 at 12:51 PM, Steffen Forkmann [email protected]
wrote:

of course the root course is something else. It's that two API functions
give different information. Reported years ago somewhere on codeplex and
also here on github.

2126 (comment)

https://github.com/NuGet/NuGetGallery/issues/2126#issuecomment-43786202
<- closed without solution

2149 https://github.com/NuGet/NuGetGallery/issues/2149 <- nothing

happens

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107634666
.

[forki]

If you can't "nuget install" then it feels pretty broken for me.
On Jun 1, 2015 19:34, "Jeffrey T. Fritz" [email protected] wrote:

I would consider this behavior an outage, but rather a tolerable service
degradation.

All read operations against the NuGet gallery contents that were present
from before the point that the index process slowed down were serviced with
no failures.

Jeff

On Mon, Jun 1, 2015 at 12:51 PM, Steffen Forkmann <
[email protected]>
wrote:

of course the root course is something else. It's that two API functions
give different information. Reported years ago somewhere on codeplex and
also here on github.

2126 (comment)

<https://github.com/NuGet/NuGetGallery/issues/2126#issuecomment-43786202

<- closed without solution

2149 https://github.com/NuGet/NuGetGallery/issues/2149 <- nothing

happens

—
Reply to this email directly or view it on GitHub
<
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107634666>
.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107649076
.

[csharpfritz]

Thank you for making my point for me: for all but a very limited number of
packages, you can NuGet install them with no problems.

That's what service degradation is.

On Mon, Jun 1, 2015 at 1:38 PM, Steffen Forkmann [email protected]
wrote:

If you can't "nuget install" then it feels pretty broken for me.
On Jun 1, 2015 19:34, "Jeffrey T. Fritz" [email protected] wrote:

I would consider this behavior an outage, but rather a tolerable service
degradation.

All read operations against the NuGet gallery contents that were present
from before the point that the index process slowed down were serviced
with
no failures.

Jeff

On Mon, Jun 1, 2015 at 12:51 PM, Steffen Forkmann <
[email protected]>
wrote:

of course the root course is something else. It's that two API
functions
give different information. Reported years ago somewhere on codeplex
and
also here on github.

2126 (comment)

<
https://github.com/NuGet/NuGetGallery/issues/2126#issuecomment-43786202

<- closed without solution

2149 https://github.com/NuGet/NuGetGallery/issues/2149 <- nothing

happens

—
Reply to this email directly or view it on GitHub
<
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107634666

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107649076>

.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107650495
.

[abdullin]

@csharpfritz, doesn't this break publishing feature of nuget completely?

The feature is described here: http://docs.nuget.org/Create/Creating-and-Publishing-a-Package

[csharpfritz]

You are still able to upload items to the gallery. They will be available
once they have been analyzed,added to the index and deployed to CDN.

Let's discuss this another way: would it be preferable to receive an error
message that NuGet is not accepting new packages, or would you like a
publish package operation to complete, and they are available some time
later?

On Mon, Jun 1, 2015 at 2:04 PM, Rinat Abdullin [email protected]
wrote:

@csharpfritz https://github.com/csharpfritz, doesn't this break
publishing feature of nuget completely?

The feature is described here:
http://docs.nuget.org/Create/Creating-and-Publishing-a-Package

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107656752
.

[abdullin]

Clear error message (failing early) would've saved us hours of debugging and frustration.

[csharpfritz]

What if an email message was sent to the owners of packages when new
versions were made publicly available when the index has completed running?

On Mon, Jun 1, 2015 at 2:38 PM, Rinat Abdullin [email protected]
wrote:

Clear error message (failing early) would've saved us hours of debugging
and frustration.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107666392
.

[abdullin]

Email would be a viable work-around for us (I'm not sure about the other nuget users, though).

Would it be possible to provide an uptime status from the indexing job as well? E.g. show the index queue depth on the status page. Here's a trivial example we setup on a current project:

If queue depth is non-zero and doesn't increase, then everybody can see that there is a problem.

A more serious solution would be to provide an SLA (e.g. publish within 5 minutes), verify it with heartbeat jobs in the QA and treat SLA violation as a failure.

[csharpfritz]

I can see the email notification as an opt-in at package upload time or
perhaps a profile preference.

As part of the work for this issue, our team is working on a customized
status page that does provide additional information. We do already
provide some information, (I'm not sure about queue depth) on an internal
status portal. Our intention is to bring some of that information to the
new status page.

Jeff

On Mon, Jun 1, 2015 at 2:49 PM, Rinat Abdullin [email protected]
wrote:

Email would be a viable good work-around for us (I'm not sure about the
other nuget users, though).

Would it be possible to provide an uptime status from the indexing job as
well? E.g. show the index queue depth on the status page. Here's a trivial
example we setup on a current project:

[image: screen shot 2015-06-01 at 23 45 39]
https://cloud.githubusercontent.com/assets/504782/7920453/8c2c5eb4-08b8-11e5-9694-ce8802f46110.png

If queue depth is non-zero and doesn't increase, then everybody can see
that there is a problem.

A more serious solution would be to provide an SLA (e.g. publish within 5
minutes), verify it with heartbeat jobs in the QA and treat SLA violation
as a failure.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107669501
.

[forki]

I am all for: allow the upload. Show only the latest indexed package across
all apis. The point is: if the api which displays all the version nos.
would be slower than the other api then nobody would ever get an error.
People would just not get the currently uploaded package. Which is fine.
On Jun 1, 2015 20:52, "Jeffrey T. Fritz" [email protected] wrote:

I can see the email notification as an opt-in at package upload time or
perhaps a profile preference.

As part of the work for this issue, our team is working on a customized
status page that does provide additional information. We do already
provide some information, (I'm not sure about queue depth) on an internal
status portal. Our intention is to bring some of that information to the
new status page.

Jeff

On Mon, Jun 1, 2015 at 2:49 PM, Rinat Abdullin [email protected]
wrote:

Email would be a viable good work-around for us (I'm not sure about the
other nuget users, though).

Would it be possible to provide an uptime status from the indexing job as
well? E.g. show the index queue depth on the status page. Here's a
trivial
example we setup on a current project:

[image: screen shot 2015-06-01 at 23 45 39]
<
https://cloud.githubusercontent.com/assets/504782/7920453/8c2c5eb4-08b8-11e5-9694-ce8802f46110.png

If queue depth is non-zero and doesn't increase, then everybody can see
that there is a problem.

A more serious solution would be to provide an SLA (e.g. publish within 5
minutes), verify it with heartbeat jobs in the QA and treat SLA violation
as a failure.

—
Reply to this email directly or view it on GitHub
<
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107669501>
.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-107670066
.

[ashic]

"Thank you for making my point for me: for all but a very limited number of
packages, you can NuGet install them with no problems.

That's what service degradation is."

@csharpfritz I have to disagree. Scenario:

• I have version 1.0.1 of PackageX.
• I upload version 1.0.2 of PackageX.
• User Y runs nuget.exe install PackageX before the next indexing operation.

In this case, degradation would be if the user get version 1.0.1, instead of 1.0.2. However, this is not what happens. User Y's command results in a "could not find 1.0.2 of PackageX" error. Until the indexing finishes, all runs of "Nuget.exe install PackageX" completely fails. This means any update of packages result in the packages being unavailable to nuget.exe until the next index.

On another note, it is interesting though that:

http://www.nuget.org/api/v2/packages/PackageX

actually downloads the latest uploaded version's (in this case 1.0.2) nupkg. So it seems that it's purely the partial update of the metadata (i.e. returning latest info, while the rest of the index isn't done) is at fault.

[I realise this is more about the actual problem, rather than the downtime reporting.]

[yishaigalatzer]

This issue is on top of our priority list.

The issue happens because NuGet has really two servers (a sql based server) and a static files based server, and they haven't been merged properly. Merging them correctly is not a small task, and we are working on two avenues here (like many of your comment correctly on above)

1. Make the indexing jobs more reliable, improve the auto healing, heart beat etc.
2. Make the status page track indexing with a delay of 15 minute going to start counting as downtime.
3. Make the search service (the thing that tells you a package exist) be synchronous with the download package available on CDN.

@abdullin I really like the idea of the index queue graph! We can open a separate bug for it, and if you already have good code you can contribute we will gladly take a PR for it. (Would you mind opening another issue with more details on your suggestion?)

We are taking this seriously and working hard to bring NuGet.org back to a more stable place. We appreciated your patience and feedback as it helps making things better.

[abdullin]

The reason why people can get so frustrated at nuget.org failures is right at the front page: "NuGet is the package manager for the Microsoft development platform including .NET. The NuGet client tools provide the ability to produce and consume packages. The NuGet Gallery is the central package repository used by all package authors and consumers."

People pay money for Microsoft development platform (we do) and they implicitly expect that a stellar NuGet experience is bundled in by default.

@yishaigalatzer, I simply use StatsD.Client (a nuget package :) ) and a out-of-box installation of Graphite+Carbon+Grafana (all - open source projects) to capture this kind of data. Wrote more about that in a separate ticket #2525

[yishaigalatzer]

@abdullin I share your frustration. The only way I know to deal with it is to work at fixing the problems and moving things forward. I'm hoping you are able to tell the difference in the coming weeks and more goodness in the coming months.

Concrete feedback like you provide here is a great way to make the product better.

Priority wise: We decided to first focus on the root cause of the services going down regularly. It is mostly related to memory allocations and poor auto recovery. We then plan to tackle the eventual consistency issue, the reason we are going for it second is that quick fixes we prototyped had raised even more concerns like the above one so the way forward is to do a proper fix like suggesting above given that this issue is around for more than a year, I thought this is the right order of thing (note that I took over from Jeff Handley a bit less than two months ago). We then plan to have statistics restored, because although that is visible and easy to rag on, is really secondary to the concerns you are raising.

We now track deployments and bugs here using milestones and dates S1, S2 are the current milestones and we plan major deployments every two weeks, where we do critical deployments daily.

It is going to be somewhat of a journey to resolve all these issues, but I assure you we put other projects aside and we are fully focused on nuget.org stability and reliability.

I hope this at least explains what is going on.

[abdullin]

@yishaigalatzer Thank you for dealing with the problem and being open with it. I think, this could make a wonderful blog post for the readers, to reassure them about nuget (it reassured me).

[maartenba]

https://nuget-dev-0-status.azurewebsites.net is a first PoC

[forki]

Neat. Kudos
On Jun 26, 2015 11:29 AM, "Maarten Balliauw" [email protected]
wrote:

https://nuget-dev-0-status.azurewebsites.net is a first PoC

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#issuecomment-115607934
.

[maartenba]

Now also has the index lag - PoC at http://nuget-dev-0-status.azurewebsites.net/

[abdullin]

Thanks a lot! Tweeted about it :)

[maartenba]

Keeping it open, work in progress still :-)

[neurospeech]

Well for time being, I wrote a small proxy server that rewrites Package query to OData query and it works fine. https://github.com/neurospeech/NuGetProxy , From this proxy, it seems this is not that difficult as one query results in correct answer and other query does not.

[maartenba]

Ideally you'd use the V3 feed going forward.

[maartenba]

http://blog.nuget.org/20150928/Meet-the-new-NuGet-status-page.html

[forki]

Very good work. Kudos
On Sep 28, 2015 7:50 PM, "Maarten Balliauw" [email protected]
wrote:

Closed #2432 https://github.com/NuGet/NuGetGallery/issues/2432.

—
Reply to this email directly or view it on GitHub
https://github.com/NuGet/NuGetGallery/issues/2432#event-421026053.

[mattfrear]

What if an email message was sent to the owners of packages when new versions were made publicly available when the index has completed running?

Hello from the future. I was reading this thread to try and figure out when my package would be indexed, and then this reminded me that NuGet sends an email when it's published. So I checked my email, boom, it's already published. Now I can use my package 👍

[jahmai-ca]

What happened to the indexing graph on status.nuget.org?

[anangaur]

@jahmai Created an issue: https://github.com/NuGet/NuGetGallery/issues/4881

[dazinator]

Hello from the future. I was reading this thread to try and figure out when my package would be indexed, and then this reminded me that NuGet sends an email when it's published. So I checked my email, boom, it's already published. Now I can use my package 👍

~I had an email to say my package was published.~

~So I told another developer and they tried to install it. However they couldn't. I checked the status of the package and it has been stuck on "Validating" for over an hour.~

~Please can you consider sending a "Published" email confirmation only when the package is actually readily available. The current email confirmation seems to be just a notification that a package was uploaded but says nothing about its availability to others. "Publishing" is all about making something available to others so an email saying the package was published, when its not yet available is a bit misleading imho.~

Apologies - ignore me - looks like I had the "published" emails for other packages uploaded at the same time, but not the package that is stuck as "validating". So if nuget.org only sends the published email after the package is available / indexed then that's fine!

[anangaur]

/cc: @karann-msft do we still send the mail after validation or after both validation and indexing? Should be the later.