Nugetgallery: Download statistics pipeline experiences multiple failures

[skofman1]

The error we are seeing is in ImportCdn job: Bulk load data was expected but not sent. The batch will be terminated.

Most CDN files fail to process due to this issue.

[alexeyo26]

Hello! Any ETA on this? I have the same issue with my packages.
Thank you,
Alexey

[skofman1]

Hello @alexeyo26 , download counts are currently not being updated for all users. We are actively investigating the issue, and will update once we know more.

[rappen]

And this has been the case at least since Jan 12, right?
Any idea if stats will be updated with lost days when you get it fixed, or simply restart at that point?

[gtaylor44]

I came here to ask the same question as @rappen

[skofman1]

@rappen , @gtaylor44 , once the issue is resolved, we will be able to process all missing data retroactively.

[unniravindranathan-okta]

Update:

Thanks to everyone who reported this issue. Unfortunately, it turned out to be a pretty complicated one and hence it took a while to resolve – our apologies for the same. The good news is that we have already deployed a fix, and are now working to backfill the data for download counts. We anticipate that you will be able to see updated download counts for packages by end of day Monday, 1/24.

[cristinamanum]

The data backfilling is still going on. We will post an update once all the backfilling is completed.

[rappen]

...reopened.... not the update we were hoping for :-/
Still problems I guess?

[skofman1]

The backfilling is completed, however we need to do a few more actions to get the E2E scenario working (i.e. so users can see the download counts).
Will close again when the issue is REALLY resolved.

[skofman1]

Download counts were updated. Closing.

[ipjohnson]

@skofman1 I know this was marked as closed but I'm still not seeing stats from a package I released over this past weekend (as well as others). Is it possible this is still not fixed or a new problem has popped up?

[joelverhagen]

@ipjohnson, it was a new problem actually. We've resolved the issue but it could take a day or two to get through our internal queue of download counts to process. What are you package IDs and versions you are observing the problem on? I'll re-open this and only close it after I see the latest count counts updated.

[ipjohnson]

Thanks @joelverhagen Grace and SimpleFixture

[rappen]

One of mine is also stuck again.
https://www.nuget.org/packages/Cinteros.XrmToolBox.PluginTraceViewer/1.2017.1.21
Same issue?

[alexeyo26]

Same thing with my packages:

https://www.nuget.org/packages/Microsoft.Research.CNTK.CpuEval-mkl/2.0.0-beta9
https://www.nuget.org/packages/CNTK.GPU/2.0.0-beta9
https://www.nuget.org/packages/CNTK.CPUOnly/2.0.0-beta9

[gtaylor44]

I still don't think this issue has been resolved. Any idea when we might see the counter playing nicely again?

[joelverhagen]

Thanks for all the feedback, folks. We are currently processing download statistics from middle of the day, January 20th. Our statistics job currently does good keeping up with the flow of data, but playing catch-up can be slow.

[gimlichael]

What is the status; still slow on the catch-up, or stalled?
No data from the 25th of January .. and maybe even earlier.

Also, a little funny, this issue is an "anagram" of former issue #3247 :-)

[joelverhagen]

@ipjohnson, it looks like the latest versions of Grace and SimpleFixture have downloads now.

@rappen, I see downloads for that version flowing now.

@alexeyo26, all versions of those packages have downloads now.

@gtaylor44, what is your package ID?

@gimlichael, your package published on the 25th now has downloads. The pipeline is running fine now so your package published yesterday should soon have downloads.

[danielwertheim]

@joelverhagen Not that I care for my simple packages but is this issue really solved now? Still no counts (despite that I know there has been downloads/installs) https://www.nuget.org/packages/MyNatsClient/ since 26th

[joelverhagen]

@danielwertheim, we care about all packages: big and small 😄. I'll take a look.

[joelverhagen]

@danielwertheim, your package looks fine. There are at least 23 downloads flowing through the pipeline right now. If you still see zero in 24 hours, let me know.

[danielwertheim]

@joelverhagen thanks