Home: NuGet.Server - Long Delay in Package Showing in Feed When .nupkg Copied to Packages Directory

[CvilleDev]

We are seeing a delay of up to 45 minutes between when a .nupkg is copied to the Packages directory for our internally hosted remote NuGet feed and when that package shows as available in the feed. It appears that the packages are not being immediately processed and filed to the appropriate package-specific subfolder within the Packages directory.

The only way that we have found to force the process is to clear the server cache and retry a fetch from our custom source. This is not a solution however as our developers do not have sufficient permissions to clear the cache and admins can not be expected to continuously monitor the feed.

Is there some means available to minimize the delay and speed up the processing?

We are using NuGet.Server version 2.11.2.0.

[maartenba]

Can you enable tracing (system.diagnostics) in web.config and see if there are any exceptions after copy? Alternatively you can use NuGet.exe push to immediately invalidate the cache.

[CvilleDev]

I enabled tracing and captured the attached logs when I tried copying a new nupkg file (Newtonsoft.Json.9.0.1.nupkg) into the Packages drop. As you can see, there does not appear to be any event captured to trigger the processing of the package. It just sits in the drop location until I either clear the server cache or restart the service.

LogFile.txt

[maartenba]

Does it pick up when you navigate to /nuget/Packages? When the app pool is down, there is also no file system watcher enabled (it will only start when navigating to /nuget/Packages once)

[CvilleDev]

Yes, when I navigate to /nuget/Packages the nupkg file is processed immediately.

[maartenba]

Then this behaviour is by design. You can try to keep the app pool warm, and then the file watcher will also remain active.

Good thing is that this URL is also hit by VS, so when someone browses the feed, the package will be processed.

[CvilleDev]

I configured the application pool to recycle every 5 minutes, thus the longest we should have to wait is 5 minutes for a new package to get picked up.

[maartenba]

It only really registers the file watcher after that URL is hit, so essentially a 5min recycle kills the file watcher every 5min

[CvilleDev]

Prior to setting the app pool recycle to 5 minutes I was seeing the following behavior.

I would reset IIS, navigate to /nuget/Packages, and then drop a new .nupkg file in the packages drop. The .nupkg would sit there until I either navigated to /nuget/Packages, reset IIS, or recycled the app pool. This is why I chose to set the app pool recycle at 5 minutes.

It would appear that the file watcher is only active when the service starts up. What is the recommended approach for keeping the file watcher active?

[Birgos]

I can confirm this problem and was able to test it locally. Only if I delete the cache via an api call, the feed gets updated and the copied packages are shown in the feed. Just calling

http://localhost:/nuget/Packages

after copying didn't helped! Indeed the package(s) were processed (extract and sort it into the folder structure) but the feed still doesn't shows the new one(s).

Not updated feed

• Copy new package(s) to the NuGet server file share
• Calling feed (.../nuget/Packages) -> integration of new package(s)
• Query feed for NuGet packages -> list still shows old packages without the new ones

Feed shows new package(s)

• step 1 + 2 from above
• Thread.Sleep with x seconds (until all new packages were processed)
• Calling api-url (.../nugetserver/api/clear-cache) for cleaning cache
• Calling NuGet-Feed -> new package(s) are listed

[maartenba]

Anything in the logs? (add a system.diagnostics in web.config)
This may be due to permissions on your system where the file watcher errors out (which will be visible in traces)

[Birgos]

Now i tried it locally on my PC (IIS Express) and with my nuget server. The logfile of both systems shows the same behavior.
First it recognizes the changes:
Verbose: File system changed. File: mongocsharpdriver.2.3.0.nupkg - Change: Changed
but later on it logs an error:
Error: Error adding package file mongocsharpdriver.2.3.0.nupkg from drop folder: Could not find file 'd:\Packages\mongocsharpdriver.2.3.0.nupkg'.

But the new file was deleted from the dropfolder and put into the correct folder structure (extracted nuspec, nupkg and hash sums)??!

Now querying the feed: http://localhost:8080/nuget/Packages?$filter=substringof(%27mongocsharpdriver%27,Id)%20and%20IsLatestVersion

still list the old package. Clearing the cache via api or deleting the .bin file - voilá the feed now shows the new version 2.3.0...

Any ideas?

Below the logfile of the server
NuGet.txt

[maartenba]

Can you try updating to this (prerelease!) version of NuGet.Server?
Install-Package NuGet.Server -Version 2.11.3-dev-0012 -Source https://dotnet.myget.org/F/nuget-build/api/v2

[Birgos]

Sorry, failed :-(
Where can I find the dependency: NuGet.Core (>= 2.14.0-rtm-830) ?

[maartenba]

Should be on this same feed

[Birgos]

Unable to find a version of 'NuGet.Core' that is compatible with 'NuGet.Server 2.14.0-dev-0003 constraint: NuGet.Core (>= 2.14.0-rtm-830)', 'NuGet.Server.Core 2.14.0-dev-0003 constraint: NuGet.Core (>= 2.14.0-rtm-830)', 'NuGet.Server.V2 2.14.0-dev-0003 constraint: NuGet.Core (>= 2.14.0-rtm-830)'.

I already tried to install with -IgnoreDependencies, repacking NuGet.Server locally and changing the dependency to -dev-0003 but without success. Then there are also similar incompatibility warnings with other packages like Microsoft.AspNet.WebApi...

[maartenba]

Can you try once more? Just pushed the package to the feed again. Note this is very prerelease :-)

[Birgos]

Finally, now I was able to install the packages! Took me some time to get the solution up and running, due to some error messages regarding dependencies of old Newtonsoft.Json packages (6.0.0.0) from _System.Net.Http.Formatting_ and the the new ones (8.0.3) used in the NuGet.Server V2 package...

I've tested the following scenario ( _after i've built me a simple integration test ;-)_ ):

• 2 local nuget server: 1 gallery site - 1 mirror site
• download a new nupg-file from gallery site and saved it into the dropfolder of the mirror site
• the file was instantly processed and sorted into the folder structure
• querying the (mirror) feed shows the version almost immediatly

So I would say it looks good and I am looking forward to the stable version!
Thank you very much for your support!

[maartenba]

Awesome! We have some work with the dependencies and a few other things but it's great to hear this solves the issue.

[ABrehm264]

@maartenba, howdy, when were you planning to get this pushed into Production? I'm hoping it'll fix my problem with the legacy drop folder not really working. It works some of the time, but normally I have to spend a whole bunch of time re-dropping packages, recycling the NuGet server application pool, and hitting either the http://myNuget/nuget or http://myNuget/nuget/packages urls until some combination of those starts cause the packages to be processed.

[nkolev92]

Issue moved to NuGet/NuGetGallery #5535 via ZenHub