Nugetgallery: Using Cloud storage link as packages path in NuGet server feed
Hi Guys,
I have hosted my own NuGet feed using NuGet.Server. we have more than 30GB NuGet packages in our server. already we have these files in our s3 amazon cloud storage also. the server memory is increasing due to these packages for every release.
My doubt is, it is possible to use the CDN(s3 amazon cloud) link as packages path instead of local path?
if yes, please provide the sample.
Thanks,
Nagaraj M.
NagarajMasub
>All comments
Hey @NagarajMasub, thanks for your interest in NuGet.Server.
Unfortunately, there is not an easy way to do this in NuGet.Server. There are some pretty heavy dependencies on the file system. For example, our ServerPackageStore directly calls into the file system. Additionally, there is a file system watcher in ServerPackageRepository. There are probably many other places too that would complicate the effort of switching to S3. Truth be told, NuGet.Server really isn't meant for these more complex scenarios. If disk space is an issue, you could potentially mitigate it by moving the packages folder to a UNC share hosting on some high capacity device. I have not tested it, but typically all of the read/write file system APIs in .NET work fine on UNC shares.
I think it would be wrong to extend NuGet.Server to the point where arbitrary file stores are possible. NuGet.Server is meant to be very simple. There isn't even a database in NuGet.Server -- it's all in memory. We have a JSON file to speed up metadata reading at start-up, but that's about as complicated as it gets.
The "right" solution here would be to switch over to NuGetGallery. This is the implementation of a NuGet package source that is meant for scale. This is the code that runs nuget.org. Our instance of NuGetGallery pushes packages to Azure Blob Storage (analogous to S3). We have a CDN in front of blob storage much like what you're describing.
In NuGetGallery, there is an IFileStorageService interface which allows arbitrary package stores. The implementation we use for Azure Blob Storage is the CloudBlobFileStorage class. You could mimic this implementation for S3. I have never tried implementing another file storage service, but the abstraction seems right if we have it working for Azure.
I hope this clears things up.
joelverhagen
on 27 Feb 2017
Related issues
xied75
路
5Comments
j3parker
路
4Comments
dannyvv
路
4Comments
yishaigalatzer
路
4Comments
coleman-c
路
5Comments
Most helpful comment
Hey @NagarajMasub, thanks for your interest in NuGet.Server.
Unfortunately, there is not an easy way to do this in NuGet.Server. There are some pretty heavy dependencies on the file system. For example, our
ServerPackageStoredirectly calls into the file system. Additionally, there is a file system watcher inServerPackageRepository. There are probably many other places too that would complicate the effort of switching to S3. Truth be told, NuGet.Server really isn't meant for these more complex scenarios. If disk space is an issue, you could potentially mitigate it by moving the packages folder to a UNC share hosting on some high capacity device. I have not tested it, but typically all of the read/write file system APIs in .NET work fine on UNC shares.I think it would be wrong to extend NuGet.Server to the point where arbitrary file stores are possible. NuGet.Server is meant to be very simple. There isn't even a database in NuGet.Server -- it's all in memory. We have a JSON file to speed up metadata reading at start-up, but that's about as complicated as it gets.
The "right" solution here would be to switch over to NuGetGallery. This is the implementation of a NuGet package source that is meant for scale. This is the code that runs nuget.org. Our instance of NuGetGallery pushes packages to Azure Blob Storage (analogous to S3). We have a CDN in front of blob storage much like what you're describing.
In NuGetGallery, there is an
IFileStorageServiceinterface which allows arbitrary package stores. The implementation we use for Azure Blob Storage is theCloudBlobFileStorageclass. You could mimic this implementation for S3. I have never tried implementing another file storage service, but the abstraction seems right if we have it working for Azure.I hope this clears things up.