Paket: [Feature Req] Possibility to specify a nupkg cache location

[CumpsD]

This issue is to discuss a Twitter idea

Initial idea

https://twitter.com/CumpsD/status/680011065250443264
https://twitter.com/CumpsD/status/680011137853820930

Background story

Paket is great because it gives me repeatable builds with versions I specify. I can safely sleep at night knowing if my project builds today and collects dust for 5 years, it will still build with Paket.

However, I don't always belief things to stay around forever, especially with central package repositories. Things happen. Who can guarantee the packages I use will be there 5 years from now? or 10 years? If I remember correctly, npm had this discussion once, which is why Proget and other proxies exist which cache packages so you keep them even if they are gone from the central place.

Idea

What if I could just tell Paket to store all nupkg files it downloads in a cache, of which I can specify the location. No need for proxy setups or anything, simply run paket, and have it store to a fileshare or something and that's your "backup". Later when restoring, Paket first checks your usual feeds, with the existing logic, and if it doesnt find anything (NuGet down, package removed?) it can fall back to your offline cache.

_Clarification:_
This cache location should be possible to specify per invocation, so you can have different ones per repository for example. Cmdline argument or paket.dependencies option?

Discuss!

[CumpsD]

Additionally, if I tell paket to store the nupkg files in the repository itself, and I use Git Large File Storage to deal with nupkg, it would be nice too :) Then they are all in the repo as a backup, without exploding the repo size.

[amazingant]

FWIW, at work my package source is a network share so that I don't have to worry about packages disappearing. I've had a few packages that randomly pulled old versions that I was depending on. This means I have to manually go out and look for updated versions of packages, so I wouldn't mind if Paket gained the ability to, say, use that network share as the "cache" location per the above suggestion.

[CumpsD]

Pinging @forki for feedback about the idea :) Before working on a PR

[forki]

Yes I think additional caches would be a nice feature for the v3 branch.

• They should be configured in the dependencies file.
• clear-cache command would need options

[forki]

Ok after node recent drama I think we shold make "additional caches" a thing.

I see two variants.

1) We specify a network share and store all package versions
2) we specify a folder inside the repo and store only the latest nupkgs

• so you can commit that folder and still gitignore the packages folder

[forki]

I propose the following syntax:

source https://nuget.org/api/v2
cache ./dependencies versions:current 

nuget Newtonsoft.Json redirects: force
nuget Argu
nuget FSharp.Core redirects: force
nuget Chessie

github fsharp/FAKE src/app/FakeLib/Globbing/Globbing.fs
github fsharp/FSharp.Data src/CommonProviderImplementation/AssemblyReader.fs

and

source https://nuget.org/api/v2
cache //hive/dependencies versions:all

nuget Newtonsoft.Json redirects: force
nuget Argu
nuget FSharp.Core redirects: force
nuget Chessie

github fsharp/FAKE src/app/FakeLib/Globbing/Globbing.fs
github fsharp/FSharp.Data src/CommonProviderImplementation/AssemblyReader.fs

The default for the versions property would be all.

[forki]

@CumpsD did you already start? If not (and if we don't see other volunteers) I would do it.

[CumpsD]

No, nothing on this yet but the proposal. At the moment also limited in time to do anything on it right away.

[amazingant]

@forki Since it isn't directly implied by the two samples you gave, just to be clear, if "current" packages are cached in the repo, it'd be nice to still be able to specify a remote cache location too. So:

source https://nuget.org/api/v2
cache ./dependencies versions:current
cache //hive/dependencies versions:all
...

Hopefully that way, if I've updated project X to use a new version of a dependency, when I go to update project Y it can find the new version of that dependency in //hive/dependencies (if needed).

[forki]

yes that makes absolutely sense.

We we would probe all caches (incl. default nuget cache) before we download from source.
I guess the order would be local caches, nuget default cache and then network shares.

(This way we would also solve #1546 (cc @agross))

[amazingant]

For my setup, that'd be great! Me and my build server have gigabit to the network share, so going there before heading out to nuget.org would keep things speedy. :smile:

[forki]

Yeah I think this could work nicely to make restore fast and reliable.

Open question is "install": if someone (MS) deletes a package from nuget.org then the resolver process would fail even if the package is still present in one of the caches.

[amazingant]

Could it take the time to check the caches and fail with a message? Something like Could not find package left-pad in source; latest version 0.0.3 found in cache //hive/dependencies, and then if you chose to paket install nuget left-pad version 0.0.3, it should be able to look in the cache(s) first.

[forki]

Nice error message:

that said after looking at the code I think we could make the cache into sources. This would probably work

[forki]

PR in #1549

[forki]

now in v3 alpha package

[forki]

docs at http://fsprojects.github.io/Paket/caches.html