Go-ipfs: Moving away from current pin system

[Voker57]

As suggested by @whyrusleeping in #4757

Current pin system has several problems, as I see them:

• Pins are stored in elaborate way in blockstore, enabling whole world to sniff your pins and introducing great deal of complexity to code like internal pins and global pin locking
• Pins are unnamed, so you can't structure/review them easily

Suggestions & reviews are welcome

My proposed PR (WIP): #4757
Another option: #4675

[hsanjuan]

The main considerations I can give as a pin-system user, out of the top of my head:

• Global locking. Add this to bitswap bugs and you end up perpetually locked.
• Can't enable/disable GC via the API. Fetching without pinning, or putting blocks without/before pinning, to avoid the pinning hurdles, leaves you exposed to GC wiping your content.
• No names (cluster added names for pins but it wouldn't hurt)
• pin/ls --recursive needs to be fast with huge lists
• Unpin should cancel any ongoing pins for the same cid, if things can happen in parallel.
• Async pinning would be nice to have:
  
  
  
  • I tell ipfs to pin something
  
  
  • I check back later to see if it's still pinning, how far it is (size fetched).
  
  

enabling whole world to sniff your pins

Is there anything different as to how the whole world can sniff other non-pinned stuff?

[b5]

I'd resurface @Kubuxu's comment raised on #4757 pointing out this would be a great candidate for the RFC process. I'd _love_ to see some of the changes mentioned here, but knowing where pinning is headed in the long run is far more important IMHO

[singpolyma]

Async pinning would be a huge win

[Kubuxu]

Some thoughts:

I think that having pins under MFS (files api) would be very nice feature. This would get us sharding for free, helps with pin exploration and solves pin naming.

We can significantly reduce complexity of GC by removing direct pins and allowing pinned items to be pinned either strongly or weakly (all content must be here vs keep whatever we have). This would allow for not extremely complex implementation of concurrent and/or incremental GC at least partially solving locking problem.

Also regarding GC, we could have /timed-tmp in MFS that would keep files for X time and after that remove them from this directory. Files would be added there when (for example): accessed from gateway, added and so on. (Just some wild idea).

In case of using MFS for pinning, under the hood we would: 1. copy the file to pin MFS; 2. start fetch of MFS directory (this operation is not written but should be simple enough).

Next question is: we would like to allow for selecting providers strategies per pinned item. This might be hard in case of pb-unixfs but should be very easy in case of cbor-unixfs (if it supports xattrs).

Some problems: MFS would have to support non-file content (CBOR, git trees, ethereum blocks) but only in limited regard (cp, mv, rm, fetch) which could lead to weird UX.

[lidel]

I feel it is worth mentioning that our GUI apps are going to move away from ipfs.pin and operate solely on ipfs.files API, making "pin" a low-level abstraction hidden from user. Will that approach be future-proof? Feedback and UI-related thoughts are appreciated at:

• UX Design: Pinning Content https://github.com/ipfs-shipyard/pm-ipfs-gui/issues/10

[Voker57]

I think that having pins under MFS (files api) would be very nice feature. This would get us sharding for free, helps with pin exploration and solves pin naming.

It is nice, but 1. Sharding is going to be used by very low percentage of users

1. Pin exploration as in discovering other people's pins? That's often a bug, not a feature.
   3. MFS pin implementation is very complex compared to datastore-based (my PR actually reduces SLOC while adding functionality) and requires 'internal' pins.

[Kubuxu]

Pin exploration as in discovering other people's pins? That's often a bug, not a feature.

No, I mean your local pins.

requires 'internal' pins.

User MFS wouldn't require that as it would be in the same tree as ipfs files command which is already wired in into GC.

[Stebalien]

Sharding is going to be used by very low percentage of users

Sharding would allow us to store and traverse large pinsets efficiently. This fixes the concern with the original issue (#4673) where the pinset was too large to operate on. Your PR would also fix this (but wouldn't give us a nice hierarchical filesystem).

[Voker57]

https://github.com/ipfs/rfcs/pull/4 made an RFC.