Gun: Expiry

[amark]

Sadly the heroku server crashed from too much memory usage. So it looks like adding an LRU expiry or some type of flushing mechanism is going to be pretty important sooner rather than later.

[amark]

Needs to have hooks for #45 :) .

[amark]

Still need this, likely important for core! So I want to keep this issue alive.

Note to people: With v0.6.1 of gun, there is still no internal expiry/eviction/LRU yet.

[lmangani]

+1

[ctrlplusb]

@amark - any workarounds? Browsers only support 5mb local storage too right, what happens when this limit is hit?

[amark]

@ctrlplusb last time I checked, when 5MB limit is hit gun will give you errors on your put but will still sync data out to the server. Just localStorage won't have it.

[leonardpauli]

It would be neat to be able to specify the flushing mechanism in options, though plugin, or with object meta data;

• Like automatically removing least used (but acked) data
• warn if there is too much unacked data (risk getting not saved) (by calling a handler I can override! (with info such as memory usage and easy way to force flush, eg. as object))
• being able to provide removal priority (hints) myself (eg. "that data is less important to keep to the end than this data, even though it's accessed more")
• set different limit (eg. "I've got a bunch of other stuff to save, so you'll only get 3MB!")

Also:

• global error handler (more than not I do operations "optimistically", don't wan't to micromanage errors on each put, but wan't to be able to retry / show warning / retract change in a global handler) (related but different issue though)

Notes:

• According to @robertheessels, "pagination" of data already works? Though if it's like "save everything in localStorage until full, then go straight to peers while keeping localStorage with old data", then it's not optimal (eg. will prevent offline ability, while the solution above won't)
• @BrockAtkinson mentioned you can setup stream to local indexeddb (ie. if browser, or other if node), thereby "unlimited storage", though still within "very" limited quota... My usecase will handle theoretically "unlimited" amounts of data, where only a tiny amount would need to be available offline (eg. you travel around the world, but you only really need your backpack (~ analogy...))

[amark]

Oh, just FYI since I forgot to mention this lib/later is a prototype for TTL stuff, you can use it like:

gun.get('foo').later(function(doc){
  console.log("goodbye:", doc);
  this.off(); // unsubscribe from the data
}, 5); // 5 seconds
// NOTE: if you use `lib/memdisk`, the unsubscribe
// will also skip writing it to disk.

These are experimental of course. And I'm working on an adapter/middleware that checks for null updates across peers and deletes the data from in-memory. Deleting from disk is harder, because every storage adapter has to specially implement that. (As demand for this increases... I'll probably add a standardized protocol for it, add support for it in RAD, and then make it optional for adapters plugging into RAD, but support it in the default ones... but this will probably be after v1.0 and not high on priority... although if somebody wants to help, that would be great!)

@LeonardPauli these are actually great use case configurations that could probably be done pretty easily with a fork to the localStorage adapter https://github.com/amark/gun/blob/master/src/adapters/localStorage.js . I don't have time for it right now, but I'll keep it in mind for future / or if anybody else who has already written a storage adapter, they probably have enough experience.

Global error handling: I want this too! I already have global error logging, so you could actually "hack" this right now by doing something weird/stupid like:

old_log = Gun.log;
Gun.log = function(a){ if(a && a.err){ globalErrorHandler(a) } old_log.call(parameter.slice().whateverTheJSis) }

Search the github issues for pagination, right now this can be done in the data structure itself, I do have a wire spec/protocol for doing byte-constrained reads, but haven't implemented it yet.

indexedDB still has a cap, but user approval can up it. Better to do an Electron app or similar to get full storage access.