Gun: Would it be possible to create a module that implements an append-only log that we could rebuild from?

[jeffbski]

One of the most important things I look for in a database is whether it keeps an append-only log that can be replayed. Along with that it should be possible to playback up to a certain point in time or up to a particular operation or to exclude an operation.

The reason this is so important is that if there happen to be a piece of bad code that deletes a bunch of properties or wreaks havoc on the system which gets propagated to all of the other peers, then all of your copies of the data are messed up. So it doesn't matter if all of your systems are up, they are all corrupted.

By having the append-only log which we can rebuild from we can go back and replay everything up to the bad operations or excluding the bad operations and not lose anything. It is an invaluable safety net. Hopefully you never have to use it but you can sleep much better knowing it is there if you need it. And one can feel better about trying out this new database if they can see that there data is being logged somewhere.

One of my colleagues once dropped a production database by accident in Oracle. He was testing things out and thought he was in development but he had switched to production. Fortunately it was not a big deal since we just rebuilt the db up to the point where he made the drop and all was good.

I imagine this could be done as an external module.

Let me know what you think.

[amark]

Yes. GUN's architecture is built around this idea (in fact, all the realtime data push is the "stream" or journal or log). Right now however we haven't actually added a module/extension to flush these to disk, but it is pretty easy.

In fact, on the server if you just use gun.wsp(server, CALLBACK) that callback will get called for every command sent over the wire. If you filter for if(req.method === 'put') you could then flush every writes to disk yourself (perhaps into your event sourced db!). Sorry we haven't built a module for it ourselves yet: EDIT: Note, we plan to in the future, but not high priority.

Couple things to note: GUN is idempotent, which means it retries writes if they don't succeed, and clients often sync their local states with the server when they connect to make sure everything is correct (in the future we hope to exchange hashes instead). AKA gun can be very chatty just as a warning. Second is that if you do want to "replay" you'd have to make sure you start with a fresh gun instance (not one that has access to the current data) and replay it in the correct order (elsewise gun's merge algorithm is always going to prioritize more recent updates from older updates).

[jeffbski]

That's fantastic. When I get a chance I'll take a look at that and see if i can help build it. I'm glad it sounds like there might be an easy way to hook in for this, that's the most important thing.

I realize there are a million things to do with a project like this, so no worries. If I can carve out some time I'll see if I could create something that will work. It may be a little while before I can get to it.

Thanks!

PS. depending on how you are tracking future plans, we can leave this issue open for others to see or close it. If we keep it open it might be good to apply some sort of label like "planned".

[amark]

@jeffbski how is the event sourcing stuff going? I also just shot you an email, from your comments in the other issue. Hope you are doing well!

[jeffbski]

I haven't had time to look at it further unfortunately.

On Tue, Nov 29, 2016 at 8:20 PM, Mark Nadal [e-mail]
wrote:

@jeffbski https://github.com/jeffbski how is the event sourcing stuff
going? I also just shot you an email, from your comments in the other
issue. Hope you are doing well!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/amark/gun/issues/250#issuecomment-263763538, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAWOQr6Qo1r54lZRtwZjvPD9AHPX8M-ks5rDN1sgaJpZM4KaTSe
.

--
Jeff Barczewski
Founder of CodeWinds
http://codewinds.com/
Online developer training

[amark]

Journal is a great idea, and should be built at some point, gonna ask that we have this moved over to: https://github.com/gundb/feature-requests though instead.

With the huge improvements shipped in 0.5+ (now on v0.6) we're closing in on a v1.0 stable production-ready release and want to clean up the "issues" section to be dedicated to bugs so that way it is easier for people to find known flaws. Thanks! I hope you are doing well.