Gun: Some comments on the documentation -- things that were unclear

[adamierymenko]

First of all: bravo. I've been resisting the urge to write this for a long time since I'm too busy with other things, but now that it exists I might very well use it.

However, the docs suffer from a common issue of being written for someone who already understands GunDB. :smile:

Here are some things that are very unclear to me:

1. How keys, paths, etc. interact and what the data model "looks like" is something I'm still wrapping my head around. It seems vaguely Redis-like, but it took me a while to grasp that. Start by explaining what exactly GunDB stores and how indexes work. I have a fairly simple SQL structure I am thinking of migrating to this and I'm still wrapping my head around how that migration would go.
2. You talk about using S3 as a storage backend. Not a bad idea. Does that mean my latency is limited to S3 latency? Does S3 become the single point of failure? I get the sense the answer is _no_ but this is not clear. Would it be correct to say that S3 is the last resort authoritative cache and that it would only be used if all my nodes went down? Or is this wrong? Again unclear.
3. If I wanted to use this on the client, I obviously would _not_ want my client to see my _entire_ database. I assume then that I would want to spin up ephemeral instances of GunDB for each login session and put data there to sync it to/from my client. Is that true? If not what is the canonical method for this?
4. Finally, what is a recommended schema (elaborating on the first question). This is also a problem with Redis docs-- there is no discussion of how the database _should_ be used, of design patterns or best practices, etc., and it's left totally up to me. That's fine except I assume the authors might have better ideas than I do about how to use the database. With SQL I know all about normalization, etc., and with Redis I eventually did figure out paths and sets and such, but with this what's the Right Way or at leas your take on the Right Way?

Not complaining too much. I am getting a sense of it, and being a developer too I understand that docs often come last.

[adamierymenko]

Ahh... two more things I can't glean from the docs:

1. What happens when I add a GunDB node to the cluster and then start making queries? Does it get a complete copy of the other cluster members' data? Does it consult S3? Or does it only get things I query? If so how do I tell it to replicate my entire cache?
2. Should all nodes be configured to hit S3 or just one "persister?" Two? Totally unclear.

[amark]

Just saw this, will read through it when I get a moment - but your initial point "written by somebody who already understands gun" rings true. I'm excited for us (the team) to look through this!

[adamierymenko]

Yeah I'm willing to help from the perspective of a naive new user who is experienced with a lot of other things: SQL, Redis, memcached, and various p2p distributed systems.

I'm the author of: https://github.com/zerotier/ZeroTierOne -- we're considering GunDB for the next version of our web admin UI and backend. Right now it's PostgreSQL but I've been wanting something... well... just like this.

[amark]

Here are some quick responses - although I believe your comments are suggestions on how we should rewrite things ( @metasean & @PsychoLlama ). But I want to answer your questions as well:

1. Keys are non-semantic ways to lookup and index your data. Aka key/value, however you can have multiple keys that point to the same thing. Paths allow you to traverse the fields/property/attributes of your data, this is very much "document-oriented".
2. Latency is not limited by S3. S3 is not a single point of failure. And S3 is not a last resort only. Let's run through this. The browser tries to "get" a key, it looks it up in memory - if it is there, we are done. Nothing more. If it is not in memory, then it looks it up on disk (localStorage) and over the network (using websockets or fallback). A GUN server receives this request and rebroadcasts it to peers, then it looks it up in memory and if it has it - we're done, we reply back to the peer. If the server does not have it in memory, it looks it up in/and/or on disk in/and/or S3. Once the server gets the data from the storage layers, it becomes cached in memory (and synced) so that way you do not have to worry about S3 latency. Once this response goes back down to the browser, it becomes cached there as well so it doesn't have to worry about server latency. GUN will still work even if S3 fails though, because storage of data is also happening in localStorage and on other peers - however S3 is the least likely to fail. S3 gets synced with by updates from gun automatically.
3. The client already does not load the entire database, only the data it requests. However this is not an authorization mechanism. Security is very specific to your needs, so if you need help with this we should discuss it further. I leave to do a tech talk in Europe on Wednesday, are you available sometime next week?
4. "Right Way" is very difficult to answer, given that the graph structure intentionally allows you to bend the database to fit your needs (not the other way around). However, from a performance perspective (no matter what database you use), indexing everything to keys is the best approach since you get O(1) lookups. Unlike other databases though, you can do this with GUN while still also having semantic structure for your data.

B1. It can, but it doesn't have to. Typically it only streams the data that it needs, however if the client side logic needs to do more complicated queries, then it'll naturally stream in and caches more data. On future lookups, the queries can be done on whats in memory - which is much faster than via the network.

B2. Only server peers hit S3 and they only do it when they don't already have the data cached.

I hope this fast response provides some initial clarifications. I'm happy to follow up. Like I said though, I'll be in Europe over the weekend, and will be back state side next week but will be pretty busy then as well. But we hope we can get you started in the mean time.

[amark]

@adamierymenko just checking in, how are things going?

[samspot]

For me, I came to the docs looking for how to setup my own peer, and after a lot of searching I can't figure it out. I was expecting this to be mentioned somewhere in either the getting started or the github readme.

[PsychoLlama]

Hey @samspot!
You're right, that should probably be added to the Getting Started page. I'll add a link there...

Right now, browsers can't connect to other browsers directly (even with WebRTC you need to start with a server), meaning you've gotta have at least one gun server that acts as a relay for your browsers.

We've got a wiki page showing how to set one up, hopefully it helps. Thanks for bringing this to our attention!

[steveharman]

What @samspot said for me too.

A real-world example of migrating to Gun would be incredibly useful, I think for a lot of people. Take say an existing simple 'hello world' mobile app which connects to a remote back-end for put & get. What are the steps to get the same app working with Gun? Setting up the remote peer; sending data from the app, retrieving data from the remote peer.

That would certainly help fill gaps in understanding for me

[amark]

@samspot and @steveharman and @ldiebold thank you for complaining about this! Jesse is working on a gun starter application boilerplate as we speak for you guys and we'll get it posted / added to docs/examples/readme/wiki whatever.

Once it is up, please let us know if it is what you were wanting or if you were wanting something else. :)

[PsychoLlama]

Got it posted! Here's the boilerplate. I tried to keep it minimal, showing how the browsers and servers work together. Hopefully it helps?

[amark]

Hey all, great recommendations. I believe we've gotten many of them pulled in now, and it has been a while since there has been activity on this issue. So I'm gonna close it BUT DO NOT HESITATE TO REOPEN as issues like this are helpful (other "discussion" oriented issues have been moved, but ones like this, on documentation improvement, to me are as [if not more important] than "bugs").

Also, please feel free to contribute directly to the wiki as well! Or it looks like https://github.com/amark/gun/issues/70 is being used as a general open issue as well. Cheers! Thanks guys!