Lizardfs: Wishlist 5.x - Multimaster Replication

[Zorlin]

It would be awesome if we had a multimaster replication scheme in LFS 5.

You could use something like the Galera library, which is currently used by MariaDB Cluster and Percona XtraDB Cluster. It is designed to be fairly agnostic as to the database in use, so it hopefully wouldn't be too difficult to adapt it to LizardFS.

[drauschenbach]

Multi-master would really bring LizardFS into the cloud world and put it at the head of the pack. Supporting Kubernetes as a runtime environment would make a decent design target, although the LizardFS stack should obviously continue to work outside of a cluster manager on a bare OS. I'd like to do this: kubectl scale lizardfs-master --replicas=5.

[cloudweavers]

Most of the pieces for making a multimaster system are already there; moving the metadata store to a cassandra-like store (in another project we use http://www.scylladb.com/ that is tens of times faster than pure cassandra) would allow you to point individual client transactions to different metadata stores. But this may have a substantial impact on performance. When the metadata is handled in a single place, no need for quorum or vector clocks are needed; this means that you are bounded by the response time of the metadata server only. Or you can trade speed for the possibility of losing writes under concurrency (like cassandra do: https://aphyr.com/posts/294-jepsen-cassandra ) which is by definition a very bad thing :-) Doing distributed systems is notoriously hard, and I believe that this may be one of the most complex things to implement (properly).

[Zorlin]

@cloudweavers Thanks for your insights as usual :) Completely agree.

[cloudweavers]

One of the things we were looking at is Alluxio, http://www.alluxio.org/ that implement a memory-backed (with the possibility to commit to disk) distributed store exactly for this kind of things. It's in java (belch) but it's a speed demon, and allows persistent backing store on any persistent medium like ssd or disk (including tiering). It is one of the things we are looking to see if it can be used with Lizard.

[drauschenbach]

I'm also watching ScyllaDB closely. I've always loved the Cassandra transaction semantics, but was never willing to operate it due to its memory profile.

DB choice can easily digress into chaos, since we all have different operational requirements. As low hanging fruit, I've always favored a storage protocol over a storage API.

For example, if the lizardfs-master had a storage API, I could build a ScyllaDB or Redis or RethinkDB storage adapter, but then I'd have to spend a lot of time learning the C++ make system and be done in a week or two. But if on the other hand lizardfs-master supported configuration via a STORE_URL=redis://myhost/1 env var or config entry, then I could build a gateway that spoke the Redis protocol in my favorite language, and be done in a day or two.

Of course I love having a stateless, shared-nothing, crash-only architecture. But I don't want to burden the team with talk of a huge change.

[cloudweavers]

@drauschenbach absolutely. I believe that integration of existing pieces would allow Lizard to grow and expand, without burdening the developers too much.