Janusgraph: Dedicated Scylla backend

[FlorianHockmann]

Scylla now has its own Java driver which is a fork of DataStax's CQL driver we currently use for the CQL backend with some Scylla specific optimizations. These optimizations are described in this presentation from the 2018 Scylla summit which hints at significant latency reductions thanks to shard awareness in the driver.

It would be great if we could use this driver together with our existing CQL backend, so we get the performance improvements without having to maintain yet another full backend.

[dorlaor]

There is also BYPASS_CACHE flag that allows per query cache bypass so scans do not pollute the cache.

[FlorianHockmann]

I have started working on this and got a first implementation that should make it possible to include JanusGraph embedded and use the Scylla driver instead of the DataStax driver. Some first JanusGraph tests are at least running and I get a log message from the driver telling that the "advanced" driver is used which is the Scylla driver.
But I don't know yet how to make this work for JanusGraph Server as that also has a dependency on our usual CQL backend which still uses the DataStax driver. So it has both drivers on its classpath and then probably the first one found on the classpath will be used, irrespective of whether cql or scylla is configured as the storage backend.

If anyone has an idea how we could select at runtime which driver we want to use, that would much appreciated.

[porunov]

@FlorianHockmann I'm not sure if it will work after we update DataStax driver to 4.x (#2169).
ScyllaDB driver is based on DataStax driver 3.x. DataStax driver 3 and 4 are binary incompatible. Most likely you will need to copy all content of the current CQLStoreManager into ScyllaStoreManager.

Also, I guess it would be interesting to compare DataStax 4.8.0 and Scylla 3.7.1-scylla-2 drivers. ScyllaDB java driver was a fork of DataStax Cassandra driver (I guess 3.7.1). There were many bug fixes and improvements after that in 3.x branch which Scylla driver 3.7.1-scylla-2 didn't get (~8 months of development. Changelog). Moreover, this year the driver 4.x is even more actively developed than the driver 3.x.
If ScyllaDB driver compared to DataStax 3.x branch, there are not many changes actually:
https://github.com/datastax/java-driver/compare/3.x...scylladb:3.7.1-scylla

I didn't research them, but I guess the changes are related to improvement of the token-aware policy as described here https://docs.scylladb.com/using-scylla/scylla-java-driver/ and a possibility to set a BYPASS_CACHE flag as noted by @dorlaor . That's definitely great and I would love to use it but the fact that ScyllaDB driver doesn't provide bug fixes which are implemented in recent DataStax driver gives me some doubts.

Here are missing commits from DataStax 3.x branch which ScyllaDB Java driver misses.
https://github.com/scylladb/java-driver/compare/3.7.1-scylla...datastax:3.x

That said, I'm totally fine with adding ScyllaDB driver into JanusGraph. I just think it could be useful if we could compare them.

[FlorianHockmann]

Thanks for your detailed comment, @porunov! You definitely make some good points and I didn't follow the work on updating the driver to version 4 close so I wasn't aware of this.
In general I would really like to avoid having to copy lots of code just to get the Scylla driver.
But it looks like it's best right now for us to wait and see whether Scylla will also upgrade their driver to version 4 and also whether they will then maintain it, at least by merging in fixes / improvements from the upstream DataStax driver.

[dorlaor]

@FlorianHockmann cheers for picking this up!
@porunov Good points, we should rebase the scylla driver over 4.x
@haaawk when can we rebase?

[haaawk]

I finished rebasing it last week, @dorlaor. If everything goes well the 4.x driver should be published to Maven repository this week.

[sergeymetallic]

Looks like there is a Scylla driver of version 4.9.0 released (https://github.com/scylladb/java-driver/commits/4.9.0-scylla-0) which is based on Datastax driver version 4.9.0. So theoretically there should not be any problem if we use it with the latest Janusgraph?