Janusgraph: CassandraBinaryRecordReader not compatible with Cassandra 3.0+

[twilmes]

JanusGraph comes bundled with Cassandra 2.1.9 and relies on 2.1.9's org.apache.cassandra.hadoop.ColumnFamilyRecordReader to read data data out of Cassandra into TinkerPop GraphComputer implementations such as the SparkGraphComputer. As of Cassandra 3.0, the schema metadata layout has been changed which causes an incompatibility where Janus's Cassandra dependency is looking for a schema table that no longer exists. More details on the Cassandra change can be found here. This effectively breaks analytic query, import, and export capabilities if you are running Cassandra 3.0+.

To reproduce, simply install any version of Cassandra 3.0 or above, point Janus at it and start it up and then attempt to run an analytics query.

graph = GraphFactory.open('conf/hadoop-graph/read-cassandra.properties')
g = graph.traversal().withComputer(SparkGraphComputer)
g.V().count()

// relevant portion of exception below

Caused by: InvalidRequestException(why:unconfigured table schema_columnfamilies)
        at org.apache.cassandra.thrift.Cassandra$execute_cql3_query_result$execute_cql3_query_resultStandardScheme.read(Cassandra.java:50297)
        at org.apache.cassandra.thrift.Cassandra$execute_cql3_query_result$execute_cql3_query_resultStandardScheme.read(Cassandra.java:50274)
        at org.apache.cassandra.thrift.Cassandra$execute_cql3_query_result.read(Cassandra.java:50189)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
        at org.apache.cassandra.thrift.Cassandra$Client.recv_execute_cql3_query(Cassandra.java:1734)
        at org.apache.cassandra.thrift.Cassandra$Client.execute_cql3_query(Cassandra.java:1719)
        at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.<init>(ColumnFamilyRecordReader.java:238)

[kedarmhaswade]

@twilmes Do you know of a workaround, or is it basically that OLAP queries are broken with JanusGraph and Cassandra 3.x?

[twilmes]

@kedarmhaswade I do not know of a workaround on this one.

[kedarmhaswade]

Thanks @twilmes. We're thinking of reimplementing some of the interfaces of the Hadoop MapReduce client in JG (e.g. instead of wrapping the org.apache.cassandra.hadoop.ColumnFamilyRecordReader, wrap org.apache.cassandra.hadoop.cql3.CqlRecordReader in org.janusgraph.hadoop.formats.cassandra.CassandraBinaryRecordReader and do the appropriate mapping of the binary types). Do you think such an approach (of replacing ColumnFamilyRecordReader) will work?

[kedarmhaswade]

The other question is whether it is okay for JG to drop the support for Cassandra-2. Currently, the janusgraph-cassandra module depends on:

        <dependency>
            <groupId>org.apache.cassandra</groupId>
            <artifactId>cassandra-all</artifactId>
            <version>${cassandra.version}</version> <!-- defined as 2.1.9 -->
        </dependency>

Since there are incompatible changes, it appears that if we were to move JG to use Cassandra-3, we'd have to drop the support for Cassandra-2 (we should not be bundling cassandra-all with both 2.1.9 and the latest 3.1.0 versions in JG), right?

Also of interest to me is this Cassandra mailing list thread which asks it back to Cassandra developers about the possible replacement of the ColumnFamilyRecordReader class which uses the internal schema of Cassandra-2. I checked that this class exists on the cassandra-2.2 branch of Cassandra sources, but has gone from its trunk. This observation makes our current issue a little different from the Cassandra bug report cited above in this report.

Any suggestions, @twilmes ?

[kedarmhaswade]

As discussed on the dev mailing list, I am going to try to replace the use of ColumnFamilyRecordReader by something that Cassandra-3 likes. I am _not_ going to make JanusGraph require Cassandra-3, because moving JG to use Cassandra-3 instead of Cassandra-2.1.9 (that it currently bundles) should be taken up as a separate task since there are quite a few dependencies (a casual change of cassandra.version to 3.10.0 in the pom.xml breaks the build with hundreds of errors).

I have to leverage the Cassandra class org.apache.cassandra.hadoop.cql3.CqlRecordReader to get rid of ColumnFamilyRecordReader. But CqlRecordReader, in Cassandra-3 uses the DataStax CQL driver. So, since I can't upgrade JG to use Cassandra-3 (first paragraph), I am going to work on recreating the Cassandra-3 CqlRecordReader in some shape and form and use it for getting the data out of Cassandra-3. In the process, I will have to introduce a dependency on DataStax Driver.

[kedarmhaswade]

@twilmes Can you assign this issue to me? I have signed the contributor agreement.

[amcp]

@kedarmhaswade I think you need to be on the committer list for us to be able to use the GitHub assign functionality. Nevertheless I think you should definitely work on this issue. Your intent is clear from the correspondence in this issue.

[kedarmhaswade]

@amcp Thank you! I think I should follow the instructions at https://github.com/JanusGraph/janusgraph/blob/master/CONTRIBUTING.md to get this done. Is that right?

[kedarmhaswade]

See https://github.com/JanusGraph/janusgraph/pull/297.

[debasishdebs]

Hi all, looks like this issue is merged to master, & got released in v 2.0. So, I did download JG2.0, and set my gremlin.hadoop.graphReader, as org.janusgraph.hadoop.formats.cassandra.CqlBridgeRecordReader

So, now I shoule be able to read from Cassandra 3 backend? But I get followng error:

java.lang.IllegalStateException: java.lang.RuntimeException: java.lang.RuntimeException: class org.janusgraph.hadoop.formats.cassandra.CqlBridgeRecordReader not org.apache.hadoop.mapreduce.InputFormat

Anything I'm doing wrong?

[marco-scoppetta]

Hey @debasishdebs ,
looking at read-cassandra-3.properties here
I would say that the only config that needs to be changed is the following:

gremlin.hadoop.graphInputFormat=org.janusgraph.hadoop.formats.cassandra.Cassandra3InputFormat

Or I am missing something @kedarmhaswade ?

Thanks