Janusgraph: range(low, high) doesn't work with mixed index

[pluradj]

Following on #252, it has been noted that order().by(property).range(low, high) does not take advantage of the mixed index. @davidclement90 noted:

IndexQuery do not support offset. So it just support ranging with no offset range(0,10). Range with offset will be do in-memory with the high range as query limit. https://github.com/JanusGraph/janusgraph/blob/a127060de7125a94261cc313697ce778ac7ec22d/janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/HasStepFolder.java#L213

Also more recently on janusgraph-users:

Query:

    g.V().hasLabel("location").
      has("location_type", 3).
      order().by("location_id").
      range(100000, 111000).toList()

Result: 8 minutes

[porunov]

The bigger offset the longer time it takes to be executed.
Query:

    g.V().hasLabel("location").
      has("location_type", 3).
      order().by("location_id").
      range(4000000, 4011000).toList()

Result: 395 minutes.

[nickyongzhang]

has the problem been solved? I come across with the same problem.

[porunov]

@nickyongzhang The issue isn't resolved yet. I think IndexQuery should contain offset parameter. Then we would be able to use it in indices.
For example, to enable ElasticSearch to use offset we should get offset from IndexQuery and set it here:
https://github.com/JanusGraph/janusgraph/blob/09fad1021e0a80fd0307ec267f827b64d33c4d65/janusgraph-es/src/main/java/org/janusgraph/diskstorage/es/ElasticSearchIndex.java#L1102

Also, notice that using big offset with ElasticSearch, Solr or Lucene isn't very good because of the deep pagination problem.
Of course, implementing using from and limit in ElasticSearch is much more efficient than the current in-memory filtering, but it won't help with deep pagination still.

I would suggest to implement something like search_after of ElasticSearch in JanusGraph but it isn't a trivial task.
Right now to use deep pagination in the above query we could use a unique parameter (in the above situation I use location_id). Then the query would look like:

    g.V().hasLabel("location").
      has("location_type", 3).
      has("location_id", P.gt(lastLocationId)).
      order().by("location_id").
      limit(1000).toList()

This query isn't ideal because ElasticSearch should still find and sort all locations with location_id > lastLocationId. In my situation location_id parameter is unique per each location and has a small gap between neighbors' location_id that is why I can predict that all returned locations will have location_id less than lastLocationId + 3000. So, I optimize my query to be like:

    g.V().hasLabel("location").
      has("location_type", 3).
      has("location_id", P.gt(lastLocationId)).
      has("location_id", P.lt(lastLocationId + 3000)).
      order().by("location_id").
      limit(1000).toList()

Hope it helps somehow.

[nickyongzhang]

@porunov Thank you for your suggestion. It's a brilliant solution although it's not the most straightforward.