Janusgraph: Search count bug

[robertdale]

From list - https://groups.google.com/d/msg/janusgraph-users/trKs-UdkLkw/-2JGlKlECQAJ

Setup

graph = JanusGraphFactory.open('conf/janusgraph-cassandra-es.properties')
mgmt = graph.openManagement()

name = mgmt.makePropertyKey('name').dataType(String.class).cardinality(SINGLE).make()
fermtype = mgmt.makePropertyKey('ferma_type').dataType(String.class).cardinality(SINGLE).make()

mgmt.buildIndex("byNameMixed", Vertex.class).addKey(name, Mapping.TEXTSTRING.asParameter()).buildMixedIndex("search"); 
mgmt.buildIndex("byFermaType", Vertex.class).addKey(fermtype).buildCompositeIndex();

mgmt.commit()

g = graph.traversal()

for (i = 0; i < 15000; i++) {
   g.addV().property("ferma_type", "com.acme.model.Person").iterate()
}
g.tx().commit()

Actual:

First count() returns 15k results.
Another query is run.
Second count() returns only 4k results.

gremlin> g.V().has("ferma_type", "com.acme.model.Person").count()
==>15000
gremlin> g.V().has("ferma_type", "com.acme.model.Person").count().profile()
==>Traversal Metrics
Step                                                               Count  Traversers       Time (ms)    % Dur
=============================================================================================================
JanusGraphStep([],[ferma_type.eq(com.acme.model...                 15000       15000          14.430    85.52
    \_condition=(ferma_type = com.acme.model.Person)
    \_isFitted=true
    \_query=multiKSQ[1]@2147483647
    \_index=byFermaType
    \_orders=[]
    \_isOrdered=true
  optimization                                                                                 0.094
  backend-query                                                                                0.000
    \_query=byFermaType:multiKSQ[1]@2147483647
  backend-query                                                                                0.000
    \_query=byFermaType:multiKSQ[1]@2147483647
  backend-query                                                                                0.000
    \_query=byFermaType:multiKSQ[1]@2147483647
  backend-query                                                    15000                       7.347
    \_query=byFermaType:multiKSQ[1]@2147483647
CountGlobalStep                                                        1           1           2.443    14.48
                                            >TOTAL                     -           -          16.874        -
gremlin> g.V().has("ferma_type", "com.acme.model.Person").count()
==>15000
gremlin> g.V().has("ferma_type", "com.acme.model.Person").has("name", "John Doe")
gremlin> g.V().has("ferma_type", "com.acme.model.Person").has("name", "John Doe").profile()
==>Traversal Metrics
Step                                                               Count  Traversers       Time (ms)    % Dur
=============================================================================================================
JanusGraphStep([],[ferma_type.eq(com.acme.model...                                             0.562   100.00
    \_condition=(ferma_type = com.acme.model.Person AND name = John Doe)
    \_isFitted=true
    \_query=[(name = John Doe)]:byNameMixed
    \_index=byNameMixed
    \_orders=[]
    \_isOrdered=true
    \_index_impl=search
  optimization                                                                                 0.179
                                            >TOTAL                     -           -           0.562        -
gremlin> g.V().has("ferma_type", "com.acme.model.Person").count()
==>4000
gremlin> g.V().has("ferma_type", "com.acme.model.Person").count().profile()
==>Traversal Metrics
Step                                                               Count  Traversers       Time (ms)    % Dur
=============================================================================================================
JanusGraphStep([],[ferma_type.eq(com.acme.model...                  4000        4000           3.352    86.66
    \_condition=(ferma_type = com.acme.model.Person)
    \_isFitted=true
    \_query=multiKSQ[1]@2147483647
    \_index=byFermaType
    \_orders=[]
    \_isOrdered=true
  optimization                                                                                 0.071
CountGlobalStep                                                        1           1           0.516    13.34
                                            >TOTAL                     -           -           3.868        -
gremlin> g.V().has("ferma_type", "com.acme.model.Person").count()
==>4000

Expected:

Second call to g.V().has("ferma_type", "com.acme.model.Person").count() should have consistently returned 15k results.

Note

Calling g.tx().commit() resets whatever is going on here as subsequently callingcount() will return the correct results.

[ThijsBroersen]

This looks equal to my post on gitter on nov 15th. The query optimizer inserted a limit(4000), perhaps here too. Turning off smart-limit (query.smart-limit=false) solved the issue.

[davidclement90]

Since with this PR https://github.com/JanusGraph/janusgraph/pull/481 we can do deep paging in mixed benckend, should we set it default value to false ?

[robertdale]

This is @ThijsBroersen's post https://gitter.im/janusgraph/janusgraph?at=5a0c6425f257ad91099b01a9
It is clearly seen in his profile() that the limit is set. It is completely missing in the above output. I think profile() should always show any applied limit.

[TitiHl]

Hi @robertdale and @pluradj ,
We have seen the same issue as describe in this post:
https://groups.google.com/forum/#!topic/janusgraph-users/K1vIQA4Anog

and for our case, this only happens on the mixed index.
Anyone looking at this issue or what is the best part to look at in the JanusGraph code?
Thanks,
Alex