Janusgraph: "Could not find type for id" error when traversing graph / Corrupted label in a vertex

Created on 7 Aug 2020  路  6Comments  路  Source: JanusGraph/janusgraph

The schema is pretty simple (few labels, properties, constraints and indexes).

  • JanusGraph 0.4.1
  • Storage Backend: Scylla 3.2.0
  • 40k vertices

For me it looks like one day in the database appears a broken vertex which makes impossible to run any queries that iterate all vertices and contains "hasLabel()" method.
We are still investigating this issue, but it does not seem like any schema changes were made right before we noticed it last time.
This is how it looks:

gremlin> g.V().hasLabel('userProduct')
12:05:44 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(~label = userProduct)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.


gremlin> g.V().hasLabel('nnname')
12:49:34 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(~label = nnname)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]

Meanwhile method "has()" works fine:

gremlin> 
gremlin> g.V().has('name', 'ddd')
12:24:37 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(name = ddd)]. For better performance, use indexes
==>v[368767136]
==>v[123867144]
gremlin>

When trying to get that vertex properties we get only ([id:155149,label:vertex]):

gremlin> g.V(155149)
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]

gremlin> 
gremlin> g.V(155149).count()
==>1

gremlin> g.V(155149).elementMap()
==>[id:155149,label:vertex]
gremlin>

P.S.
The problem is critical and already reached our production database...:((

kinbupossible
Source
picture valpid13

Most helpful comment

@vladuk - for sure. We patched version 0.3.3 as that's what we're running, but the code is still present in 0.4.1 and 0.5.2 unchanged, and the update _should_ work on both versions (though obviously test prior to applying anything). The fix involves the name method of the JanusGraphSchemaVertex.class file.

    @Override
    public String name() {
        if (name == null) {
            JanusGraphVertexProperty<String> p;
            if (isLoaded()) {
                StandardJanusGraphTx tx = tx();
                p = (JanusGraphVertexProperty) Iterables.getOnlyElement(RelationConstructor.readRelation(this,
                        tx.getGraph().getSchemaCache().getSchemaRelations(longId(), BaseKey.SchemaName, Direction.OUT),
                        tx), null);
            } else {
                p = Iterables.getOnlyElement(query().type(BaseKey.SchemaName).properties(), null);
            }
            //Preconditions.checkState(p!=null,"Could not find type for id: %s", longId());
            if (p != null) {
                name = p.value();
            }
        }
        //assert name != null;
        if (name !=null ) {
            return JanusGraphSchemaCategory.getName(name);
        } else {
            return "missing";
        }

    }

You can see we've essentially just removed the precondition for now, along with the null assertion and replaced them with values if they're null to ensure that the gremlin can return appropriately. Long-term there should also likely be a WARN message that's printed here as well so that there is a state you can watch for if this issue happens in a prod environment, along with a more unique name for the field than "missing" (as it's theoretically possible that a user has a key named "missing", though such a key still would work with this fix, just probably confuse downstream code).

You can then either build your own JanusGraph from scratch, or we just built the core jar (use mvn jar in the janusgraph-core folder) and swap the core jar in your gremlin server to update the patch. We were trying to fix a prod issue (as it sounds like you guys might be) so we kinda did a quick-n-dirty to get it resolved.

picture PatrickRice-KSC on 17 Aug 2020
👍3

All 6 comments

It would be good if you add the stacktrace

farodin91 picture farodin91 on 9 Aug 2020
👍2

It would be good if you add the stacktrace

java.lang.NullPointerException: Could not find type for id: 155149
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:250)
    at org.janusgraph.graphdb.types.vertices.JanusGraphSchemaVertex.name(JanusGraphSchemaVertex.java:57)
    at org.janusgraph.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:121)
    at org.janusgraph.graphdb.types.system.ImplicitKey.computeProperty(ImplicitKey.java:83)
    at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeImplicitKeyQuery(BasicVertexCentricQueryBuilder.java:211)
    at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:99)
    at org.janusgraph.graphdb.util.ElementHelper.getValues(ElementHelper.java:41)
    at org.janusgraph.graphdb.query.condition.PredicateCondition.evaluate(PredicateCondition.java:68)
    at org.janusgraph.graphdb.query.condition.And.evaluate(And.java:55)
    at org.janusgraph.graphdb.query.graph.GraphCentricQuery.matches(GraphCentricQuery.java:153)
    at org.janusgraph.graphdb.query.QueryProcessor.lambda$getFilterIterator$2(QueryProcessor.java:133)
    at com.google.common.collect.Iterators$7.computeNext(Iterators.java:652)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.janusgraph.graphdb.query.ResultSetIterator.nextInternal(ResultSetIterator.java:54)
    at org.janusgraph.graphdb.query.ResultSetIterator.<init>(ResultSetIterator.java:44)
    at org.janusgraph.graphdb.query.QueryProcessor.iterator(QueryProcessor.java:66)
    at com.google.common.collect.Iterables$7.iterator(Iterables.java:613)
    at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.executeGraphCentryQuery(JanusGraphStep.java:156)
    at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.lambda$null$1(JanusGraphStep.java:95)
    at java.lang.Iterable.forEach(Iterable.java:75)
    at org.janusgraph.graphdb.tinkerpop.optimize.JanusGraphStep.lambda$new$2(JanusGraphStep.java:95)
    at org.apache.tinkerpop.gremlin.process.traversal.step.map.GraphStep.processNextStart(GraphStep.java:157)
    at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
    at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:197)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
    at org.apache.tinkerpop.gremlin.console.Console$_closure3.doCall(Console.groovy:255)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:37)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:52)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at org.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:463)
    at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:168)
    at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:201)
    at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
    at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
    at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:83)
    at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)
    at org.codehaus.groovy.tools.shell.Shell$leftShift$1.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
    at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:93)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:138)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
    at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
    at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:97)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
    at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:168)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:234)
    at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:502)

@farodin91

valpid13 picture valpid13 on 10 Aug 2020

One more interesting detail which could help.
"ID" and "label" of the vertex are only returned if the error raised in current transaction:

gremlin> g.tx().rollback()
==>null
gremlin> 
gremlin> g.V(155149).elementMap()
gremlin> 
gremlin> g.V().hasLabel('nnname')
08:43:40 WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [(~label = nnname)]. For better performance, use indexes
Could not find type for id: 155149
Type ':help' or ':h' for help.
Display stack trace? [yN]
gremlin> 
gremlin> g.V(155149).elementMap()
==>[id:155149,label:vertex]
gremlin> 
gremlin> 
gremlin> g.tx().rollback()
==>null
gremlin> 
gremlin> g.V(155149).elementMap()
gremlin>
valpid13 picture valpid13 on 10 Aug 2020

We have seen this as well, and wound up compiling our own version of JG (specifically, just the JanusGraphSchemaVertex class) that replaced that null pointer exception with code that returned "missing" (string literal) for the name instead of throwing a null pointer exception, then using a local gremlin connection to query / investigate / remove the offending nodes.

It's a bit frustrating that this check explicitly prevents you from writing any queries (even removes) that would actually fix the issue. Seems like this should be a WARN followed by a blank string return instead of an NPE that blocks all the queries that would be required to actually fix this issue.

PatrickRice-KSC picture PatrickRice-KSC on 12 Aug 2020
👍2

@PatrickRice-KSC
Thank you for your reply. Could you share a patch for that specific JG version ?

vladuk picture vladuk on 13 Aug 2020
👀1

@vladuk - for sure. We patched version 0.3.3 as that's what we're running, but the code is still present in 0.4.1 and 0.5.2 unchanged, and the update _should_ work on both versions (though obviously test prior to applying anything). The fix involves the name method of the JanusGraphSchemaVertex.class file.

    @Override
    public String name() {
        if (name == null) {
            JanusGraphVertexProperty<String> p;
            if (isLoaded()) {
                StandardJanusGraphTx tx = tx();
                p = (JanusGraphVertexProperty) Iterables.getOnlyElement(RelationConstructor.readRelation(this,
                        tx.getGraph().getSchemaCache().getSchemaRelations(longId(), BaseKey.SchemaName, Direction.OUT),
                        tx), null);
            } else {
                p = Iterables.getOnlyElement(query().type(BaseKey.SchemaName).properties(), null);
            }
            //Preconditions.checkState(p!=null,"Could not find type for id: %s", longId());
            if (p != null) {
                name = p.value();
            }
        }
        //assert name != null;
        if (name !=null ) {
            return JanusGraphSchemaCategory.getName(name);
        } else {
            return "missing";
        }

    }

You can see we've essentially just removed the precondition for now, along with the null assertion and replaced them with values if they're null to ensure that the gremlin can return appropriately. Long-term there should also likely be a WARN message that's printed here as well so that there is a state you can watch for if this issue happens in a prod environment, along with a more unique name for the field than "missing" (as it's theoretically possible that a user has a key named "missing", though such a key still would work with this fix, just probably confuse downstream code).

You can then either build your own JanusGraph from scratch, or we just built the core jar (use mvn jar in the janusgraph-core folder) and swap the core jar in your gremlin server to update the patch. We were trying to fix a prod issue (as it sounds like you guys might be) so we kinda did a quick-n-dirty to get it resolved.

PatrickRice-KSC picture PatrickRice-KSC on 17 Aug 2020
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pluradj picture pluradj  路  4Comments
amcp picture amcp  路  5Comments
amyth picture amyth  路  4Comments
chupman picture chupman  路  3Comments
code5hot picture code5hot  路  4Comments