Janusgraph: Not able to delete ghost vertices!!

[anamika121]

Over the period of time we are observing some ghost vertices on our production setup where if we do gremlin query for having msid =18038893 shows ghost vertex.
gremlin> g.V().has('msid',18038893)
==>v[7244255240]

but if we try to delete this vertex it shows that Vertex with id 7433638024 was removed.
gremlin> g.V().has('msid',18038893).drop()
Vertex with id 7244255240 was removed.
Type ':help' or ':h' for help.
Display stack trace? [

gremlin> g.V().has('msid',18869037).valueMap()
Vertex with id 7433638024 was removed.
gremlin> g.V().has('msid',18133305)
==>v[7228928032]

I am not able to recreate vertex with given msid again in the graph. It seems that index vertex has already been deleted but index on msid has not been updated correctly and has been corrupted. Kindly suggest how to remove such cases.

[Miroka96]

Similar issue for me:

gremlin> v = g.V(2598101162).next()
==>v[2598101162]
gremlin> g.V(v).outE()
==>e[iq0ledl-16yubm2-3cwl-37s][2598101066-badHost->4168]
gremlin> g.V(v).outE().outV()
==>v[2598101066]
gremlin> g.V(v).outE().outV().next() == v
==>true

I think, my (unique) index is broken, because the same vertex is reachable from two different vertex IDs. And if I step from a vertex out to an edge and back to the outVertex, I get one which is equal but has a different vertex ID.

Maybe my unique Index prevents the duplicates from being found.
I have 400k vertices in my graph, created by only 160k unique data elements. They have been inserted through Spark in parallel with 96 threads.
My backend is Cassandra.

[yosemiteAGM]

Have been having a similar issue:
We've been running a homegrown dataGrooming tool for over a year to look for ghost/phantom vertices and remove them from our graph. Most of them seem to be caused by updates and deletes coming in on different db nodes (we run JanusGraph on HBase) at the same time.

Up until recently, once the vertex was identified, we could just do vertex.remove() to get rid of it. But now, while we get no errors running the remove() command and committing it, the vertices are still in the graph the next time I look. (Like animika121's issue above).

I tried using g.V().drop() just as a change of pace, but that did not help. Short of re-loading the db from a GraphSON snapshot, is there a way to get these vertices out of our graph?
It seems like if a vertex can be retrieved, there should be a way to remove it even if it is in a ghost-like (ie. corrupted) state.

[yosemiteAGM]

NOTE - We have found a work-around for now. Not only can we retrieve a "removed" vertex, but can still update it. If we first add this node's indexed properties back, commit the update and then do the remove again, the node is successfully removed from the db. So it appears that there must have been an index pointing at the node that was not getting cleared when the node was removed. Still hoping that there is a more elegant way to do this?

[pankajydv]

Thanks @yosemiteAGM - we can now at least delete these ghosts. Basically the steps we followed is as follows:

g.V().has('msid',18038893).valueMap()
==>[]
g.V().has('msid',18038893).property('msid', 18038894)
g.V().has('msid',18038894).valueMap()
==>[msid:[18038894]]
g.V().has('msid',18038894).drop()

[mad]

JG has special job for removing ghost vertices GhostVertexRemover that job was fixed in here #1731

[li-boxuan]

As @mad pointed out, GhostVertexRemover job can be used to delete ghost vertices. Using it in MapReduce would need some custom work, though. Created https://github.com/JanusGraph/janusgraph/issues/2555 to enhance it so that it can be used out-of-the-box.