Janusgraph: Index Backend always using Elasticsearch

[jamesdbaker]

Using the latest release (janusgraph-0.2.0-hadoop2.zip), it appears that regardless of the provided configuration, an Elasticsearch index backend is always used. I have tried all of the following configurations, and all of them throw an error saying JanusGraph is unable to connect to Elasticsearch (despite the configurations specifying an index backend other than Elasticsearch should be used).

The command I'm using to start JanusGraph is: bin/gremlin-server.sh conf/gremlin-server/gremlin-server.yaml, with gremlin-server.yaml configured to point to the following properties files.

Configuration 1 (based on janusgraph-hbase.properties):

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=hbase
storage.hostname=127.0.0.1
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.5

index.search.backend=lucene

Configuration 2:

...

index.search.backend=org.janusgraph.diskstorage.lucene.LuceneIndex

Configuration 3:

...

index.search.backend=solr

Configuration 4:

...

index.search.backend=anyoldvalue

The relevant bit of the log (which is the same regardless of configuration) is:

3338 [main] INFO  org.janusgraph.diskstorage.Backend  - Configuring index [search]
4155 [main] WARN  org.janusgraph.diskstorage.es.rest.RestElasticSearchClient  - Unable to determine Elasticsearch server version. Default to FIVE.
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.lang.Thread.run(Thread.java:748)
4178 [main] WARN  org.apache.tinkerpop.gremlin.server.GremlinServer  - Graph [graph] configured at [conf/gremlin-server/janusgraph-hbase-server.properties] could not be instantiated and will not be available in Gremlin Server.  GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
    at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
    at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
    at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:110)
    at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:89)
    at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:110)
    at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:354)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
    ... 13 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
    at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69)
    at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
    at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:464)
    at org.janusgraph.diskstorage.Backend.<init>(Backend.java:149)
    at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1897)
    at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:136)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
    ... 18 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
    ... 26 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Connection refused
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:231)
    ... 31 more
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.lang.Thread.run(Thread.java:748)

If I don't specify an index backend, then the server starts as expected with no errors.

[robertdale]

Likely, you initially started janusgraph with a configuration containing index.search.backend=elasticsearch. This would have stored the configuration in the database. Afterwards, that configuration property is ignored from the file and pulled only from the database.

index.*.backend is a GLOBAL_OFFLINE configuration - http://docs.janusgraph.org/latest/config-ref.html#_index

See how to change GLOBAL_OFFLINE configurations - http://docs.janusgraph.org/latest/configuration.html#configuration-global

[jamesdbaker]

I'm not sure that is the case, because:
a) I don't recall initially trying elasticsearch as a backend, the intention was always to use lucene
b) It is only an issue when I try to set an index back end in the file config (i.e. if I don't set one then it works fine).

Is there anyway of examining the database to check this? (If it makes a difference, I'm using the HBase storage back end)

[robertdale]

mgmt = graph.openManagement()
mgmt.get('index.search.backend')

[jamesdbaker]

Yes, it is showing up as elasticsearch. Is it possible to change it? Or would I need to create a new database?

Also, does it default to elasticsearch if I don't set one, I'm still confused as to how it became elasticsearch in the first place?

[fbalicchia]

Hi @robertdale ,
I've notice the same behavior. Using cql as starage backend Janus v0.0.2 use as index.search.backend Elasticsearch ignoring my properties file

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.cassandra.keyspace=jgex
storage.hostname=127.0.0.1
index.search.backend=lucene
index.search.directory=../db/lucene

instead using cassandra as storage engine is correct. It uses Lucene as search index backend

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandra
storage.cassandra.keyspace=jgex
storage.hostname=127.0.0.1
index.search.backend=lucene
index.search.directory=../db/lucene

both test are made starting from a clean enviroment.

From documentation I don't see nothing about this behavior but probably I'm in fault.
IMHO this smell like a bug
Am I in wrong ?

[robertdale]

@fbalicchia Unable to reproduce

Steps

cd apache-cassandra-3.0.13/
rm -rf data
./bin/cassandra -p run.pid >console.log
cd ../janusgraph-0.2.0-hadoop2/
rm -rf db
cat > conf/foo.properties << EOF                               
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.cassandra.keyspace=jgex
storage.hostname=127.0.0.1
index.search.backend=lucene
index.search.directory=../db/lucene
EOF
./bin/gremlin.sh << EOF
graph = JanusGraphFactory.open('conf/foo.properties')
mgmt = graph.openManagement()
mgmt.get('index.search.backend')
name = mgmt.makePropertyKey('name').dataType(String.class).make()
mgmt.buildIndex('name', Vertex.class).addKey(name).buildMixedIndex("search")
mgmt.commit()

g = graph.traversal()
g.addV().property('name', 'janus')
g.tx().commit()
g.V().has('name', textContainsPrefix('j')).valueMap(true)
g.V().has('name', textContainsPrefix('j')).profile()
:q
EOF

find db

Output

[rdale@xxx ~]$ cd apache-cassandra-3.0.13/
[rdale@xxx apache-cassandra-3.0.13]$ rm -rf data
[rdale@xxx apache-cassandra-3.0.13]$ ./bin/cassandra -p run.pid >console.log
[rdale@xxx apache-cassandra-3.0.13]$ cd ../janusgraph-0.2.0-hadoop2/
[rdale@xxx janusgraph-0.2.0-hadoop2]$ rm -rf db
[rdale@xxx janusgraph-0.2.0-hadoop2]$ cat > conf/foo.properties << EOF                               
> gremlin.graph=org.janusgraph.core.JanusGraphFactory
> storage.backend=cql
> storage.cassandra.keyspace=jgex
> storage.hostname=127.0.0.1
> index.search.backend=lucene
> index.search.directory=../db/lucene
> EOF
[rdale@xxx janusgraph-0.2.0-hadoop2]$ ./bin/gremlin.sh << EOF
> graph = JanusGraphFactory.open('conf/foo.properties')
> mgmt = graph.openManagement()
> mgmt.get('index.search.backend')
> name = mgmt.makePropertyKey('name').dataType(String.class).make()
> mgmt.buildIndex('name', Vertex.class).addKey(name).buildMixedIndex("search")
> mgmt.commit()
> 
> g = graph.traversal()
> g.addV().property('name', 'janus')
> g.tx().commit()
> g.V().has('name', textContainsPrefix('j')).valueMap(true)
> g.V().has('name', textContainsPrefix('j')).profile()
> :q
> EOF

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
plugin activated: janusgraph.imports
plugin activated: tinkerpop.server
plugin activated: tinkerpop.utilities
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/janusgraph-0.2.0-hadoop2/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/janusgraph-0.2.0-hadoop2/lib/logback-classic-1.1.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16:49:19 WARN  org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
plugin activated: tinkerpop.hadoop
plugin activated: tinkerpop.spark
plugin activated: tinkerpop.tinkergraph
gremlin> graph = JanusGraphFactory.open('conf/foo.properties')
==>standardjanusgraph[cql:[127.0.0.1]]
gremlin> mgmt = graph.openManagement()
==>org.janusgraph.graphdb.database.management.ManagementSystem@839df62
gremlin> mgmt.get('index.search.backend')
==>lucene
gremlin> name = mgmt.makePropertyKey('name').dataType(String.class).make()
==>name
gremlin> mgmt.buildIndex('name', Vertex.class).addKey(name).buildMixedIndex("search")
==>name
gremlin> mgmt.commit()
==>null
gremlin> 
gremlin> g = graph.traversal()
==>graphtraversalsource[standardjanusgraph[cql:[127.0.0.1]], standard]
gremlin> g.addV().property('name', 'janus')
==>v[4208]
gremlin> g.tx().commit()
==>null
gremlin> g.V().has('name', textContainsPrefix('j')).valueMap(true)
==>[id:4208,label:vertex,name:[janus]]
gremlin> g.V().has('name', textContainsPrefix('j')).profile()
==>Traversal Metrics
Step                                                               Count  Traversers       Time (ms)    % Dur
=============================================================================================================
JanusGraphStep([],[name.textContainsPrefix(j)])                        1           1           1.143   100.00
    \_condition=(name textContainsPrefix j)
    \_isFitted=true
    \_query=[(name textContainsPrefix j)]:name
    \_index=name
    \_orders=[]
    \_isOrdered=true
    \_index_impl=search
  optimization                                                                                 0.174
                                            >TOTAL                     -           -           1.143        -
gremlin> :q
[rdale@xxx janusgraph-0.2.0-hadoop2]$ 
[rdale@xxx janusgraph-0.2.0-hadoop2]$ find db
db
db/lucene
db/lucene/name
db/lucene/name/_0.si
db/lucene/name/segments_1
db/lucene/name/_0.cfs
db/lucene/name/write.lock
db/lucene/name/_0.cfe
[rdale@xxx janusgraph-0.2.0-hadoop2]$ 

• hostnames have been changed to protect the innocent.

[fbalicchia]

Hi,
thanks for response, I've prepare an example where is possible to verify it using java api.
Please Try to checkout example.
The snipped that I've used to reproduce the problem is this

JanusGraph graph = JanusGraphFactory.open("plain-java-example/conf/janusgraph-cassandra-lucene.properties");
JanusGraphManagement janusGraphManagement = graph.openManagement();
String searchBackend = janusGraphManagement.get("index.search.backend");
LOGGER.info("backend {}" , searchBackend);

Case OK

after typing

bin/janusgraph.sh clean
bin/janusgraph.sh start

execute

with this conf

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandra
storage.cassandra.keyspace=jgex
storage.hostname=127.0.0.1
index.search.backend=lucene
index.search.directory=../db/lucene

INFO i.b.example.Example3TraversalApi - backend lucene

Case KO

bin/janusgraph.sh stop
bin/janusgraph.sh clean
bin/janusgraph.sh start

_remove db folder from tiny project_

use this conf

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.cassandra.keyspace=jgex
storage.hostname=127.0.0.1
index.search.backend=lucene
index.search.directory=../db/lucene

relaunch

_INFO i.b.example.Example3TraversalApi - backend elasticsearch_

In this case the only thing that I change is configuration storage.backend

from my point of view seems all ok. I do not seem to have made any amazed mistakes ?
Thanks for attention

[robertdale]

@fbalicchia I see the problem. For your cql config, you are not setting the keyspace hence you are picking up the default keyspace and configuration.
You should be using storage.cql.keyspace=jgex with the cql driver. Each driver has its own namespace.

[fbalicchia]

@robertdale thanks. It works for me. I was starting to debug to figure out the problem, but I did not understand why in org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration#GraphDatabaseConfiguration
globalWrite at the first startup was frozen....

Thanks again .. from my point of view this issue can be closed

[pluradj]

The documentation on CQL storage backend #385 should cover this scenario. I can imagine people running into this same issue again.

[msure4]

@robertdale Hello Robert, i try to install janus graph db on hbase as backend and serach index is solr. I did all the step which you provide on previoues comments. But still i'm getting same error.
My janusgraph location: /var/lib/
foo.properties location: /tmp/
Here are my steps:

1. cat > /tmp/foo.properties << EOF
   gremlin.graph=org.janusgraph.core.JanusGraphFactory
   storage.backend=hbase
   storage.hbase.keyspace=jgex
   storage.hostname=127.0.0.1
   index.search.backend=solr
   index.search.directory=../db/solr
   EOF

1. ./bin/gremlin.sh << EOF
   graph = JanusGraphFactory.open('/tmp/foo.properties')
   mgmt = graph.openManagement()
   mgmt.get('index.search.backend')
   name = mgmt.makePropertyKey('name').dataType(String.class).make()
   mgmt.buildIndex('name', Vertex.class).addKey(name).buildMixedIndex("search")
   mgmt.commit()
   g = graph.traversal()
   g.addV().property('name', 'janus')
   g.tx().commit()
   g.V().has('name', textContainsPrefix('j')).valueMap(true)
   g.V().has('name', textContainsPrefix('j')).profile()
   :q
   EOF

Error:
16:42:28 WARN org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration - Local setting index.search.backend=solr (Type: GLOBAL_OFFLINE) is overridden by globally managed value (elasticsearch). Use the ManagementSystem interface instead of the local configuration to control this setting.
16:42:29 WARN org.janusgraph.diskstorage.es.rest.RestElasticSearchClient - Unable to determine Elasticsearch server version. Default to FIVE.
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
Type ':help' or ':h' for help.
Display stack trace? [yN]mgmt = graph.openManagement()

Please help me to resolve this issuse.

Thank you in Advance!!

Note: In my system i cuoldn't install any of cassandra,lunce and ES. I'm using backend as defult hbase and search index is solr which it came with CDH 5.12.

[Stefanqn]

tbh for me it is not expected behavior: If I start JanusGraph w/ a bad configuration, switch to a good one, but it still fails. This happens due to the fact that it now resolves the index.search.backend setting from a DB, not from the configuration file any more.

[Mhs-Aaron]

where can i find the example for use hbase an es with janusgraph???
i I've prepared to verify it using java api. but only can see data in hbase , es is null ,how to understand this??