Janusgraph: JanusGraphFactory:dropStorage deletes Solr cloud core, which cannot recreated by Janus

[wojciechwojcik]

Introduced under https://github.com/JanusGraph/janusgraph/commit/8522787d6fabd08784105a5937aa83b66d9daa5a#diff-84ae066f641a0f2d5e0e9e726acca7b4

Up to now Solr cloud collection was not completely removed by dropStorage method. Right now it is.
The problem is, Janus can drop the collection, but later it cannot re-create it while building mixedIndex

properties
````

storage backend

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=192.168.99.100
schema.default=none

index backend

index.search.backend=solr
index.search.solr.mode=cloud
index.search.solr.zookeeper-url=192.168.99.100:2181
index.search.solr.wait-searcher=true
````

In my case the removed collection is the only one defined on Solr.
Dropping it means it has to be recreated manually which is not acceptable for Junit testing, where dropStorage method is supposed to be used.

````
JanusGraph graph = JanusGraphFactory.open(config);
JanusGraphFactory.drop(janusGraph);

graph = JanusGraphFactory.open(config);
JanusGraphManagement mgmt = graph.openManagement().
mgmt.buildIndex("collectionName").addKey(....).buildMixedIndex("search");
````

Gives an exception:

org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://192.168.99.100:8983/solr: Can not find the specified config set: collectionName at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177) ~[solr-solrj-6.6.0.jar:6.6.0 5c7a7b65d2aa7ce5ec96458315c661a18b320241 - ishan - 2017-05-30 07:32:54] at org.janusgraph.diskstorage.solr.SolrIndex.createCollectionIfNotExists(SolrIndex.java:1039) ~[janusgraph-solr-0.2.0-SNAPSHOT.jar:?] at org.janusgraph.diskstorage.solr.SolrIndex.register(SolrIndex.java:301) ~[janusgraph-solr-0.2.0-SNAPSHOT.jar:?] at org.janusgraph.diskstorage.indexing.IndexTransaction.register(IndexTransaction.java:101) ~[janusgraph-core-0.2.0-SNAPSHOT.jar:?] at org.janusgraph.graphdb.database.IndexSerializer.register(IndexSerializer.java:107) ~[janusgraph-core-0.2.0-SNAPSHOT.jar:?] at org.janusgraph.graphdb.database.management.ManagementSystem.addIndexKey(ManagementSystem.java:547) ~[janusgraph-core-0.2.0-SNAPSHOT.jar:?] at org.janusgraph.graphdb.database.management.ManagementSystem$IndexBuilder.buildMixedIndex(ManagementSystem.java:660) ~[janusgraph-core-0.2.0-SNAPSHOT.jar:?] ...

Possible solutions:

• clear all items in collection without dropping collection definition
• least give an option to drop only Janus storage backend only (restore previous behavior possibly under a different method)

[wojciechwojcik]

FYI @sjudeng

[sjudeng]

Are you sure the configset is uploaded to start with? Using drop with Solr Cloud will remove the collection but not the configset.

gremlin> graph = JanusGraphFactory.open('conf/janusgraph-cassandra-solr-cloud.properties')
==>standardjanusgraph[cassandrathrift:[127.0.0.1]]
gremlin> GraphOfTheGodsFactory.load(graph)
==>null
gremlin> JanusGraphFactory.drop(graph)
==>null
gremlin> graph = JanusGraphFactory.open('conf/janusgraph-cassandra-solr-cloud.properties')
==>standardjanusgraph[cassandrathrift:[127.0.0.1]]
gremlin> GraphOfTheGodsFactory.load(graph)
==>null

JanusGraph uses this in Solr unit tests, where the configset is uploaded only once at the start of the test suite (see here).

[wojciechwojcik]

Configset is uploaded but named search, not collectionName.

[sjudeng]

I think it needs to be uploaded under collectionName per docs, though this only needs to be done once during test suite setup. But does it work if you try to create the index before dropping? Or are you're using index.search.solr.configset=search (it's not shown in the above config).

[wojciechwojcik]

No I am not using index.search.solr.configset=search, and I know have to do it manually then.
My point is, we should not automatically drop a collection if by design we are later unable to automatically re-create it.

[sjudeng]

But doesn't it recreate it? The example above creates vertices and edges collections (see here), drops them and then recreates them without error. In the case here if you upload the configset under collectionName then are you still unable to drop and recreate?

[pluradj]

The configset collectionName would still be in Zookeeper under /configs/collectionName after the call to JanusGraphFactory.drop(graph). @wojciechwojcik How are you creating the collection manually in your test case? What is your Solr version?

[wojciechwojcik]

Platform: Windows
docker 17.06.0-ce
Solr 6.6.1 (dockerized)
Zookeeper 3.4.10 (dockerized)
I create the collection like this:

````
docker cp conf solr:/opt/solr/server/solr/configsets/search

COLLECTION_NAME="collectionName"
CONF_NAME="search"

echo "Creating collection: $COLLECTION_NAME using configset: $CONF_NAME"
docker exec -i --user=solr solr solr create_collection -c "$COLLECTION_NAME" -shards 2 -replicationFactor 1 -confname "$CONF_NAME" -confdir "$CONF_NAME" -p 8983

echo "Uploading config to zookeeper"
docker exec -i --user=solr solr solr zk -upconfig -z zookeeper:2181 -n "$CONF_NAME" -d "/opt/solr/server/solr/configsets/$CONF_NAME"
````

I guess it may have to do with issue #315 reported earlier.

[sjudeng]

We do something similar to run tests against an external Solr server. See TESTING.md and janusgraph-solr/src/test/resources/docker-compose.yml. But the configsets are uploaded once (at container start) and then the full test suite runs, which creates/drops collections without issue.

[wojciechwojcik]

Ok. Make sure you do NOT use index.search.solr.configset in your testing.

[sjudeng]

Is it an issue? You mentioned above you weren't using index.search.solr.configset here. I'm not sure if this parameter is covered in tests so that might narrow the issue down if you are using it.

[wojciechwojcik]

For me the the param index.search.solr.configset is broken and I do not use it.
When set all your index data instead of going to collection named after your indexname goes to collection from configset param, which is not something I want.

[pluradj]

It sounds like we're dealing with 2 issues here with SolrCloud:

1. If index.search.solr.configset is not specified, then the collection should be cleared but not removed from disk.

2. Figure out why index.search.solr.configset isn't working for @wojciechwojcik because it does work.

Also in your Docker steps above, the call to upconfig is unnecessary since the create_collection command already is uploading the configuration from -confdir into Zookeeper. It seems like it should be -confdir "/opt/solr/server/solr/configsets/$CONF_NAME" though.

[sjudeng]

@pluradj Agreed on (2) but I'd like steps to reproduce the issue with (1) when configset is uploaded for each collection name (e.g. CONF_NAME=COLLECTION_NAME).

[pluradj]

Here's how I reproduced (1 terminal window, 1 Gremlin Console):

// create collection (collectionName) and configset (search)
$SOLR_HOME/bin/solr create_collection -c collectionName -n search -d $JANUSGRAPH_HOME/conf/solr

// verify collection (collectionName) is created
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) is created
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // 1. create graph without configset
    gremlin> graph = JanusGraphFactory.build().set('storage.backend', 'hbase').set('storage.hbase.table', 'woj').set('index.jg.backend', 'solr').set('index.jg.solr.mode', 'cloud').set('index.jg.solr.zookeeper-url', '127.0.0.1:9983').set('index.jg.solr.wait-searcher', true).open()

// verify table is created
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) still exists
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // drop graph
    gremlin> JanusGraphFactory.drop(graph)

// verify table is dropped
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) is dropped
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // 2. create graph without configset again
    gremlin> graph = JanusGraphFactory.build().set('storage.backend', 'hbase').set('storage.hbase.table', 'woj').set('index.jg.backend', 'solr').set('index.jg.solr.mode', 'cloud').set('index.jg.solr.zookeeper-url', '127.0.0.1:9983').set('index.jg.solr.wait-searcher', true).open()

// verify table is created
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) was not created
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // attempt to define the mixed index (exception thrown)
    gremlin> mgmt = graph.openManagement()
    gremlin> name = mgmt.makePropertyKey('name').dataType(String.class).make()
    gremlin> nameIndex = mgmt.buildIndex('collectionName', Vertex.class).addKey(name).buildMixedIndex('jg')

    // drop graph
    gremlin> JanusGraphFactory.drop(graph)

// verify table is dropped
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) does not exist
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // 3. create the graph with configset
    gremlin> graph = JanusGraphFactory.build().set('storage.backend', 'hbase').set('storage.hbase.table', 'woj').set('index.jg.backend', 'solr').set('index.jg.solr.mode', 'cloud').set('index.jg.solr.zookeeper-url', '127.0.0.1:9983').set('index.jg.solr.configset', 'search').set('index.jg.solr.wait-searcher', true).open()

// verify table is created
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) was not created yet
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // attempt to define the mixed index
    gremlin> mgmt = graph.openManagement()
    gremlin> name = mgmt.makePropertyKey('name').dataType(String.class).make()
    gremlin> nameIndex = mgmt.buildIndex('collectionName', Vertex.class).addKey(name).buildMixedIndex('jg')
    gremlin> mgmt.commit()

// verify table is created
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) is created
find $SOLR_HOME/server -name "collectionName*"

// verify a collection named search is not created
find $SOLR_HOME/server -name "search*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

    // drop graph
    gremlin> JanusGraphFactory.drop(graph)

// verify table is dropped
echo list | $HBASE_HOME/bin/hbase shell

// verify collection (collectionName) is dropped
find $SOLR_HOME/server -name "collectionName*"

// verify configset (search) still exists
$SOLR_HOME/bin/solr zk ls -r /configs -z 127.0.0.1:9983

[sjudeng]

@pluradj Thanks. But where is the error/unexpected result?

[pluradj]

section 2 -- // attempt to define the mixed index (exception thrown)
it throws the exception from the original message

I'd question whether this is a valid scenario though. One could argue that the user should create the collection manually again after it is dropped. For example, if the user deletes the collection from the UI or using $SOLR_HOME/bin/solr delete -c collectionname, the local folder is deleted.

It seems that the JUnit test case might have been taking advantage of a bug in the JanusGraph drop logic. We really need to get to the bottom of why index.search.solr.configset isn't working for @wojciechwojcik because using that property is the correct solution.

[sjudeng]

I see. The tests and I have always let JanusGraph handle the creation (and recently dropping) of the collection, in which case the collection name will match the configset name (see SolrIndex#createCollectionIfNotExists). The above error does not occur when the collection name matches the configset name. But if the collection is defined outside of JanusGraph and associated with a configset with a different name, as above and in the docs, then I agree with @wojciechwojcik that JanusGraph shouldn't delete it. I'm okay with restoring the original clear as opposed to drop logic.

[pluradj]

For me the the param index.search.solr.configset is broken and I do not use it.
When set all your index data instead of going to collection named after your indexname goes to collection from configset param, which is not something I want.

@wojciechwojcik I have not seen this behavior before. Do you have more details or reproduce steps on this?

[wojciechwojcik]

@pluradj as sjudeng pointed out the same in his last comment: _let JanusGraph handle the creation (and recently dropping) of the collection, in which case the collection name will match the configset name_
This implies there is always a single collection created for all mixed indexes that janus manages and its name is fixed to configset name. This is not something I want.

I want the collection to be named the same as mixed index name, which currently can only be done with manual solr config.