Janusgraph: Add Support for Elasticsearch 7.x

[mikehowey1]

Feature to support Elasticsearch 7.x

[porunov]

Partially related discussion: https://groups.google.com/forum/#!topic/janusgraph-dev/7BF9Qi7dbAs

[mikehowey1]

Thanks for the info on the discussion @porunov . I posted my thoughts on it also.

[YangSongGit]

My janusgraph version is 0.3.2. Exception occurred when run gremlin-server.sh:

java.lang.IllegalArgumentException: Unsupported Elasticsearch server major version: 7.1.0

Is there any plugin to support elasticsearch 7.x ?

[FlorianHockmann]

Is there any plugin to support elasticsearch 7.x ?

No, there is not. That is why we have this feature request here.

[porunov]

Since ES6 mapping types are deprecated. In ES7 we cannot use mapping types unless we use include_type_name parameter which will result in a deprecation warning. In ES8 this parameter will be removed and mapping types will not be supported.

So, at some point we will need to introduce breaking change which will require drop support for mapping types and re-indexing of all indexes created by earlier versions of JanusGraph.

Currently we can add support for ES7 by adding include_type_name during index creation (temporary solution) or drop support for mapping types right now (breaking changes).

[porunov]

Also, I think adding support for ES7 will most likely provide with some small breaking changes because some parameters were removed like maxRetryTimeout which is currently used by JanusGraph

[mikehowey1]

I would be in favor of dropping support for mapping types right now given ES is already 7.2.
Looks like JanusGraph is just passing the maxRetryTimeout value along to the ES REST client if it was configured. So, if that holds true then I don't see that as being a big deal as it's just a config change for users. Plus, per the documentation on why it was removed it sounds like it wasn't very reliable to begin with... 'The setting was removed as its counting mechanism was not accurate and caused issues while adding little value.'

[porunov]

@mikehowey1 Yes, it is better to drop support for mapping types right now but it isn't a trivial solution and requires some time that is why most likely we will see this issue be resolved in 0.5.0 version.

I tried today to add simple support for ElasticSearch 7 (with include_type_name workaround) in hope to add this feature in 0.4.0 release but wasn't successful. My local project's integration tests worked for ES7 with this update but global travis tests are failing.

If someone knows what causes those errors please share some info or take my changes to add support for ES7.

[porunov]

Does anyone know if there is a list of parameters which can be used (or cannot be used) during an index creation for ES7? For example, in ES7 parameter index.max_result_window cannot be specified during an index creation and this parameter should be updated after the index is created. I need a list of all parameters which can or cannot be used during an index creation.

[porunov]

Looks like I was able to add support for ES7 (without dropping mapping types right now) but I am experiencing the issue described here: #1320 . Have anyone seen this issue before?
Here it the commit: https://github.com/porunov/janusgraph/commit/5ba3ecf405f237235165dca230a0691c62f77aac
Here is the build: https://travis-ci.org/porunov/janusgraph/builds/551903685?utm_source=github_status&utm_medium=notification

Update:
Resolved by updating maven-resources-plugin to version 3.1.0

[porunov]

TODO list after upgrade to ES7:

• [ ] Drop mapping support to prepare for ES8
• [ ] Optimize search queries to not always count total_hits
• [ ] Optimize search queries to not always use Scroll API (for example on queries where only single entity is needed)
• [ ] Research solution to imitate scroll usage by using search_after for queries with sorting by unique parameter / parameters
• [ ] Optimize retrieval of entity fields. Currently only id and score are needed during the retrieval. We should either retrieve only them (without other fields) or make some logic which retrieves necessary fields from index database and then skip making additional request to the storage database to retrieve the same fields (i.e. prevent wasteful operation).