Thehive: Support Elasticsearch 6.x clusters

[rhaist]

Support the current Elasticsearch 6.x stack/clusters

Request Type

Bug

Work Environment

| Question | Answer
|---------------------------|--------------------
| OS version (server) | Debian
| OS version (client) | Stretch (stable)
| TheHive version / git hash | 3.x
| Package Type | DEB

Problem Description

During a test installation of the hive with our Elastic 6.x cluster the migration failed at version 13.

Steps to Reproduce

1. Install thehive from the official repo on debian stretch
2. Point to an external ES6 cluster
3. Read logs.

Possible Solutions

The following issue upstream at Elastic might give further hints: https://discuss.elastic.co/t/unable-to-create-index-with-more-that-1-type-in-6-x/106089

Complementary information

Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Create a new empty database
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 2
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 3
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 4
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 5
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 6
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 7
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 8
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 9
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 10
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 11
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 12
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.s.MigrationSrv - Migrate database from version 0, add operations for version 13
Jun 27 10:04:03 hive authbind[16095]: [error] o.e.s.MigrationSrv - Migration fail
Jun 27 10:04:03 hive authbind[16095]: org.elasticsearch.transport.RemoteTransportException: [castleblack][10.0.0.1:9300][indices:admin/create]
Jun 27 10:04:03 hive authbind[16095]: Caused by: org.elasticsearch.transport.RemoteTransportException: [shadowtower][10.0.0.2:9300][indices:admin/create]
Jun 27 10:04:03 hive authbind[16095]: Caused by: java.lang.IllegalArgumentException: Rejecting mapping update to [the_hive_13] as the final mapping would have more than 1 type: [dblist, data, case_artifact_job, caseTemplate, case_task, reportTemplate, case_task_log, alert, audit, case_artifact, user, case, dashboard]
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:408)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:280)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:443)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132)
Jun 27 10:04:03 hive authbind[16095]:         at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
Jun 27 10:04:03 hive authbind[16095]: [info] o.e.ErrorHandler - POST /api/maintenance/migrate returned 400

[saadkadhi]

https://github.com/TheHive-Project/TheHiveDocs/blob/master/FAQ.md#do-you-support-elasticsearch-6x-or-later

[romans8]

Any movement on this? Elasticsearch doesn't do anything with 5.x anymore - it's only in critical bugfix mode

[romans8]

I know it was stated thing are moving to graphDB a couple months back in the next major release 4.0.

ES7 just launched and Kubernetes seem to like 6+.

How can I help things move forward and contribute? What's the best way?

[saadkadhi]

We found out indeed and very recently that ES 5.6 is dead. We have published a blog post that I invite you to read at: https://blog.thehive-project.org/2019/05/06/an-apology/.

TL;DR we are currently working on having a supported ES version in TheHive & Cortex. We will come up with a concrete action plan in the upcoming days.