Elasticsearch: [ML] Data Frame Analytics fails when destination index exists but has no mappings

[movelg]

Elasticsearch version: 7.8.0
Description of the problem including expected versus actual behavior:
Data frame analytics jobs fails when the destination index exists. But the documentation says:
"If the destination index exists, it is used as is. You can therefore set up the destination index in advance with custom settings and mappings."

Steps to reproduce:
Run in Dev Tools Console

POST test-source-index-a2/_doc
{"value1": 1, "target": 1}

POST test-source-index-a2/_doc
{"value1": 2, "target": 4}

POST test-source-index-a2/_doc
{"value1": 3, "target": 9}

PUT test-dest-index-a2

PUT _ml/data_frame/analytics/test-dfa-job-a2
{
"description": "",
"source": {
"index": "test-source-index-a2"
},
"dest": {
"index": "test-dest-index-a2"
},
"analyzed_fields": {
"excludes": []
},
"analysis": {
"regression": {
"dependent_variable": "target"
}
},
"model_memory_limit": "20mb"
}

POST _ml/data_frame/analytics/test-dfa-job-a2/_start

-> Kibana Machine Learning / Data Frame Analytics shows status "failed"

Provide logs (if relevant):

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[dimitris-athanasiou]

Could you please take a look in the job messages in kibana for that job? It should have the reason it failed. I have reproduced the steps you included and the job finished successfully for me.

[movelg]

Kibana does not show any reason for the failure. I guess the error is because of the supposed lack of training documents. Kibana shows: {"training_docs_count":0,"test_docs_count":0,"skipped_docs_count":0}. The odd thing is that training_docs_count is zero only if the destination index exists.

[dimitris-athanasiou]

Could you paste the result of the following request please?

GET _ml/data_frame/analytics/test-dfa-job-a2/_stats

[movelg]

Results for GET _ml/data_frame/analytics/test-dfa-job-a2/_stats:

{
"count" : 1,
"data_frame_analytics" : [
{
"id" : "test-dfa-job-a2",
"state" : "failed",
"progress" : [
{
"phase" : "reindexing",
"progress_percent" : 1
},
{
"phase" : "loading_data",
"progress_percent" : 0
},
{
"phase" : "feature_selection",
"progress_percent" : 0
},
{
"phase" : "coarse_parameter_search",
"progress_percent" : 0
},
{
"phase" : "fine_tuning_parameters",
"progress_percent" : 0
},
{
"phase" : "final_training",
"progress_percent" : 0
},
{
"phase" : "writing_results",
"progress_percent" : 0
}
],
"data_counts" : {
"training_docs_count" : 0,
"test_docs_count" : 0,
"skipped_docs_count" : 0
},
"memory_usage" : {
"peak_usage_bytes" : 0
},
"node" : {
"id" : "V_vMRLVNRNe8MXwCgT2TRA",
"name" : "OpenSOCElastic",
"ephemeral_id" : "rCXEgwUrQLGh8o5UMxnOOw",
"transport_address" : "10.0.0.1:9300",
"attributes" : {
"ml.machine_memory" : "33650077696",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "20"
}
},
"assignment_explanation" : ""
}
]
}

[dimitris-athanasiou]

OK, there is no reason in that response either.

The next step to understand what is happening here is to look into the log. When the job failed there should be an error message in the log.

[movelg]

These should be the relevant lines from elasticsearch.log:

[2020-09-02T13:43:12,435][INFO ][o.e.c.m.MetadataCreateIndexService] [Elastic1] [test-source-index-a2] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
[2020-09-02T13:43:12,663][INFO ][o.e.c.m.MetadataMappingService] [Elastic1] [test-source-index-a2/YOp2ChX1QDGeN0MY8ZXeMQ] create_mapping [_doc]
[2020-09-02T13:43:17,326][INFO ][o.e.c.m.MetadataCreateIndexService] [Elastic1] [test-dest-index-a2] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-09-02T13:43:24,910][INFO ][o.e.x.m.a.TransportStartDataFrameAnalyticsAction] [Elastic1] [test-dfa-job-a2] Starting data frame analytics from state [null]
[2020-09-02T13:43:25,139][INFO ][o.e.x.m.d.DataFrameAnalyticsManager] [Elastic1] [test-dfa-job-a2] Using existing destination index [test-dest-index-a2]
[2020-09-02T13:43:25,139][ERROR][o.e.x.m.d.DataFrameAnalyticsTask] [] [test-dfa-job-a2] Setting task to failed
java.util.NoSuchElementException: null
at com.carrotsearch.hppc.AbstractIterator.next(AbstractIterator.java:41) ~[hppc-0.8.1.jar:?]
at org.elasticsearch.common.collect.ImmutableOpenMap$1.next(ImmutableOpenMap.java:135) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.xpack.ml.dataframe.DestinationIndex.updateMappingsToDestIndex(DestinationIndex.java:202) ~[?:?]
at org.elasticsearch.xpack.ml.dataframe.DataFrameAnalyticsManager.lambda$reindexDataframeAndStartAnalysis$12(DataFrameAnalyticsManager.java:275) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:89) [elasticsearch-7.8.0.jar:7.8.0]
[...]
[2020-09-02T13:43:25,259][INFO ][o.e.x.m.d.DataFrameAnalyticsTask] [Elastic1] [test-dfa-job-a2] Updated analytics task state to [failed] with reason [null]

[dimitris-athanasiou]

Thank you so much for providing the log lines so quickly!

I have now reproduced the problem. The code expects that the destination index has at least some mappings in 7.x versions. This is not an issue on master. I will work on a fix.

[dimitris-athanasiou]

@movelg As a workaround, you can add mappings to the destination index before you start the job.

In fact, even if this bug is fixed, the idea behind allowing pre-existing destination indices is so that the user can customize the mappings. We do not copy missing mappings with those we see in the source index when that is the case. Effectively, it is the responsibility of the user to provide mappings for the fields that exist in the job's source index.

[movelg]

Thanks for the workaround. For my use case I had to transform the ML results using a default pipeline. I don't need custom mappings.

[dimitris-athanasiou]

If you don't need custom mappings, then you can also start the job with a destination index that does not exist.