Elasticsearch: [ML] Errors when running nginx module jobs

[pheyos]

Found in version

• 7.0.0-beta1 BC2

Steps to reproduce

• Use the nginx module to create ML jobs

Expected result

• The 5 jobs defined in the module are created without errors

Actual result

• The elasticsearch.log shows a lot of errors of the form:

[DEBUG][o.e.a.b.TransportShardBulkAction] [elastic-ro] [.ml-anomalies-shared][0] failed to execute bulk item (index) index {[.ml-anomalies-.write-dr_remote_ip_request_rate][doc][dr_remote_ip_request_rate_influencer_1489273200000_3600_nginx.access.remote_ip_837166815_39], source[{"job_id":"dr_remote_ip_request_rate","result_type":"influencer","influencer_field_name":"nginx.access.remote_ip","influencer_field_value":"d59e:bcc0:846e:c67e:12fb:c430:f37c:9aec","nginx.access.remote_ip":"d59e:bcc0:846e:c67e:12fb:c430:f37c:9aec","influencer_score":5.767481395370252,"initial_influencer_score":5.767481395370252,"probability":2.1435293947279067E-6,"bucket_span":3600,"is_interim":true,"timestamp":1489273200000}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:177) ~[elasticsearch-7.0.0-beta1.jar:7.0.0-beta1]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:74) ~[elasticsearch-7.0.0-beta1.jar:7.0.0-beta1]

[...]

Caused by: java.lang.ClassCastException: org.elasticsearch.index.mapper.KeywordFieldMapper cannot be cast to org.elasticsearch.index.mapper.ObjectMapper
    at org.elasticsearch.index.mapper.DocumentParser.getDynamicParentMapper(DocumentParser.java:898) ~[elasticsearch-7.0.0-beta1.jar:7.0.0-beta1]

Additional information

• The .ml-anomalies-shared result index does not contain mappings for the job's term field mappings.

[elasticmachine]

Pinging @elastic/ml-core

[pgmpofu]

@pheyos May I work on this issue?

[droberts195]

@pgmpofu have you already worked out where the root cause of this problem is? If not then I don't think it's a good first issue, as it involves quite a complex interaction of the UI, backend and pre-defined modules. I would suggest that you try to find something simpler for a first contribution.

[pgmpofu]

@droberts195 Thank you, I will look for a more beginner-friendly issue.

[droberts195]

@pheyos the nginx module has been removed from Kibana in 7.0. Did you mean the nginx_ecs module?

[droberts195]

On closer inspection I see that it _would_ have been the nginx module that caused this. It doesn't exist in 7.0 now. But I can reproduce the problem running 6.7 Kibana against 7.0 Elasticsearch, so there's something worth investigating here. I think the bug is revealed by:

[2019-03-14T10:59:20,068][INFO ][o.e.x.m.j.p.JobResultsProvider] [Davids-MacBook-Pro-7.local] Index already exists
[2019-03-14T10:59:20,084][INFO ][o.e.x.m.j.p.JobResultsProvider] [Davids-MacBook-Pro-7.local] Index already exists
[2019-03-14T10:59:20,096][INFO ][o.e.x.m.j.p.JobResultsProvider] [Davids-MacBook-Pro-7.local] Index already exists
[2019-03-14T10:59:20,109][INFO ][o.e.x.m.j.p.JobResultsProvider] [Davids-MacBook-Pro-7.local] Index already exists

If the anomalies index doesn't exist then we create it. If it does exist we add extra mappings.

But because 5 jobs are created simultaneously they all think they need to create the index but 4 of them fail due to one of them getting there first. And the bug is that then we don't add the extra mappings for those 4 jobs.

[pheyos]

@droberts195 I think when I noticed this the first time, the nginx module was still available in 7.0 (was removed a bit later).