Micrometer: Spring Boot 2.2.0 with Micrometer 1.3.0 for ElasticSearch 7.1 not working

[nemanjanedic]

I'm trying to get the basic metrics into the ElasticSearch index on AWS. I'm using Spring Boot 2.2.0 with WebFlux. For Micrometer I'm using Elastic Registry 1.3.0. After starting the app, I'm getting these errors on the app side:

2019-11-04 10:56:24.495 ERROR 1510 --- [trics-publisher] i.m.elastic.ElasticMeterRegistry : failed to send metrics to elastic (sent 55 metrics but created 0 metrics):
{
    "took": 0,
    "errors": true,
    "items": [
        {
            "index": {
                "_index": "",
                "_type": "metrics-2019-11",
                "_id": null,
                "status": 500,
                "error": {
                    "type": "string_index_out_of_bounds_exception",
                    "reason": "String index out of range: 0"
                }
            }
        }, ...

And on the ElasticSearch side, the error is this:

[2019-11-04T02:48:01,553][WARN ][r.suppressed ] [76ba019e12d26993bb1160a3b9e7e9b0] path: __PATH__ params: {index=, id=metrics_template, type=_template}
java.lang.StringIndexOutOfBoundsException: String index out of range: 0
at java.lang.String.charAt(String.java:658) ~[?:1.8.0_172]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexOrAliasName(MetaDataCreateIndexService.java:172) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:144) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:608) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:286) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:690) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

I did a basic endpoint configuration and that is all. I also tried setting the index name into the application.yml but it did not help.

Any ideas on how to overcome this issue?

[izeye]

@nemanjanedic Thanks for the report!

I tried to reproduce this with a sample and a standalone Elasticsearch 7.1.1, but I couldn't reproduce it. I'm not sure if it's something specific to AWS.

Could you please try it with a standalone Elasticsearch?

[nemanjanedic]

I'll try but it would not help much if it cannot work with AWS ES.

[izeye]

@nemanjanedic Thanks for the quick feedback! I just wanted to be sure if it's something specific to AWS or not as mentioned above. It'd be even better if you could provide a complete and minimal sample which reproduces the problem you're seeing.

[nemanjanedic]

Sorry for not able to answer before. I did manage to make this working with the latest standalone version of ElasticSearch (7.4.2) with the Bitnami AWS image. If I have time in the upcoming weeks, I'll create a sample app with the AWS managed ELK stack.

[kvaidas]

I might have had this issue recently which I think I solved by removing a trailing forward slash in management.metrics.export.elastic.host.

[shakuzen]

I might have had this issue recently which I think I solved by removing a trailing forward slash in management.metrics.export.elastic.host.

Thanks for the info @kvaidas. Would you be able to confirm if that helps any for you @nemanjanedic?

[nemanjanedic]

I am sorry, but I do not have time at the moment to check that as I moved to a different version of ELK stack (unmanaged).

[shakuzen]

I am sorry, but I do not have time at the moment to check that as I moved to a different version of ELK stack (unmanaged).

No problem. Thank you for the quick response. We'll close this until someone else can verify there's an issue we need to fix here. We can reopen the issue if that happens. Thanks.