Micrometer: Check NaN and infinity values validity across all meter registries

Created on 5 Jan 2019  路  6Comments  路  Source: micrometer-metrics/micrometer

See https://github.com/micrometer-metrics/micrometer/pull/1116#discussion_r245373370

Meter registry checklist

  • [x] AppOptics: #1109
  • [ ] Atlas: N/A
  • [x] CloudWatch: #1138
  • [ ] Datadog: N/A
  • [x] Dynatrace: #1116
  • [x] Elasticsearch: #1135
  • [ ] Ganglia: N/A
  • [ ] Graphite: N/A
  • [x] Humio: #1174
  • [x] InfluxDB: #1138
  • [ ] JMX: N/A
  • [x] KairosDB: #1138
  • [x] New Relic Insights: #1150
  • [ ] Prometheus: N/A
  • [ ] SignalFx: N/A
  • [ ] Stackdriver: N/A
  • [x] StatsD: #778
  • [x] Wavefront: #778
picture izeye
👍1

All 6 comments

We need to handle it correctly everywhere, but some monitoring systems do indeed handle NaN and Infinity.

jkschneider picture jkschneider on 5 Jan 2019
👍1

@izeye is there still more work to do for this issue?

shakuzen picture shakuzen on 22 Jan 2019

@shakuzen I created one more PR (https://github.com/micrometer-metrics/micrometer/pull/1174) for this but didn't check all yet 馃槄

izeye picture izeye on 22 Jan 2019

I added a checklist for meter registries and checked all the items which already handle NaN and infinities for gauges, time gauges and function counters.

Any help for unchecked items would be appreciated.

izeye picture izeye on 22 Jan 2019
👍1

For Elastic I am still getting errors:

[2020-03-23T11:22:19,308][DEBUG][o.e.a.b.TransportShardBulkAction] [AUDIT] [metrics-2020-03][0] failed to execute bulk item (index) index {[metrics-2020-03][_doc][wumzBnEBbzVK4unYMuEs], source[{"@timestamp":"2020-03-23T09:22:19.300Z","name":"tomcat_servlet_request","type":"timer","host":"SERVER1","name_tag":"default","count":0.0,"sum" :NaN,"mean":0.0}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
    at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:191) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:74) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:202) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:114) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:81) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:895) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:109) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.runWithPrimaryShardReference(TransportReplicationAction.java:374) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$doRun$0(TransportReplicationAction.java:297) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$24(IndexShard.java:2791) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:113) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:285) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2765) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryOperationPermit(TransportReplicationAction.java:836) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:293) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.handlePrimaryRequest(TransportReplicationAction.java:256) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) [x-pack-security-7.6.1.jar:7.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:315) [x-pack-security-7.6.1.jar:7.6.1]
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:750) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.1.jar:7.6.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Non-standard token 'NaN': enable JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS to allow
 at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@17c75ca3; line: 1, column: 258]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2667) ~[jackson-core-2.8.11.jar:2.8.11]
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:832) ~[jackson-core-2.8.11.jar:2.8.11]
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52) ~[elasticsearch-x-content-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:429) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112) ~[elasticsearch-7.6.1.jar:7.6.1]
    at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71) ~[elasticsearch-7.6.1.jar:7.6.1]
ramunask picture ramunask on 23 Mar 2020

@ramunask Thanks for the report!

Based on the info, FunctionTimer.totalTime() seems to have returned NaN somehow. I created #1932 to resolve it.

izeye picture izeye on 23 Mar 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

wilkinsona picture wilkinsona  路  3Comments
ITman1 picture ITman1  路  4Comments
adrianboimvaser picture adrianboimvaser  路  3Comments
alecharp picture alecharp  路  4Comments
wilkinsona picture wilkinsona  路  5Comments