Micrometer: NaN gauge values not always filtered out

[shakuzen]

1116 intended to fix the issue of sending NaN values to Dynatrace, which is not valid JSON and results in a failure. In trying to use the DynatraceMeterRegistry, I get a failure like the following:

2019-03-29 16:08:14.272 ERROR 61557 --- [trics-publisher] i.m.dynatrace.DynatraceMeterRegistry     : failed to send metrics to dynatrace: {"error":{"code":400,"message":"Could not map JSON at 'series[22].dataPoints' near line 1 column 2524"}}

Debugging and looking at the request payload, I can see that NaN is trying to be sent in a dataPoint like:

{
    "timeseriesId": "custom:system.cpu.usage",
    "dataPoints": [
        [
            1553842062787,NaN
        ]
    ]
}

Looking at the code, it seems the issue is that some gauges can return a finite value one moment and NaN the next.

https://github.com/micrometer-metrics/micrometer/blob/6c657f93479b03835478153c33cde4122b7b7ac2/implementations/micrometer-registry-dynatrace/src/main/java/io/micrometer/dynatrace/DynatraceMeterRegistry.java#L148-L150

My theory is that on line 149 ms.getValue() is returning a finite number and on line 150 ms.getValue() is subsequently returning NaN.

[izeye]

@shakuzen Oops, sorry. If you didn't start working on it yet, I can work on it although its fix seems obvious already thanks to your analysis.

[shakuzen]

@izeye feel free to send a pull request. I've got plenty of other things to work on :)

[izeye]

@shakuzen Thanks for the prompt response! I created https://github.com/micrometer-metrics/micrometer/pull/1345 to resolve this.

[shakuzen]

It is interesting to note that com.sun.management.OperatingSystemMXBean#getSystemCpuLoad is sometimes returning NaN which its JavaDoc doesn't indicate it should ever do. JavaDoc states it should return a negative value if unavailable. This could be a non-issue if we replace this implementation with OSHI, but maybe we want to do something about it in the maintenance branches anyways.

[izeye]

@shakuzen Thanks for the more information! I think it needs to be fixed regardless of it as there could be any custom meter having this behavior.

[shakuzen]

I think it needs to be fixed regardless of it as there could be any custom meter having this behavior.

I agree. I mean that in addition to this we may want to look into the unexpected behavior more since it could affect other registries.