Micrometer: DatadogMeterRegistry StackOverflowError

[jens-meiss]

Hi,

we receive endless Stackoverflow exceptions if the datadog endpoint is not available anymore for any reason or they have to restart the servers or something:

The message in DD lookes like this:

{"timestamp":"2019-03-18T23:47:21.187+00:00","message":"failed to send metrics to datadog","logger":"i.m.datadog.DatadogMeterRegistry","thread_name":"datadog-metrics-publisher","status":"WARN","stack_trace":"java.lang.StackOverflowError: null\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044)\n\tat java.base/java.util.Collections$UnmodifiableCollection$1.hasN

I don't see any logs other than this, no warning that metrics couldn't be sent or any other exception.

If i look into DatadogMeterRegistry i just see only one line where "hasNext()" is used, maybe you can find the loop in your code.

We'r using "micrometer-registry-datadog-1.1.2.jar" which is shipped with spring-boot 2.1

[izeye]

Based on the provided stack trace, its wrapped collection's iterator seems to be its iterator itself. I'm not sure how it could happen at the moment.

@jens-meiss Could you check if this happens with a simple sample?

[jens-meiss]

@izeye yes of course, but currently i don't know how to reproduce it. We had this issue only two times in the past 4 months, so its verry rare. But it happens on all services at exact the same time (that mean it couldn't be a problem in our source code and from my point of view must be somewhere in the library).

Can you tell me what example you want to have? thx :)

[izeye]

@jens-meiss Thanks for the quick feedback! I expected it's reproducible with your specific configuration and a minimal sample to reproduce it would help if possible.

[jens-meiss]

if i find the time i'll create an example e.g. with a mocked datadog endpoint and a simple service which sends custom metrics to this endpoint and the just killing the endpoint and see what happens if this enpoint can't be reached anymore ... cause it lookes like something like this, the DD dns entry has been changed or the endpoint is not available for any reason ...

the strange thing is that we have to restart all services to get the custom metrics working again ... so it's not self healing if the endpoint is back online....

we run our services with "java -Dnetworkaddress.cache.ttl=60" to solve the infinite dns caching, but it seems in this case it's not a problem to resovle the - maybe - new ip after a restart on DD site ...

[izeye]

One way to trigger StackOverflowError in Iterator.hasNext() of Collections.unmodifiableCollection() is repeatedly wrapping List with Collections.unmodifiableList() although I couldn't find any possibility in Micrometer. Maybe in other library Micrometer depends on or somewhere else.

I'm not sure there's other way to trigger this.

See https://github.com/izeye/samples-java-branches/commit/c41f8b2098dc8e7399ff8f38bf80e83d078fb2b0

[jens-meiss]

ok, thank you for your analysis, i hoped that it's easy to find the loop, but as always with these kind of errors it's not :) ...

i'll come back if i have a reproduceable example

[olamelin]

Hi,
TL;DR we get the stack overflow every 15 s as well

We are experiencing the exact same thing with the stack overflow
We are using spring-boot 2.1.3.RELEASE with the datadog dependency (micrometer 1.1.3).
The application itself is using spring-boot-starter-webflux as well as spring-cloud-starter-gateway if that can be of relevance.
We have not configured any custom metrics but we have noticed that each time the publish method is supposed to be called (that is each 15 s according to our configuration) we can see the stack overflow error java.base/java.util.Collections$UnmodifiableCollection$1.hasNext(Collections.java:1044) ...

This is our configuration

management:
  metrics:
    export:
      datadog:
        enabled: true
        step: 15s
        uri: https://app.datadoghq.eu
        application-key: changme 
        api-key: changeme

[izeye]

@jens-meiss @olamelin Based on the "java.base" part, both of you seem to use JDK 9+. Could you tell me the exact version of it? I'm not sure it's relevant but it might help to reproduce it.

[olamelin]

We are running the code as a docker image with the base image openjdk:11-jre-slim

[jens-meiss]

same here:

• dockerhub openjdk:11-jre-slim
• https://hub.docker.com/_/openjdk

[shakuzen]

In the next day or two, we're planning to have maintenance releases for 1.0.x and 1.1.x. I'd like to figure this out and resolve it for those releases, but we need to get to root cause on this. I'll try to spend a little time digging in between wrapping up some other issues in time for the releases, but any help would be greatly appreciated.

[olamelin]

I am not sure but it could be related to this one https://bugs.openjdk.java.net/browse/JDK-8214129 which will be fixed in openjdk 11.0.3 which is scheduled for mid April.

[olamelin]

Since my last comment I created a parallel image of our code that was based on a openjdk12 image. In order to see if the JDK issue was the cause.
The one using openjdk12 has not reported the stack overflow the last 6 days but the parallel image (which was restarted) has shown this error multiple times.
In other words I think it is safe to say that this issue is caused by the bug described here https://bugs.openjdk.java.net/browse/JDK-8214129

[shakuzen]

@olamelin thank you for the testing and reporting back. Ouch, though. It doesn't seem there's anything we can do in Micrometer if that's the cause.

[jens-meiss]

ok, we'll wait for the next jdk-11 updates and see if we'll have the problem again, we had the problem only 2 times in the past 4 months, maybe it will take a while until we see it again ...

@olamelin thanks for you investigation
@izeye you can close the ticket if you want to, i'll open a new one if we see the error again after the jdk-11.0.3 release