Micrometer: connection time out in sending metrics to datadog

[pchatsu]

We use io.micrometer:micrometer-core:1.1.0 with spring boot actuator.
We have sometimes failed to send metrics to datadog during running application.

[Log]
message: failed to send metrics to datadog
module: i.m.d.DatadogMeterRegistry
line: 146
exception: java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)

Any ideas?

[izeye]

@pchatsu Thanks for the report but it seems to be "connect timed out" from JDK, so there’s nothing Micrometer can do for it. You can tweak connect timeout configuration to fit your environment if it helps.

[shakuzen]

It sounds like your application may not be able to connect quickly to the Datadog API. The connectTimeout can be configured with the HttpSender used with the DatadogMeterRegistry. For now, it can also be configured with the connectTimeout method on the Config class - it defaults to 1 second.

[pchatsu]

@izeye @shakuzen thanks for your advice! I'll try to configure connectTimeout.

[fkoehler]

We had this issue very often Increasing connectTimeout to 5 seconds helped in all cases

[shakuzen]

thanks for your advice! I'll try to configure connectTimeout.

Let us know if that worked for you. I'll close this for now and we can reopen it if there are some changes necessary in Micrometer.