Micronaut-core: Issue with Consul registered service with unresolvable hostname

[radpet]

Task List

• [x] Steps to reproduce provided
• [x] Stacktrace (if present) provided
• [ ] Example that reproduces the problem uploaded to Github
• [x] Full description of the issue provided (see below)

The following use case may cause all services implemented using micronaut connected to consul to be marked as down even if they are up and running.

Micronaut uses https://github.com/micronaut-projects/micronaut-core/blob/e54f6e6e3077ce24815bdebf9ffe95daf530fc51/discovery-client/src/main/java/io/micronaut/discovery/consul/client/v1/HealthEntry.java to deserialize the responses from consul

InetAddress is used in the jackson deserialization but since the hostname is not an ip and it is not resolvable from the current context the following exception is throw:

io.micronaut.http.codec.CodecException: Error decoding JSON stream for type [interface java.util.List]: simpleapp-6dfdb49cd7-8p9z8: Name or service not known (through reference chain: java.util.ArrayList[0])2019-08-03 09:20:0906:20:09.470 [nioEventLoopGroup-3-2] DEBUG i.m.http.client.DefaultHttpClient - Unable to decode response body using codec JsonMediaTypeCodec:Error decoding JSON stream for type [interface java.util.List]: simpleapp-6dfdb49cd7-8p9z8: Name or service not known (through reference chain: java.util.ArrayList[0])

Thus since the exception is not caught the health endpoint is marked as down.

Steps to Reproduce

Let's have 2 services A and B

1. In service A use locally resolvable dns address as hostname (If the hostname is not resolvable at the beginning micronaut fast-fails which is great)
2. Have another service B which cannot resolve the hostname

Expected Behaviour

Since the services do not have any interactions between, B's service status should not be marked as down.

Actual Behaviour

B overall service status is marked as down even though there are no interactions between the service. Furthermore a single dns error is able to bring the whole cluster down.

Environment Information

• Operating System:
• Micronaut Version: 1.1.2/1.1.3
• JDK Version: 11

Example Application

[radpet]

As a workaround one may disable few out of the box functionalities:
consul.client.discovery.enabled: false
endpoints.health.discovery-client.enabled: false

[graemerocher]

In scenarios where the host name cannot be resolved you should use the prefer-ip-address setting:

https://docs.micronaut.io/latest/guide/configurationreference.html#io.micronaut.discovery.consul.ConsulConfiguration$ConsulRegistrationConfiguration

[radpet]

Even if prefer-ip-address is used it does not prevent the cascade failure in case of dns resolution.
Should prefer-ip-address property be used only with valid ip values because currently you can set a non/resolvable hostname?

[graemerocher]

In this scenario you should replace DiscoveryClientHealthIndicator with an implementation that either ignores any exceptions or just returns ok

@Replaces(DiscoveryClientHealthIndicator.class)
class MyIndicator implements HealthIndicator {
    public Publisher<HealthResult> getResult() {
         return Flowable.just(
             HealthResult.builder("discovery-client", HealthStatus.UP).build()
         );
    } 
}

[graemerocher]

If you wish to supply a PR with configurable exception handling logic then the code that handles this can be found here:

https://github.com/micronaut-projects/micronaut-core/blob/master/management/src/main/java/io/micronaut/management/health/indicator/discovery/DiscoveryClientHealthIndicator.java#L61