Micrometer: Export Spring Boot Health Check Information

[tf-martin-maier-moessner]

I want to export health information provided by Spring Boot's health endpoint (/health) to Prometheus. Is there an easy (standard) way to do this with micrometer?

[jkschneider]

Here is a code snippet we refer to every once in a while when this comes up:

@Configuration
class HealthMetricsConfiguration {
    // This should be a field so it doesn't get garbage collected
    private CompositeHealthIndicator healthIndicator;

    public HealthMetricsConfiguration(HealthAggregator healthAggregator,
                                      List<HealthIndicator> healthIndicators,
                                      MeterRegistry registry) {

        healthIndicator = new CompositeHealthIndicator(healthAggregator);

        for (Integer i = 0; i < healthIndicators.size(); i++) {
            healthIndicator.addHealthIndicator(i.toString(), healthIndicators.get(i));
        }

        // presumes there is a common tag applied elsewhere that adds tags for app, etc.
        registry.gauge("health", emptyList(), healthIndicator, health -> {
            Status status = health.health().getStatus();
            switch (status.getCode()) {
                case "UP":
                    return 3;
                case "OUT_OF_SERVICE":
                    return 2;
                case "DOWN":
                    return 1;
                case "UNKNOWN":
                default:
                    return 0;
            }
        });
    }
}

I think what's awkward about providing an OOTB health "gauge" is that we have to map health statuses to an arbitrary set of numbers.

Still seeking a structure that makes sense generally. Any ideas?

[checketts]

In regards to Prometheus 'up' checks 1=up and 0=down. However those numbers aren't useful since I end up running a count against the values anyways. If I were to give a valuation I would recommend:

up=1
unknown=0
down=-1
out_of_service=-2

That way I could filter on servers serving up < 1 to easily find problems.

[jkschneider]

The ordering is even debatable it seems. If I were to take a stab at it, it would be:

up = 1
out_of_service=0
unknown=-1
down=-2

This presumes that new instances that are starting (i.e. in a red/black deployment) will report out_of_service until they finish cache warming, etc at which point they report up. I've always used down to indicate something has gone sideways. But of course, everybody uses the statuses differently.

[checketts]

I'm good with those

[jkschneider]

Added a guide on this with https://github.com/micrometer-metrics/micrometer-docs/commit/ca3cae90daa1986f47f812affc79512dab614b11. Available here.

I can also see registering several gauges for health, one per status and report 0 or 1 depending on whether the status matches. Which model is preferable seems to depend on what you want to do with it.

[muatik]

@jkschneider the code snippet works but only at the application startup stage. So, if one of the health indicators is changed, this will not be reflected in the metrics

[jkschneider]

@muatik Good point. You can add an ApplicationListener on HealthChangedEvent and perform essentially the same logic there. This issue is open to track this.

[mweirauch]

Can't reproduce the "the code snippet works but only at the application startup stage".

https://stackoverflow.com/questions/50625517/get-notified-when-spring-boot-healtcheck-status-changes/50639770#50639770

[bkez322]

I would be really excited to have a consistent OOB solution for health metrics.

Personally, I think the approach of having an independent gauge for each status is the most intuitive by far.

[jkschneider]

We have a doc on it here. I just don't see how we could do it generally.

The range of numbers you map the statuses to are of course arbitrary, as are the ordering of the statuses to some extent. For example, in this case you could set an alert on "health < 2" to capture unknown and down states. Whether unknown is consider "better" or "worse" than down is a matter of preference.

[bkez322]

I completely agree that mapping each possible status to a numeric value is pretty hard to generalize for everybody.

I am advocating for the second method described in that doc, such that you have a metric called health, with a status tag (whose possible tag values include up, down, etc), and where the numeric value is always either 0 (state does not match) or 1 (state does match).

This approach strikes me as being perfectly intuitive. It does not make assumptions of which status is “better” or “worse”. It seems generalized enough to safely apply to anybody’s app... especially if it’s possible to derive any custom health statuses and set those as status tag-values as well. Perhaps I’m overlooking something?

[jkschneider]

@bkez322 So any alert would be based on checking potentially multiple states? I suppose it isn't so bad if there are a limited number of possibilities.

[bkez322]

@jkschneider That is how I would imagine it would work.

Most people (including me) will probably be fine just monitoring for up and ignoring everything else. Which is great- that’s really simple!

But of course, if someone had a desire to explicitly monitor for every possible state, it’s pretty easy to do that too!

[jkschneider]

The Spring Boot team has decided to defer adding this to Spring Boot 2.x until 2.2.

[michaelmcfadyen]

Is there an open issue on the spring boot project for this?

[izeye]

https://github.com/spring-projects/spring-boot/issues/14087 is the only issue for this I can find at the moment and it has been declined. This seems to be cloeable unless things change since the Spring Boot team’s decision.

/cc @philwebb

[checketts]

We could add a snippet to the docs that demonstrates a recommendation. I have a sample I would be happy to use.

[izeye]

@checketts I thought it's already been documented here. Do you have another in mind?

[checketts]

That works great! The only change my code uses is to make the 'bad' statuses negatives:

UP: 1
DOWN: -1
OUT_OF_SERVICE: -2
Default: 0

That simplifies my charting in Prometheus

[barlog-m]

Updated variant for Spring Boot 2.2.x

@Configuration
public class HealthMetricConfig {
    @Bean
    MeterRegistryCustomizer<MeterRegistry> healthRegistryCustomizer(HealthContributorRegistry healthRegistry) {
        return registry -> registry.gauge("health", emptyList(), healthRegistry, health -> {
            var status = aggregatedStatus(health);
            if ("UP".equals(status.getCode())) {
                return 1;
            }
            return 0;
        });
    }

    private static Status aggregatedStatus(HealthContributorRegistry health) {
        var healthList = health.stream()
            .map(r -> ((HealthIndicator) r.getContributor()).getHealth(false).getStatus())
            .collect(Collectors.toSet());
        var statusAggregator = new SimpleStatusAggregator();
        return statusAggregator.getAggregateStatus(healthList);
    }
}

[dlennon-ie]

For us old Java 8 folks...
Also I have a CompositeHealthContributor

@Configuration
public class HealthMetricConfig {
    @Bean
    MeterRegistryCustomizer<MeterRegistry> healthRegistryCustomizer(HealthContributorRegistry healthRegistry) {
        return registry -> registry.gauge("health", Collections.emptyList(), healthRegistry, health -> {
            Status status = aggregatedStatus(health);
            if ("UP".equals(status.getCode())) {
                return 1;
            }
            return 0;
        });
    }

    private static Status aggregatedStatus(HealthContributorRegistry health) {
        Set<Status> healthList = new HashSet<>();
        healthContributorIterator(healthList, health.iterator());

        StatusAggregator statusAggregator = new SimpleStatusAggregator();
        return statusAggregator.getAggregateStatus(healthList);
    }

    private static void healthContributorIterator(Set<Status> healthList, Iterator<NamedContributor<HealthContributor>> iter) {
        while (iter.hasNext()) {
            HealthContributor contrib = iter.next().getContributor();
            if (CompositeHealthContributor.class.isAssignableFrom(contrib.getClass())) {
                CompositeHealthContributor healthContributor = (CompositeHealthContributor) contrib;
                healthContributorIterator(healthList, healthContributor.iterator());
            } else {
                HealthIndicator healthContributor = (HealthIndicator) contrib;
                Status status = healthContributor.getHealth(false).getStatus();
                healthList.add(status);
            }  
        }
    }
}

[renanreismartins]

@jkschneider since it will not become part of spring, should we update the doc and close it?
I could gladly do that when we decide on a final solution

[shakuzen]

@renanreismartins thank you for the reminder on this. See https://github.com/micrometer-metrics/micrometer-docs/pull/100 for the resolution on this. Closing as there's no action to take in this repository.

[jensim]

Sorry for kicking a dead issue.. But google sent me here, and CompositeHealthIndicator used in the example is now deprecated - and suggesting I should use the CompositeHealthContributor instead.

Would this solution be a working substitute @shakuzen @jkschneider ?
Sorry for going from Java to Kotlin, if that syntax is forrein to you..

import io.micrometer.core.instrument.MeterRegistry
import org.springframework.boot.actuate.health.HealthIndicator
import org.springframework.boot.actuate.health.Status
import org.springframework.context.annotation.Configuration
import java.util.function.ToDoubleFunction

@Configuration
internal class HealthMetricsConfiguration(
        private val healthIndicators: List<HealthIndicator>,
        registry: MeterRegistry
) {

    init {
        registry.gauge("composite_health", Unit, ToDoubleFunction {
            val allUp = healthIndicators.all { it.health().status == Status.UP }
            return@ToDoubleFunction if (allUp) {
                0.0
            } else {
                1.0
            }
        })
    }
}

[snicoll]

@jensim exposing health check as a metric is documented in the Spring Boot Reference Guide.

[jensim]

Thanks @snicoll! In case anyone else than I end up here ill just paste the config i ended up using implementing the recommendation from your like to spring boot reference guide.

import io.micrometer.core.instrument.Gauge
import io.micrometer.core.instrument.MeterRegistry
import org.springframework.boot.actuate.health.HealthEndpoint
import org.springframework.boot.actuate.health.Status
import org.springframework.context.annotation.Configuration

@Configuration
internal class HealthMetricsConfiguration(
        healthEndpoint: HealthEndpoint,
        registry: MeterRegistry
) {

    init {
        Gauge.builder("composite_health", healthEndpoint, {
            when (it.health().status) {
                Status.UP -> 0.0
                else -> 1.0
            }
        }).register(registry)
    }
}

[chinky1]

Hi All,

I want to import details also and update details with status change but its not working.

Code to import Health with details in prometheus is below in this my status is updating when its Down but details are not

`

   public MeterRegistryCustomizer<MeterRegistry> healthRegistryCustomizer(HealthContributorRegistry healthRegistry) {

   return registry -> healthRegistry.stream()
                          .forEach(namedContributor -> registry.gauge("health", 
                           Details(namedContributor), 
                           healthRegistry, health -> {
   var status = ((HealthIndicator) health
                       .getContributor(namedContributor.getName()))
                       .getHealth(true)
                       .getStatus();
                      return healthToCode(status);
  }));}

private static  Iterable<Tag> Details(NamedContributor<HealthContributor>conr) {
HealthContributor contrib = conr.getContributor();
HealthIndicator healthContributor = (HealthIndicator) contrib;
String name = conr.getName();
tagList.add(Tag.of("name", name));    
String details = healthContributor.getHealth(true).getDetails().toString();
tagList.add(Tag.of("details", details));
return tagList;  
 }

public static int healthToCode(Status status) {
 return status.equals(Status.UP) ? 1 : 0;
}

`

I am also having one Custom health for which I want to Import and update health details in prometheus

below is code for CustomHealth
public Health health() { if (!isRunning) { return Health.down().withDetail(message, "failed").withDetail("Id",Id).build(); } return Health.up().withDetail(message, "Running").build(); }

To explain more in Example

while health status is up on Load of application (Its Working fine)

"/actuator/health" endpoint
"customHealth":{"status":"UP","details":{"job":"Running"}}

"/actuator/prometheus" endpoint
component_health{details="{job=Running}",name="customHealth",} 1.0

**at some condition custom Health status is down its status is updating but details is not updating at prometheus end point it remains old one.

"/actuator/health" endpoint
"customHealth":{"status":"DOWN","details":{"job":"failed","jobIds":"abc"}}

"/actuator/prometheus" endpoint
component_health{details="{job=Running}",name="customHealth",} 0.0**

[checketts]

@chinky1 Could you please open a new ticket (if this is a issue) or ask this same question on Stack Overflow (if you are trying to understand usage)?

[formatq]

@chinky1 hey, take a look on this example https://gist.github.com/formatq/b791cdc2def2c91dfbad08dc82fb1170

# HELP health_status HealthCheck result in prometheus's response format
# TYPE health_status gauge
health_status{application="java-service",type="main",} 0.0
health_status{application="java-service",type="db",database="PostgreSQL",validationQuery="isValid()",} 1.0
health_status{application="java-service",type="diskSpace",total="506332180480",exists="true",threshold="10485760",free="412188921856",} 1.0
health_status{application="java-service",type="ping",} 1.0