Micrometer: Metrics not appearing in prometheus endpoint

[marcoschulte]

We have a funny behavior that started appearing with RC8, if we go back to RC7 it disappears: The common Spring metrics do not appear in the Prometheus endpoint like jvm_memory_committed_bytes, system_cpu_count, etc.

Even more funny, the behavior is only happening in one of two of our projects, even though most dependencies are the same - we use Spring Boot 1.5.10, Spring Cloud Edgware.SR2, Micrometer 1.0.2.

The problem seems to lie in the order the auto configurations are applied:

In the project that has no issues, io.micrometer.spring.autoconfigure.MetricsAutoConfiguration#meterRegistryPostProcessor is executed first, then io.micrometer.spring.autoconfigure.export.prometheus.PrometheusMetricsExportAutoConfiguration#prometheusMeterRegistry. Therefore the postprocessor is instantiated first and finally registers all Spring metrics.

In the project that is not working, the order is swapped and the postprocessor does not get invoked with the PrometheusMeterRegistry bean.

This is funny, as PrometheusMetricsExportAutoConfiguration is annotated with @AutoConfigureAfter({MetricsAutoConfiguration.class}).

The documentation on @AutoConfigureAfter is non existent but if I understand the kind of only resource in the internet correctly (https://stackoverflow.com/questions/42284478/autoconfigureafter-not-working-as-desired) it is just used to create the dependency graph but does not necessarily influence the order of creation? In that case, adding an (unused) parameter with type MeterRegistryPostProcessor to the factory method io.micrometer.spring.autoconfigure.export.prometheus.PrometheusMetricsExportAutoConfiguration#prometheusMeterRegistry might not be too elegant but already solve the issue?

Thanks
Marco

[marcoschulte]

Btw. for anyone else having this issue, this fixes it for now:

@Configuration
public class FixPrometheusMetrics {
    public FixPrometheusMetrics(MeterRegistryPostProcessor postProcessor, PrometheusMeterRegistry registry) {
        postProcessor.postProcessAfterInitialization(registry, "");
    }
}

[jkschneider]

This smells like some configuration in your app is forcing early init of one of the metrics auto-configurations which messes up the order somewhat. It can be a little hard to track down, but if you breakpoint where PrometheusMeterRegistry is created in PrometheusMetricsExportAutoConfiguration, you should see something like preInitSingletons (or something like that) in the call stack.

If you see registerBeanPostProcessors anywhere in the call stack you know something bad is probably happening.

[ptahchiev]

Same here. I have the exact same behaviour - for me it is HikariDataSourceMetricsPostProcessor which requires PrometheusMeterRegistry and this is why it is instantiated very early.

[ptahchiev]

I think it's something @snicoll might want to have a look.

[ptahchiev]

Ok, I finally tracked it down. The flow goes like this: MethodSecurityExpressionHandler -needs-> PermissionEvaluator -needs-> AclService -needs-> DataSource -starts-> HikariDataSourceMetricsPostProcessor -needs-> PrometheusMeterRegistry so this registry is created very early.

[marcoschulte]

Sorry for not answering in such a long time, have been on vacation.

Anyways, @jkschneider, you were right on. I do see registerBeanPostProcessors in the call stack. I don't assume that we have some uncommon/weird configuration in our app, it's probably a dependency graph similar to @ptahchiev's. If it helps I could also evaluate this.

[jkschneider]

@ptahchiev Do you have a simple app that you could share that demonstrates the early evaluation starting with MethodSecurityExpressionHandler? That would be a big help.

[ptahchiev]

Here it is: https://github.com/ptahchiev/spring-boot-metrics-problem

[ptahchiev]

TBH I don't think it's your problem. Like I said HikariDataSourceMetricsPostProcessor (spring) needs the PrometheusMeterRegistry.

[golonzovsky]

We had similar issue on our side. In our case we have BeanPostProcessor, which replaces RestTemplate of oauth2 RemoteTokenServices (in order to enable timeouts and zipkin tracing), so all JVM metrics were missing.

In our case we used following hack similar to one from @marcoschulte (as MeterRegistryPostProcessor is package private now)

@Bean
InitializingBean forcePrometheusPostProcessor(BeanPostProcessor meterRegistryPostProcessor, PrometheusMeterRegistry registry) {
    return () -> meterRegistryPostProcessor.postProcessAfterInitialization(registry, "");
}

We hope this would go away in our case with support of server-oauth2 in next spring security, as this hack is quite fragile..

Dependency graph looked like this:
OurBeanPostProcessor -> restTemplateBuilder -> metricsRestTemplateCustomizer -> prometheusMeterRegistry
which caused prometheusMeterRegistry to be created too early when most of MeterBinders are not yet there..

[lukago]

@jkschneider @golonzovsky @izeye
I have similar issue, I have separate maven module for metrics and separate maven module for spring boot app. @golonzovsky solution solved the problem but is there any othre option for that? If i copy metrics module to app module it solves the problem as well but I would like to keep it separated.
I also have database in separate module and no prometheus metrics are presented, healthcheck does not work as well.

[shakuzen]

@lukago this is more of a Spring question than a Micrometer question. Something is causing early initialization, so things need to be configured in a way that don't cause that. A stackoverflow question with a sample project will probably get the help needed.

[Klapsa2503]

Following solution does not work with SpringCloud / Kafka

@Configuration
public class FixPrometheusMetrics {
    public FixPrometheusMetrics(MeterRegistryPostProcessor postProcessor, PrometheusMeterRegistry registry) {
        postProcessor.postProcessAfterInitialization(registry, "");
    }
}

My issue is that logback metrics are not updated after app was initialized. I have investigated this issue and here is what i have discovered:

• BeanPostProcessor:postProcessAfterInitialization adds prometheus turbo filter to logback
• spring-cloud-stream registers new binder and calls SpringApplication:run which in result resets logback configuration which now does not have prometheus turbo filter

My workaround is to invoke meterRegistryPostProcessor at later stage:

@Configuration
@RequiredArgsConstructor
public class ActuatorConfig {

    private final BeanPostProcessor meterRegistryPostProcessor;
    private final PrometheusMeterRegistry registry;

    @EventListener
    public void onApplicationEvent(ApplicationPreparedEvent event) {
        meterRegistryPostProcessor.postProcessAfterInitialization(registry, "");
    }
}

this does not look good, i was trying to discover why spring-cloud-stream calls 1SpringApplication:run1 multiple times but i failed