Hikaricp: Micrometer gauges not visible in prometheus endpoint

[marsom]

Environment

HikariCP version: 2.7.4
JDK version     : 1.8.0_144
Database        : Oracle
Driver version  : 12.2.0.1.0

Problem

In my prometheus endpoint I do not see any gauge defined in MicrometerMetricsTracker.java (i.e. total, idle, active, pending gauge).

however if i remove all class gauge properties and replace gauge registration with the code below it works (I copied the MicrometerMetricsTracker and made my changes, and registred my tracker instead of the one included in this lib).

I'd tried to debug it a bit, but I did not see why the current code in hikaricp does not work for me. it looks perfeclty valid for me.

Feel free to close this ticket, a soon micrometer is not in rc-mode, I'will test this again.

MicrometerMetricsTracker.java

      this.totalConnectionGauge = Gauge.builder(METRIC_NAME_TOTAL_CONNECTIONS, poolStats, PoolStats::getTotalConnections)
         .description("Total connections")
         .tags(METRIC_CATEGORY, poolName)
         .register(meterRegistry);

      this.idleConnectionGauge = Gauge.builder(METRIC_NAME_IDLE_CONNECTIONS, poolStats, PoolStats::getIdleConnections)
         .description("Idle connections")
         .tags(METRIC_CATEGORY, poolName)
         .register(meterRegistry);

      this.activeConnectionGauge = Gauge.builder(METRIC_NAME_ACTIVE_CONNECTIONS, poolStats, PoolStats::getActiveConnections)
         .description("Active connections")
         .tags(METRIC_CATEGORY, poolName)
         .register(meterRegistry);

      this.pendingConnectionGauge = Gauge.builder(METRIC_NAME_PENDING_CONNECTIONS, poolStats, PoolStats::getPendingThreads)
         .description("Pending threads")
         .tags(METRIC_CATEGORY, poolName)
         .register(meterRegistry);

MicrometerMetricsTracker.java (my change)

        Gauge.builder(METRIC_NAME_TOTAL_CONNECTIONS, Integer.class, (i) -> poolStats.getTotalConnections())
                .description("Total connections")
                .tags(METRIC_CATEGORY, poolName)
                .register(meterRegistry);
        Gauge.builder(METRIC_NAME_IDLE_CONNECTIONS, Integer.class, (i) -> poolStats.getIdleConnections())
                .description("Idle connections")
                .tags(METRIC_CATEGORY, poolName)
                .register(meterRegistry);
        Gauge.builder(METRIC_NAME_ACTIVE_CONNECTIONS, Integer.class, (i) -> poolStats.getActiveConnections())
                .description("Active connections")
                .tags(METRIC_CATEGORY, poolName)
                .register(meterRegistry);
        Gauge.builder(METRIC_NAME_PENDING_CONNECTIONS, Integer.class, (i) -> poolStats.getPendingThreads())
                .description("Pending threads")
                .tags(METRIC_CATEGORY, poolName)
                .register(meterRegistry);

[osialr]

I had a similar problem with my own implementation.

Micrometer gauges wrap their obj input with a WeakReference. So by the time the gauge gets read, PoolStats was already Garbage Collected and the gauge returns a default of NaN.

The fix is to store PoolStats as a field in MicrometerMetricsTracker so that it stays alive. Your fix works because it keeps a strong reference around by binding to the lambda.

[Cuneytt]

I experience the same issue with Spring Boot 2.0.0.Release ( HikariCP:2.7.8, micrometer-registry-prometheus:1.0.1). The code for registering the gauges look the same as the code snippet at the top of the issue:

this.totalConnectionGauge = Gauge.builder(METRIC_NAME_TOTAL_CONNECTIONS, poolStats, PoolStats::getTotalConnections)
         .description("Total connections")
         .tags(METRIC_CATEGORY, poolName)
         .register(meterRegistry);

Is there anything that needs to be done manually in the code to get this working as there is no PR for this issue?

[osialr]

The fix would look something like this. Sorry that I don't have time to make a PR.

public class MicrometerMetricsTracker implements IMetricsTracker {

  /// ... other fields

  @SuppressWarnings("unused")
  // Must keep a reference to pool stats because gauges uses WeakReference.
  // See implementations of micrometer's `MeterRegistry::newGauge` for details
  private final PoolStats poolStats;

   MicrometerMetricsTracker(final String poolName, final PoolStats poolStats, final MeterRegistry meterRegistry)
   {
      // Save strong reference to pool stats here because micrometer gauges wrap the input in a WeakReference
      this.poolStats = poolStats;

      // Rest of constructor
   }

[Qsi1984]

I experience the same with Spring Boot 2.0. Some Metrics like "hikaricp.connections" have a value of NaN. If I try to use Spring Boots Feature to send the metrics to New Relic, it will not work because NaN is not a valid json value.
Are there plans to fix this?

[shakuzen]

This was fixed with #1108. Sorry I missed linking this issue when I opened the pull request.
@brettwooldridge it would be nice for Micrometer users if there were a patch release including the fix.

[Qsi1984]

Thanks for your reply. Seems to be Fixed with version 3.0.0. I was able to add that dependency to my Spring Boot application and the metrics can now successfully be sent to New Relic.
Hopefully the Spring Boot Team will upgrade the HikariCP-Version in their upcoming 2.0.1 Release.

[wilkinsona]

We can't jump from 2.7.8 to 3.0.0 in a maintenance release of Spring Boot. From Boot's perspective, it would be preferable if there was a 2.7.9 release of Hikari that incorporated the necessary change.

[brettwooldridge]

I’ll look at creating a v2.7.9 build today.