Micrometer: Add possibility to unregister metrics

[lazystone]

My use case is that I have several instances of a service, but only one of them collects some database related metrics.

So if all services report metrics then it looks a little bit strange - some report zero, and one report correct values.

And I can't set it up during service start-up time, because it might be situations when one service stops to collect those metrics and another one starts(after cluster re-balance for example).

It would be nice to have a possibility to disable or de-register some metrics when needed and turn them on or register them again.

[checketts]

Good point. I was investigating using a MeterFilter that could take effect, but that wouldn't remove metrics that had already been created.

[lazystone]

For now I use some kind of workaround - if gauge should be disabled then in the ToDoubleFunction I return Double.NaN
But then my logs are spammed by messages that those metrics could not been exported.

[crmky]

I have similar requirement on that.

E.g. I monitor database level metrics in a database cluster. The database can be created and deleted (although rarely) on the fly, it will be good if I can delete the related metrics if a database is deleted.

[eaglerainbow]

+1 yet another case here: In my use case the monitoring of a set of counters happens on instance level. Those instances do only have a limited lifetime (in the order of minutes). The number of instances may become large (over time, but only a small set of instances are live in parallel) and this may then cause trouble with the performance of the MeterRegistry, if not cleaned up (besides the fact that it "looks strange").
My current idea to solve it would be to deregister the metrics on protected void finalize() (which in my case would be Good Enough™️, even though not triggered predictably).

[eaglerainbow]

Side note on the topic of "how to solve this problem in a unit test": I am using Micrometer in a Spring environment. I could work around that problem by making use of

@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)

By this, each time the Spring Application Context is rebooted and a new (empty) instance of the MeterRegistry is created. Restarting the Application Context is quite costly, though, but acceptable in my limited case. For sure a clear() method at the MeterRegistry would have been preferred.

[eaglerainbow]

Is this related also to #368 ?
Some answers sound differently, but the title and this post sounds very similar...

[eaglerainbow]

I would have tried to contribute to this topic here, but I am currently unable to get the gradle build environment running (either fails on parsing the pom of ehcache:2.10.5 or the unit test io.micrometer.core.instrument.binder.system.ProcessorMetricsTest.hotspotCpuMetrics fails). Moreover, there is also no contribution statement (or it at least I could not find it).

[eaglerainbow]

After some tricks, I could make gradle work just so-so. At https://github.com/eaglerainbow/micrometer/tree/micrometer-er-479 I tried to start implementing a proof-of-concept, making the MeterRegistry to handle de-registrations. However, I am much too little into micrometer to be able to assess all the next steps: Here is what I could figure out myself, though:

• I have the impression that CompositeMeterRegistry needs to learn handling this case too, as it also listens to onMeterAdded events. However, the context is entirely unclear to me (as there are no comments on what nonCompositeDescendants really means or does).
• I am mainly coming from the Prometheus' world. From the fact that PrometheusMeterRegistry directly registers new meters into a locally-controlled CollectorRegistry, I assume that all target-specific MeterRegistries need to support the removal procedure, too. For sure, that task will be way beyond my capabilities.
• With f1d1d0c9c6fc971d4ab206a00c797e7843c7c7f7 I started to prototype a deregistration for Prometheus, but currently I am struggling with setting up an appropriate test environment.
• I had a quick look into CloudWatchMeterRegistry to get a second opinion and there it seems that due to the usage of getMeter(), my initial change might already be sufficient to do the trick (I did not test that though!)
• I also had a brief look at the JMXMeterRegistry, which is nice in that sense that all its operations are almost completely provided by DropwizardMeterRegistry. Looking at com.codahale.metrics.MetricRegistry, it turns out that it has a method remove(String), which we could feed with a hierarchicalName(id). Looking at the dependencies of DropwizardMeterRegistry it looks reasonable to assume that this approach could (at least half)-provide support also for Ganglia and Graphite (besides JMX).
• InfluxMeterRegistry uses MeterPartition internally, which in turn uses getMeter() again. So, there we also might be lucky (seems to be a similar case as CloudWatchMeterRegistry). The same also appears to hold true for Wavefront, SignalFX, NewRelic, Elastic, datadog.
• Atlas again seems to be more like prometheus, but I could not find support for removing meters at com.netflix.spectator.api.AbstractRegistry. I have no clue how this could be done there. It would be a bad thing, if removal support would be optional to the MeterRegistry, due to the fact that one target technology does not support it.
• For statsd the coding currently appears to me that pollableMeters has taken over the role of an underlying "MetricRegistry" for the polling cases. However, the coding at io.micrometer.core.instrument.distribution.HistogramGauges gives me a hard time to understand whether removing it from pollableMeters would also lead to success for distribution-based metrics.

I am willing to continue contributing to this endeavor here, but I would be thankful for some further assistance from someone more experienced than me...

[theodiefenthal]

I am also highly interested in this feature. Once implemented, this would also allow something like an "update/edit".
My current scenario: From a third party library, I have a callback method public void onOpen(Context ctx). I want to have a gauge metric on this ctx object. But during the lifetime of my "master"-object that ctx object can be closed multiple times and be opened again later on (as new object!) => Right now, I need to make a wrapper around my Gauge so that the Gauge data object is static and holds the possibly varying context. Just because of the metrics. Ideally, I'd prefer to just unregister the gauge if the context is closed and reregister it when reponed.

@eaglerainbow I'm also rather new to Spring Boot and use metrics only in combination with InfluxDB, where I'm more experienced with. But regarding some of your points

• As far as I understand nonCompositeDescendants: A CompositeMeterRegistry can hold multiple MeterRegistrys. Those hold CompositeMeterRegitrys can of course be again CompositeMeterRegistry, thus making of kind of a tree, e.g.:

   /                      \
InfluxDB           Composite
                    /                 \
             Simple             Prometheus

And nonCompsiteDescendants means that the children are finally tree nodes, thus "real" registries and not composite anymore.

• As far as I checked out InfluxMeterRegistry, I totally agree that changing getMeter would be sufficient here. And from database perspective, it is totally OK that a meter just stops delivering values. In the end, that's what I like to have and see the "gap" that there is no more data ingested.

[Purfakt]

We really need this feature too:

• We're binding metrics from CPU, RAM, network interfaces and storages in a docker environment.
• Some times (during an update, a restart, a crash, ...) containers are restarted and network interfaces and dynamic storages are recreated.
• Some times, we create new containers and some others are deleted.

It would be great if we could tell micrometer to stop bothering with calling the toDouble method from an already bound metric.

[jkschneider]

Fixed with fc37580df416b1438a2f14e0f60b1009a6c42662.

[cfecherolle]

Side note on the topic of "how to solve this problem in a unit test": I am using Micrometer in a Spring environment. I could work around that problem by making use of

@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)

By this, each time the Spring Application Context is rebooted and a new (empty) instance of the MeterRegistry is created. Restarting the Application Context is quite costly, though, but acceptable in my limited case. For sure a clear() method at the MeterRegistry would have been preferred.

Actually, this might not be enough to reload the whole Spring context for each test and reset the registry, in case of Spring Boot Integration tests. In my case I had to also had this annotation to make the registry clear actually work:
@TestExecutionListeners({DirtiesContextTestExecutionListener.class})

I know this issue is outdated but, might help someone!
(in my case, I had no access to the Registry instance in my integration tests, thus couldn't call clear() on it)

[jefersonm]

Side note on the topic of "how to solve this problem in a unit test": I am using Micrometer in a Spring environment. I could work around that problem by making use of

@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)

By this, each time the Spring Application Context is rebooted and a new (empty) instance of the MeterRegistry is created. Restarting the Application Context is quite costly, though, but acceptable in my limited case. For sure a clear() method at the MeterRegistry would have been preferred.

Actually, this might not be enough to reload the whole Spring context for each test and reset the registry, in case of Spring Boot Integration tests. In my case I had to also had this annotation to make the registry clear actually work:
@TestExecutionListeners({DirtiesContextTestExecutionListener.class})

I know this issue is outdated but, might help someone!
(in my case, I had no access to the Registry instance in my integration tests, thus couldn't call clear() on it)

Not sure if is the same scenario for everyone, but as I'm using Micrometer with Prometheus:

<dependency>
  <groupId>io.micrometer</groupId>
   <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

I had to make sure PrometheusMeterRegistry is removed after each test round:

@Autowired PrometheusMeterRegistry prometheusMeterRegistry;

@AfterEach
void shutdownMicrometerRegistry() {
        Metrics.globalRegistry.remove(prometheusMeterRegistry);
}