Micrometer: Provide statsd support

[jkschneider]

Any such implementation will have to be configured or somehow discover the particular statsd server it is interacting with so as to support the many specializations of the protocol (e.g. Datadog's).

[robinst]

@jkschneider Any updates on this?

[jkschneider]

Trying to shore up the core API in advance of the Spring Boot 2 M4 milestone which is scheduled for release on September 14. After that, I'd like to circle back and pick up this and CloudWatch as two additional backends.

[gitrc]

statsd is the best. hook us up.

[jkschneider]

@gitrc Any particular flavor? Vanilla statsd or dogstatd? Something else?

[aantono]

I’d vote for DogStatsD! :)

[gitrc]

@jkschneider Thank you for work on this. DogStatsD would meet our requirement of tagging support. We are currently using telegraf and their variant of statsd for tag support. Since telegraf supports DogStatsD we'd be good to go.

It looks like the initial intent in issue #20 was to support DogStatsD and then you went for direct DataDog integration. Was this just based on a particular need?

[jkschneider]

Initially I wasn't aware that you could ship metrics direct to Datadog via their API. Once I learned this, I knew we would support both ultimately, but assumed that folks would prefer direct-to-API if given the choice (since it's one less piece of infrastructure to stand up in an environment, and there really isn't any advantage to using dogstatsd that I can see).

Since then, experience has proven me wrong. Seems like most want the datadog agent to still be present, perhaps because in many cases people already have it running? Perhaps it better suits an organizations division of responsibility regarding credentials management?

[jramsdale]

We prefer vanilla statsd. Is it substantially different from DataDog? Would it be a lot of additional effort to support both?

[jkschneider]

Sorry I haven't been clear. The intent is to support vanilla statsd _plus_ whichever variants are useful to the community. In the case of dogstatsd, tagging support is critical.

[jramsdale]

Got it. Good to hear! It did make sense to support vanilla in addition to an extension.

[estelle-peterson]

dogstatsd would help us move forward with our upgrade to spring boot 2.0.0.m4 / spring 5 and using spring reactor!

[jkschneider]

Etsy has a great listing of statsd resources.

[jkschneider]

The telegraf flavor is documented here. Dogstatsd here.

[jkschneider]

Some notes on the implementation as it stands right now (I expect we will make some refinements yet):

1. StatsD expects timings in milliseconds. Telegraf and Dogstatsd forward the timings to the monitoring system in milliseconds as well. I find the resultant graphs quite confusing, because scales can be difficult to comprehend (e.g. when something reliably takes 100ns, you see 1e-4 on a Datadog graph). Elsewhere in Micrometer, we either report in nanos so you are always scaling up or seconds because that is a true base unit of time, depending on the backend.

2. We do per second rate normalization in DatadogMeterRegistry and InfluxMeterRegistry, which StatsD is not doing. I think this makes it much easier to construct latency and throughput graphs with sensible units on the back side, but the inconsistency worries me a bit. We may have to revisit this before 1.0 GA.

3. There is no implementation of sampling right now (|@0.5 to only ship 50% of samples). With the UDP client being backed by reactor-netty, I simply could not saturate the client's backing queue enough in my tests to cause problems. So, I chose to err on the side of simplicity right now knowing that we can swing back around and add sampling support if necessary.

[jkschneider]

In this initial implementation, we support 3 flavors, configurable via StatsdConfig#flavor():

1. Etsy (what we have been referring to as "vanilla" StatsD). Tags are flattened into the name using HierarchicalNameMapper, the same mechanism we use for other hierarchical systems like Graphite.
2. Datadog
3. Telegraf

@gitrc helpfully pointed out that Telegraf does support Datadog's format, but requires a non-default option to be flipped on. Since it wasn't much difficulty to support Telegraf's flavor, I went ahead and added it.

[robinst]

Trying this out now. Anyone else looking for how to use this, this is the dependency (on Maven Central):

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-statsd</artifactId>
        <version>1.0.0-rc.2</version>
    </dependency>

Then use io.micrometer.statsd.StatsdMeterRegistry.

To use a default config, it looks like it's necessary to do this (not sure if that's intended, feels a bit weird):

        StatsdConfig config = new StatsdConfig() {
            @Override
            public String get(String key) {
                return null;
            }
        };