Micrometer: Associate multiple values with the same tag key

[cuneytcalishkan]

Hello,

We are currently using DataDog for reporting our metrics and wanted to replace it with micrometer and statsd using DATADOG flavor. However, we found out that a use case currently working on DataDog is not supported in micrometer.
DataDog allows to have a single value or key:value format in the reports, see documentation.

This also allows us to report a metric with the same key repeating with different values.
e.g. _metric_key:metric_value_1, metric_key:metric_value_2_

I have come across issue #1170. In our case, we receive a list of values from input and we want to report these values individually with the same tag key, which in return allows us to use graphs like sum/avg by _metric_key_.

I am well aware that not all reporting APIs support this but maybe the individual API integrations should take into account this use case where it is valid for some integrations like DataDog.

Best regards,
Cuneyt

[checketts]

You can get this benefit by using multiple counters.

for(String input: inputs) {
  registry.counter("my.metric",Keys.of("metric_key", input)).inc()
}

Is there any reason this wouldn't work? (Unless you are trying to tie this metric to some sort of requests total and this action would inflate that count, in which I would recommend you create a separate counter to track that)

[cuneytcalishkan]

For a single key that might be a solution. However imagine a case where you add different tags along the lifetime of a request and in the end only once this metric is reported. e.g. some service_1 called for metric_value_1 and some other service_2 called for metric_value_2. In this case, we end up with multiple tag keys that have multiple values which means an exponential number of metrics to report instead of only one.

Another example that we cannot use suggested approach is:
We receive a request with metric_value_1 and we measure the time as t1. // nominal case
We receive a request with metric_value_2 and we measure the time as t2. // nominal case
We receive a request with metric_value_1, metric_value_2 and we measure the time as t3. // In this case t3 doesn't correspond to the time spent serving for metric_value_1 or metric_value_2 alone but to both of them combined and in this case the time reported will be wrong with the suggested approach.

[shakuzen]

we end up with multiple tag keys that have multiple values which means an exponential number of metrics to report instead of only one.

That is how tags work if you keep making unique combinations of key/values. There is one time series for each unique metric name/type and tag key/value. Is the difference that you're expecting previous tag key/values to not be reported if they haven't been recorded recently?

Otherwise I'm not understanding the issue here. You can make metrics with the same tag name and different tag values using the current API already. Could you illustrate the issue with a code sample to make it more clear?
Note that the metric recording API is not coupled to the reporting mechanism, which changes depending on the backend.

[cuneytcalishkan]

Hello,

I think the use case is pretty simple that we are facing, no need for a code sample. We would like to be able to report a metric with the following tags: key:value_1, key:value_2. This is currently the case with DataDog client where you don't need to actually provide a key/value pair at all. But if the string value you provide includes a : (colon), the left hand side of the string is treated as a key the right hand side as value.

Currently this is not supported because in the dedup method of Tags class, this is prevented.

I have tried sending metrics with this use case and only 1 of the tags is sent, even captured network packets to make sure that there is no problem with DataDog monitoring.

[shakuzen]

Thank you for helping me understand. I didn't realize we were deduplicating tags by key name always. I was able to reproduce the issue now.

[mathurn]

Hi @shakuzen, has a fix for this use case been implemented? We've also have multiples tags with different values for the same key.

[balazssandor]

I am also still curious if we have a solution for this.

[harspras]

Any updates on this?

[jayshao]

I'd also like to upvote this. We have a number of use cases where our data-model is multi-valued, and it would be nice to represent each value as a tag.

If we understand the way tags are persisted (e.g. split off into separate independent time-series) it seems like this should be feasible - excepting deduping on input

[shakuzen]

Sorry for the lack of update on this. While I understand the technical limitation we have right now in Micrometer and that Datadog allows such a tagging model, I'm not sure we can easily justify supporting this for Datadog when it isn't portable to other backends and we would have to deduplicate for reporting to all other backends.

Just so everyone is clear in this issue, you can already record metrics that have different possible values for the same tag. For example, http.server.requests may tag a method which corresponds to the HTTP method. There are different HTTP methods, so it can have different values. The difference here is that this issue is to record a metric with multiple values at the same time, which doesn't happen with the HTTP server request metrics. One recording corresponds to one request, which will only have one value of HTTP method at a time, even though different requests may have different values. So we already support something like the following (pseudocode representing the state of metrics):

http.server.requests.count uri=/books method=get 5
http.server.requests.count uri=/books method=post 3

From this you can derive that the /books uri received 5 GET requests and 3 POST requests, for a total of 8 requests. What we do not support is recording a metric with a tag method having values post and get for the same metric value (e.g. count). I couldn't think of a good example with a valid use case, so I continue with this example.

http.server.requests.count uri=/books method=get 3
http.server.requests.count uri=/books method=get method=post 4
http.server.requests.count uri=/books method=post 2

While not a realistic example, I suppose this would be interpreted as 3 GET requests, 2 POST requests, and 4 requests where the method was POST and GET.

I am surprised there are this many people with use cases for this. Again, the lack of portability and breaking change required for this is a significant hurdle to get over. We would have to allow this in common API, which affects all users.