Micrometer: Events in New Relic are missing tags (only single tag is provided)

[larsduelfer]

Hi all,

we are currently also evaluating the meter registry for New Relic besides the one for Prometheus. In New Relic, micrometer metrics are represented as Events. However, the Events only have a single tag and are missing additional information.

Example:

When selecting the Event "tomcatThreadsConfigMax" in New Relic Insights every event instance only has an attribute / tag called "name" which seems to be the thread pool name. When now streaming metrics from multiple microservices to New Relic, it is not possible to select the right service (by name of the spring boot application, for example).

Are we missing something or is this a bug in the registry for New Relic?

Thanks and regards, Lars

[fustic]

hi @jkschneider,
you can add your own tags.

package com.configuration;

import static java.util.Arrays.asList;

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tag;
import io.micrometer.spring.autoconfigure.MeterRegistryCustomizer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

@Slf4j
@Configuration
public class MicrometerConfiguration {
  // https://serverfault.com/questions/427018/what-is-this-ip-address-169-254-169-254
  private static final String EC2_METADATA_ENDPOINT = "http://169.254.169.254";

  private final String serviceName;
  private final String environmentName;
  private final String serviceVersion;
  private final String ec2InstanceId;

  /**
   * Micro meter configuration to include data about infra.
   * @param name service name
   * @param version service version including git hash
   */
  @Autowired
  public MicrometerConfiguration(
      @Value("${spring.application.name}") String name,
      @Value("${spring.application.version}") String version,
      @Value("${ENVIRONMENT_NAME:unknown}") String environmentName
  ) {
    this.serviceName = name;
    this.environmentName = environmentName;
    this.serviceVersion = version;
    this.ec2InstanceId = readEc2InstanceId();
  }

  @Bean
  MeterRegistryCustomizer<MeterRegistry> configureCommonTags() {
    return registry -> registry.config().commonTags(asList(
        Tag.of("service", serviceName),
        Tag.of("environment", environmentName),
        Tag.of("serviceVersion", serviceVersion),
        Tag.of("instanceId", ec2InstanceId)
    ));
  }

  private static String readEc2InstanceId() {
    try {
      return new RestTemplate().getForObject(
          EC2_METADATA_ENDPOINT + "/latest/meta-data/instance-id",
          String.class
      );
    } catch (Exception e) {
      log.error("Couldn't fetch instance ID from EBS", e);
      return "";
    }
  }
}

[jkschneider]

Ah, I misunderstood the original question. I thought @larsduelfer was saying that in spite of the fact that he had added common tags, only one was being shipped to New Relic!

[larsduelfer]

@jkschneider: I was actually exactly saying this: New Relic only receives a single tag for each metric. Are we now on the same page? So for me this is not a "question" but still a "bug".

[fustic]

Unfortunately, I am not on the same page and I couldn't really understand what do you mean @larsduelfer . With the example I provided above the tomcatThreadsConfigMax looks like this:

{
          "event": {
            "serviceVersion": "1.0-f82ff3f",
            "environment": "MY_GREEN_ENV",
            "instanceId": "i-0a686e59aa251c858",
            "service": "SERVICE_NAME",
            "name": "ajp-nio-8009",
            "value": 200,
            "timestamp": 1547219538437
          }
        },

[larsduelfer]

@fustic I would expect, that all tags gathered by micrometer for a metric are by default added to the metrics exposed by the vendor-specific registry (at least this is how I understood the concept so far). The same metrics exposed by using the prometheus registry contain a whole bunch of tags out of the box. I would expect the same tags to be contained in metrics streamed to NewRelic using the corresponding registry. This is not the case at the moment.

[jkschneider]

@larsduelfer I don't see what you're describing happening. The following metric with two tags ships two tags to New Relic:

Counter counter = registry.counter("counter", "method", "actual", "anothertag", "value");

[
  {
    "eventType":"counter",
    "throughput":536,
    "anothertag":"value",
    "method":"actual"
  }
]

Now if we've gotten the exposition format wrong, or it's changed recently, that's another thing I think.

[larsduelfer]

@jkschneider We got confused here on our side, sorry for that! The additional tags we saw in prometheus are added by prometheus when scraping the endpoints (additional kubernetes metadata is added there). So they were not originating from micrometer. Therefore, we could never see them in NewRelic. So the only way is to add them manually as described further above when using NewRelic.