Spring-cloud-netflix: Not able to see Netflix default metrics in /metrics endpoint

[KJoyner]

In the documentation, there is discussion on default metrics on every Spring MVC request. Perhaps it is a mis-understanding of mine, but I do not see those in the metrics endpoint. This is true whether I am using servo metrics or spectator metrics.

On the other hand, when I used the Spring Cloud Netflix Contrib libraries from https://github.com/netflix-spring-one/spring-cloud-netflix-contrib, I can see these metrics within the /metrics endpoint.

Should these default metrics be included within the /metrics endpoint?

In partcular, I see a lot of metrics that look as follows when I use the Spring Cloud Netflix Contrib libraries:

"rest(statistic=count,bucket=01s,status=200,exceptionType=none,caller=unknown,handlerName=home,uri=root,method=GET)": 0,
"rest(statistic=count,bucket=01s,status=404,exceptionType=none,caller=unknown,handlerName=errorHtml,uri=error,method=GET)": 0,
"rest(statistic=max,bucket=01s,status=200,exceptionType=none,caller=unknown,handlerName=home,uri=root,method=GET)": 0,
"rest(statistic=max,bucket=01s,status=404,exceptionType=none,caller=unknown,handlerName=errorHtml,uri=error,method=GET)": 0,
"rest(statistic=totalOfSquares,bucket=01s,status=200,exceptionType=none,caller=unknown,handlerName=home,uri=root,method=GET)": 0,
"rest(statistic=totalOfSquares,bucket=01s,status=404,exceptionType=none,caller=unknown,handlerName=errorHtml,uri=error,method=GET)": 0,
"rest(statistic=totalTime,bucket=01s,status=200,exceptionType=none,caller=unknown,handlerName=home,uri=root,method=GET)": 0,
"rest(statistic=totalTime,bucket=01s,status=404,exceptionType=none,caller=unknown,handlerName=errorHtml,uri=error,method=GET)": 0,

[spencergibb]

What metrics are you talking about? Spring Cloud provides hystrix metrics to the /metrics endpoint if hystrix is enabled.

[KJoyner]

In the documentation, it says the following:

Default Metrics Collection
Without any additional dependencies or configuration, a Spring Cloud based service will autoconfigure a Servo MonitorRegistry and begin collecting metrics on every Spring MVC request. By default, a Servo timer with the name rest will be recorded for each MVC request which is tagged with:

HTTP method

HTTP status (e.g. 200, 400, 500)

URI (or "root" if the URI is empty), sanitized for Atlas

The exception class name, if the request handler threw an exception

The caller, if a request header with a key matching netflix.metrics.rest.callerHeader is set on the request. There is no default key for netflix.metrics.rest.callerHeader. You must add it to your application properties if you wish to collect caller information.

Set the netflix.metrics.rest.metricName property to change the name of the metric from rest to a name you provide.

If Spring AOP is enabled and org.aspectj:aspectjweaver is present on your runtime classpath, Spring Cloud will also collect metrics on every client call made with RestTemplate. A Servo timer with the name of restclient will be recorded for each MVC request which is tagged with:

HTTP method

HTTP status (e.g. 200, 400, 500), "CLIENT_ERROR" if the response returned null, or "IO_ERROR" if an IOException occurred during the execution of the RestTemplate method

URI, sanitized for Atlas

Client name

[spencergibb]

What version of spring-cloud-netflix are you using?

[KJoyner]

1.1.0 M4

[spencergibb]

There might be a flaw in the docs that say "Without any additional dependencies".

My guess is, if you add the following dependency it will work:

    <dependency>
        <groupId>com.netflix.servo</groupId>
        <artifactId>servo-core</artifactId>
    </dependency>

[KJoyner]

This library already looks to be included in org.springframework.cloud:spring-cloud-starter-ribbon:1.1.0.M4. Gradle dependencies show, com.netflix.servo:servo-core:0.7.5 -> 0.10.0.

Tried to manually add but didn't seem to make a difference. One thing to note is that if I don't use specator, I do see some servo metrics but not the "rest" metrics. When I add spectator, I no longer see the servo metrics even though it seems that it should wrap and still show them.

[spencergibb]

ping @jkschneider

[s4s0l]

Actually if you use spectator metrics you are talking about are not shown in /metrics endpoints but there are tags status=xxx attached, but there is something broken that makes redirection to /error counted as separate request so when some rest throws exception it will be recorded as http/200 and then again as http/500 but for /error endpoint...

[joetravis]

Not sure if this issue is being discussed anywhere else, but I tried digging into this, and here's what I found:

• The default metrics mentioned by @KJoyner are getting collected and stored in a DefaultMonitorRegistry
• When Spectator is in use (namely, com.netflix.spectator.api.Registry is available), the ServoMetricsAutoConfiguration autoconfig is not active
• This means that there is no MetricReaderPublicMetrics bean fed by a ServoMetricReader
• There is a MetricReaderPublicMetrics bean fed by a SpectatorMetricReader, but this only has access to ServoRegistry, not the entire MonitorRegistry where the default metrics are stored (all of this is configured within SpectatorMetricsAutoConfiguration)
• The metrics endpoint uses all of the available MetricReaderPublicMetrics to report metrics, but since Spectator is in use, it does not have access to the Servo stored metrics

So the docs are correct in stating that the metrics are collected, but the portion indicating that they will be included by actuator in /metrics appear to be incorrect in this instance:

Practically, this means that your code may use a mixture of Servo monitors and Spectator meters and both will be scooped up by Spring Boot Actuator MetricReader instances and both will be shipped to the Atlas backend.

I have found that when I include the relevant Servo Reader and PublicMetrics beans in my own config (in a project that includes Spectator), I do see the default MVC metrics appear, but then the Spectator metrics appear twice (included through both the Servo and Spectator Readers).

Please let me know if it would be helpful to provide any code samples or projects.

[edethoor]

@spencergibb @dsyer
The counter metrics are not showned when we call the /metrics endpoint because of that code in the SpectatorMetricService.class.

The SpectatorHandlerInterceptor class is only available in this project : https://github.com/netflix-spring-one/spring-cloud-netflix-contrib. Maybe this project should be merged in the spring cloud netflix project ?

[spencergibb]

ping @jkschneider

[ShawnTuatara]

I was trying out the Spectator integration with Spring and ran into this issue as well. Any status update on it? Did it ever make it into the spring cloud netflix project?

[spencergibb]

Closing this due to inactivity. Please re-open if there's more to discuss.

[bob-rohan]

Like others, I was trying out the spectator integration (1.4.2-RELEASE) with spring boot (1.5.3.RELEASE) and ran into issues.

@spencergibb could this be re-opened please? It would be great to see one of the following:

• a sample project to demonstrate spectator metrics being published to atlas backend
• a fix delivered so that spectator metrics are published to the atlas backend
• updated documentation (to reflect that spectator metrics are not published to atlas backend)

For others, as a workaround..

I found the comment from @joetravis very helpful. By backing out spring-cloud-starter-netflix-spectator, and introducing servo-core (0.10.1 - version via spring-cloud-netflix-dependencies), I was able to get metrics added to the registry and subsequently published to atlas backend.

It may help others to know that ServoMetricsAutoConfiguration provides hierarchical naming strategy, which can be overridden by providing a dimensional naming strategy bean:

@Bean public ServoMetricNaming servoMetricNaming() { return new DimensionalServoMetricNaming(); }

I'll refrain from commenting on the difference between dimensional and hierarchical metrics, as it's already been well documented: Dimensional vs. Hierarchical Metrics.

Without dimensional naming strategy, the fine grained metrics (collected per MVC request), would have been reported as "one" coarse grained metric. i.e

hierarchical

normalized.servo.rest.totaltime: 0,
normalized.servo.rest.count: 0,
gauge.servo.rest.min: 0,
gauge.servo.rest.max: 0,

rather than many fine grained metrics..

dimensional

rest(method=GET,statistic=totalTime,status=200,type=NORMALIZED,unit=MILLISECONDS,uri=search_findUsers): 0,
rest(method=GET,statistic=count,status=200,type=NORMALIZED,unit=MILLISECONDS,uri=search_findUsers): 0,
rest(method=GET,statistic=min,status=200,type=GAUGE,unit=MILLISECONDS,uri=search_findUsers): 0,
rest(method=GET,statistic=max,status=200,type=GAUGE,unit=MILLISECONDS,uri=search_findUsers): 0,

This makes a massive difference to the amount of insight you can gain from these metrics when they get to the backend.

[jkschneider]

@bob-rohan We've dropped servo integration in spring-cloud-netflix recently because this responsibility has shifted to micrometer.io. Spring Boot 2's OOTB metrics collection is based on Micrometer, which is a dimensional metrics facade supporting Atlas among many other popular monitoring systems.

Micrometer support has been backported to Spring 1.5.x via an additional dependency. The endpoint metrics that previously existed in spring-cloud-netflix via servo come autoconfigured now in Boot 2's actuator, and are autoconfigured in Boot 1.5.x if you include micrometer-spring-legacy.