Spring-cloud-gateway: OutOfMemoryError - issue with metrics

Created on 25 Jul 2018 · 14Comments · Source: spring-cloud/spring-cloud-gateway

Application fails with OutOfMemoryError after several days without restart. Issue is caused by Spring Boot issue #13895 - there is no possibility to disable request auto timing using property management.metrics.web.server.auto-time-requests. Value of this property should be set to false in case of spring-cloud-gateway otherwise it might be an issue if URIs contains path variables with large cardinality. Instead of recording single metric with placeholders multiple metrics will be created.
Seems like issue #171 was related to same root cause - in case of #171 removing micrometer dependency removed MeterRegistry from class-path and as a result org.springframework.boot.actuate.autoconfigure.metrics.web.servlet.WebMvcMetricsAutoConfiguration got disabled by rule @ConditionalOnBean(MeterRegistry.class).

Source

EdgarsM

Most helpful comment

I believe we can use the following WebFluxTagsProvider implementation as a workaround until this gets a proper fix.

@Component
@Primary
public class CustomWebMVcTagsProvider implements WebFluxTagsProvider {

  private static final Tag URI_UNKNOWN = Tag.of("uri", "UNKNOWN");

  @Override
  public Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable ex) {
    return Arrays.asList(WebFluxTags.method(exchange), uri(exchange),
        WebFluxTags.exception(ex), WebFluxTags.status(exchange));
  }

  private Tag uri(ServerWebExchange exchange) {
    Tag tag = URI_UNKNOWN;
    if (exchange != null) {
      final Route route = exchange.getAttribute(GATEWAY_ROUTE_ATTR);
      if (route != null) {
        tag = Tag.of("uri", route.getUri().toString());
      }
    }
    return tag;
  }
}

Cheers

ivelkov on 3 Sep 2018

👍3

All 14 comments

Current workaround is to remove Micrometer dependency just as in #171 but in case you would still like to collect metrics for your application WebFluxMetricsAutoConfiguration auto configuration can be excluded so that you can still benefit from all metrics except HTTP request auto timing.

EdgarsM on 25 Jul 2018

I dont see what there is to do in the Gateway if the problem is with Boot.

ryanjbaxter on 25 Jul 2018

I think it would be good to disable auto time requests in spring-cloud-gateway (perhaps override defaults of management.metrics.web.server.auto-time-requests) or clearly state that by not disabling auto request timing it could blow up metric cardinality. What are your thoughts?

EdgarsM on 25 Jul 2018

Ping @jkschneider

spencergibb on 25 Jul 2018

I think we should instead introduce a MeterFilter#maximumAllowableTags for this metric much like we do for RestTemplate metrics where we can't guarantee that folks don't incorrectly use RestTemplate with path variable substitution.

If you'd like, you can create a variation on this that does something else like logging periodically when the maximum is reached and pointing folks to probably the better solution which is to use their domain knowledge of the structure of their URIs to construct a MeterFilter that rewrites tag values to reduce cardinality.

jkschneider on 25 Jul 2018

I believe we can use the following WebFluxTagsProvider implementation as a workaround until this gets a proper fix.

@Component
@Primary
public class CustomWebMVcTagsProvider implements WebFluxTagsProvider {

  private static final Tag URI_UNKNOWN = Tag.of("uri", "UNKNOWN");

  @Override
  public Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable ex) {
    return Arrays.asList(WebFluxTags.method(exchange), uri(exchange),
        WebFluxTags.exception(ex), WebFluxTags.status(exchange));
  }

  private Tag uri(ServerWebExchange exchange) {
    Tag tag = URI_UNKNOWN;
    if (exchange != null) {
      final Route route = exchange.getAttribute(GATEWAY_ROUTE_ATTR);
      if (route != null) {
        tag = Tag.of("uri", route.getUri().toString());
      }
    }
    return tag;
  }
}

Cheers

ivelkov on 3 Sep 2018

👍3

@EdgarsM is this still a problem since the Boot issue was fixed in Boot 2.0.6?

ryanjbaxter on 21 Feb 2019

@EdgarsM

Spring Boot fixed this problem by mapping the unknown url to a specific tag after 2.0.8

https://github.com/spring-projects/spring-boot/pull/15609

youzipi on 23 Feb 2019

So this means that the http_server_requests metric tag uri will be "UNKNOWN" where as previously it was the actual path, right?

tony-clarke-amdocs on 23 Feb 2019

So this means that the http_server_requests metric tag uri will be "UNKNOWN" where as previously it was the actual path, right?

only the urls that have no match pattern will be unknown. in gateway, that is the proxied urls. gateway's own urls will still be mapped to its pattern.

youzipi on 24 Feb 2019

AFAICT, all route uri's (ie uri's that haven't been processed by filters) will get tagged as part of gateway's metrics (see https://github.com/spring-cloud/spring-cloud-gateway/blob/0f153377f3bc4c3374e251a57fa3f4189df53810/spring-cloud-gateway-core/src/main/java/org/springframework/cloud/gateway/filter/GatewayMetricsFilter.java#L96-L102). We can't do the processed uris because of the explosion problem.

Unless anyone can convince me otherwise, I'm closing this as fixed in boot.

spencergibb on 25 Feb 2019

@spencergibb the issue was fixed in spring-boot, but the problem is that http.server.requests is not usable anymore in Gateway. All routes in Gateway will get UNKNOWN tag. Maybe it makes sense to override WebFluxTagsProvider bean in Gateway with the one provided by @ivelkov ? I can make PR if you think this is a good idea.