Skipper: Prometheus route metrics still available after deleting route

[wndhydrnt]

While working with Skipper, I recognized that metrics of deleted routes are still exposed on the Prometheus endpoint "/metrics".

traefik faced a similar issue and documented their solution in the source code[1].
There is also a question regarding obsolete metrics in the prometheus-users group[2].

1: https://github.com/containous/traefik/blob/a09dfa3ce10f222ab6e10ac11386d330698ff6f8/metrics/prometheus.go#L48-L59
2: https://groups.google.com/d/msg/prometheus-users/EbumSJYsJyc/wvdL_NqqAAAJ

---

Steps to reproduce

1. I'm running Skipper Version 2c45dcac04f8e6e4da6891eacd0f9592942192cc (built and executed locally) and Kubernetes v1.11.6 on minikube
2. Deploy an application:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment-prod
  labels:
    app: web
    release: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
      release: prod
  template:
    metadata:
      labels:
        app: web
        release: prod
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

---
kind: Service
apiVersion: v1
metadata:
  name: web-service-prod
spec:
  selector:
    app: web
    release: prod
  ports:
  - protocol: TCP
    port: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: skipper
  name: web-ingress
spec:
  rules:
  - host: web.dev.local
    http:
      paths:
      - backend:
          serviceName: web-service-prod
          servicePort: 80
        path: /

1. Send a request to the application: curl -i -H'Host: web.dev.local' 127.0.0.1:9999
2. Check the "/metrics" endpoint of Skipper. Metrics like the following are exposed:

skipper_response_duration_seconds_sum{code="200",method="GET",route="kube_default__web_ingress__web_dev_local_____web_service_prod"} 0.000525136
skipper_response_duration_seconds_count{code="200",method="GET",route="kube_default__web_ingress__web_dev_local_____web_service_prod"} 1

1. Delete the Ingress: kubectl delete ingress web-ingress
2. Check the "/metrics" endpoint of Skipper again.

Expected result

No metrics with the route ID kube_default__web_ingress__web_dev_local_____web_service_prod are exported.

Actual result

All metrics with the route ID kube_default__web_ingress__web_dev_local_____web_service_prod are still exported.

[szuecs]

@wndhydrnt why do you expect this?
And what do you think is the benefit?

I see that we can save some memory, but we would have to have an additional goroutine to cleanup old data and normally I would only do a cleanup goroutine, if it makes sense, because for example memory growth is unreasonable. For example we cleanup in client based ratelimit old data from clients that are outdated, such that you can't attack the memory footprint.
Metrics are not bound to data from clients, but the Host header has to be part of a route to make it into a metric, such that I don't see the problem.

[wndhydrnt]

Hello @szuecs, thanks for the quick response!
You are right, I should have elaborated more on this.

This is why the behaviour seemed odd to me and why I created the issue:

• For me personally, it is unexpected that a route ID that does not exist at the "/routes" endpoint is still exported at the "/metrics" endpoint. I thought that what is exposed at the "/routes" endpoint is the "source of truth" and metrics are kind of a derivative that can only exist if the route ID is known to Skipper.
  To give an example, it is as if cAdvisor would export metrics of Docker containers that have long been stopped and deleted on the host.
• I am used to this behaviour of only reporting what is currently available due to my work with other exporters over the years.
  Prometheus Exporters like haproxy_eporter or redis_exporter, do not export metrics of previously available entities that have been deleted/removed.
  In case of haproxy_eporter, HAProxy itself removes frontends/backends from its stats endpoint. The exporter just scrapes the result.
  In case of the redis_exporter, an explicit Reset() is done [1].
  I've also implemented Prometheus exporters myself and exported only metrics of entities that exist at the point in time when the metrics endpoint is scraped.
• The metrics need to be stored on the Prometheus server that scrapes Skipper even though they are of no use anymore since the route is gone. A bit of storage wasted.
• The metrics are still visible in dashboards, e.g. when generating variables in Grafana based on the value of the route label of a metric.
• Calculations in Prometheus can return wrong values because metrics are kept in Prometheus. This is the issue that traefik was facing if I understand their post in prometheus-users right. I will follow up with an example query tomorrow. Currently at home and without a good Skipper setup.

I'd like to add that I never thought of this from the point of memory consumption of the Skipper process. It is more about the usability (can't think of a better word) of the metrics when working with Skipper on a daily basis.

I hope the points above provide more context to the issue. If not, feel free to post more questions.

1: https://github.com/oliver006/redis_exporter/blob/e4e6fe5c5d2d76a926489d047aba751834a04ca6/exporter/redis.go#L339-L340

[szuecs]

Thanks, I think it's a feature request and I would be happy if we get a PR to have some cleanup goroutine that deletes old data.

If I understand correctly GaugeVectors are easy to reset, because metrics never rely on old data, but other metrics can not easily be reset. So I would expect that we need a cleanup goroutine, that handles the cleanup, which would mean we are eventual consistent, which IMO makes sense, because it is a usability feature.

[AlexanderYastrebov]

Connecting the dots. Filter cleanup is a part of route cleanup, so these #202 #1552 may be related