Metrics-server: For prometheus, where to scrape metrics from metrics-server?

[cofyc]

Hi,

After setup, I only get this apis endpoints from https://localhost:443/:

# curl -k -XGET -H "Authorization: Bearer $TOKEN" https://localhot:443/
{
  "paths": [
    "/apis",
    "/apis/metrics",
    "/apis/metrics/v1alpha1",
    "/healthz",
    "/healthz/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/swaggerapi/"
  ]
}

Where does metrics-server expose metrics in Prometheus format?

I want to config my prometheus to scrape metrics from metrics-server, because I need information from kubelet /stat/summary endpoint.

[DirectXMan12]

metric server does not expose the underlying metrics -- it's not intended to be used for this purpose. You'll need to scrape them directly from the nodes. In 1.7.3+, they should be available at $KUBELET/metrics/cadvisor.

[kfox1111]

please provide more info on what metrics-server is, if not a server for exposing metrics.

[r4j4h]

I found some documentation around metrics-server on the official docs page: https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/

[piosz]

@kfox1111 see #21

[yashubh]

@cofyc have you got answer of your query ? I am also at the same point , new in this domain , any help appreciated

[marcel-dempers]

I was a little confused. When I heard of heapster's deprecation I recall not pointing directly to any server named heapster or seeing it anywhere in my prometheus scrape config.

When I look at metrics-server docs, I see the new URL is /apis/k8s.metrics.io/

I currently scrape the below to get node and pod metrics. Is this good or should I be using /apis/k8s.metrics.io ?

https://kubernetes.default.svc:443/api/v1/nodes/[NODENAME]/proxy/metrics/cadvisor
https://kubernetes.default.svc:443/api/v1/nodes/[NODENAME]/proxy/metrics

[DirectXMan12]

Prometheus does not scrape heapster. It goes directly to the nodes, or via the API server proxy to the nodes. What you're doing is correct.

/apis/metrics.k8s.io (and, consequently, metrics-server) is intended for consumption by kubernetes system components so that we're not tied to a particular TSDB. It's not designed as a way for metrics pipelines to gather metrics.

More succinctly:

• controllers and kubectl top read from /apis/metrics.k8s.io via metrics-server
• metrics pipelines, like Prometheus, sysdig, etc should read directly from the nodes' /metrics/cadvisor endpoint
• human users should use read from their metrics pipeline, or kubectl top if they need a really basic quick view
• \

[showermat]

@DirectXMan12 I think this is in need of an update given the decision to remove cAdvisor endpoints (kubernetes#68522). The metrics are still available through /apis/metrics.k8s.io, but I can't find anywhere they're exposed for user consumption. Is there a new recommended way to obtain these metrics with (for example) Prometheus, or am I stuck falling back on a metrics-server exporter?

[DirectXMan12]

That mentioned issue also involves making sure the relevant metrics are exposed on Prometheus format somewhere discoverable on the node.

[deadc]

we made a metrics-server-exporter for metrics server, can be useful for those who miss the heapster metrics

[shamimgeek]

could you tell me which endpoint should i check to see metrics provided by kublet on nodes ?

[rehevkor5]

Since metrics-server doesn't supply metrics about itself, is there a good way to monitor it?