Kube-state-metrics: Add optional GET arguments to /metrics

[dohnto]

Current situation

Our company have on-premise kubernetes cluster for several teams. One team use several namespaces. We want to run single kube-state-metrics as a deployment inside kubernetes and we want to allow teams to run their own prometheus and scrape kube-state-metrics deployment.

However, every team has to have their special relabel configuration and drop other namespaces otherwise dataseries count would rise significantly. Also most of the time series are transferred through networked and they are just dropped by prometheus.

Proposed solution

Would it be possible to add optional GET arguments to metricsPath? I could imagine something like:

/metrics?namespace=team1-staging&namespace=team1-production
/metrics?collectors=deployments&namespace=team1-staging

Although I am not sure whether this is compatible with gRPC. I am willing to do the implementation but first I would like to discuss it with maintainers.

Thank you for any ideas or tips how to solve our issue.

[brancz]

Generally I like all that you're proposing and I think it would make sense to add these filters by parameters. However you are inevitably going to run into a scalability problem, where the sheer amount of objects is too much for a single kube-state-metrics deployment, at which point you have to deploy a kube-state-metrics instance per namespace and or collector anyways, which is already possible today. The latter also makes a lot of sense from a security/access control view, as kube-state-metrics then really only has access to what it must view, instead of what it could view.

[dohnto]

@brancz thank you for quick response.

Ok I understand. It is probable, that we will end up running several kube-state-metrics instances (per namespace). But it would be awesome, if we could run kube-state-metrics per team (but one team owns few namespaces).

I appreciate that you implemented --namespace argument but it would be awesome, if this argument could be passed multiple times. Would you accept such extension? Or another way how to express that we want are interested not in single but multiple namespaces.

Or is it possible to achieve this using Roles, Rolebindings and ServiceAccounts? To limit the scope of what kube-state-metrics is able to see?

[brancz]

I appreciate that you implemented --namespace argument but it would be awesome, if this argument could be passed multiple times. Would you accept such extension? Or another way how to express that we want are interested not in single but multiple namespaces.

Yes I think this would be a great addition, the way I would implement this would be by instantiating multiple caches for each namespace.

As I said, even the first thing you proposed is something I think is useful (although rather complicated to implement I think with the standard go client).

[dohnto]

For a consistency, wouldn't it be better to add extra arguments
--namespaces=test,stage (note the plural, same as --collectors) and after deprecate the --namespace?

Or can anyone help me with the interface the upstream prefers?

[brancz]

for compatibility reasons, I'd prefer not to change the name of the flag, but extend the usage to accept lists as you suggested

[andyxning]

@brancz @dohnto

However, every team has to have their special relabel configuration and drop other namespaces otherwise dataseries count would rise significantly. Also most of the time series are transferred through networked and they are just dropped by prometheus.

IIUC, the relabel config for Prometheus happens before scrape. So, it should have no network overhead. As the useless metrics have been dropped from the target. Am i right? @brancz

As I said, even the first thing you proposed is something I think is useful (although rather complicated to implement I think with the standard go client).

Seems Prometheus client and server do not support query parameter?

I appreciate that you implemented --namespace argument but it would be awesome, if this argument could be passed multiple times. Would you accept such extension?

Extending --namespace to support multiple times is preferred in order to keep backward compatibility.

[brancz]

IIUC, the relabel config for Prometheus happens before scrape. So, it should have no network overhead. As the useless metrics have been dropped from the target. Am i right? @brancz

I believe what @dohnto was talking about was "metric relabeling" which does happen at scrape time, not discovery.

Seems Prometheus client and server do not support query parameter?

The prometheus server supports it all, but the golang client is rather spartan on this end.

[dohnto]

Please see #395 for my naive solution.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close