Kube-state-metrics: Filter annotation "annotation_kubectl_kubernetes_io_last_applied_configuration"

[towolf]

Kube-state-metrics 1.7.1 adds new kube_*_annotations metrics.

I disabled most of them because that sounded really spammy.

--metric-blacklist="kube_[^i][a-z]+_annotations"

But I wanted to keep one exception: kube_ingress_annotations. There is no other way to capture the ingess class, than via the annotations: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

We use kubectl apply in our CI/CD pipeline and now every _annotations metric contains the full object itself in the annotation_kubectl_kubernetes_io_last_applied_configuration label.

Now, we don't store sensitive stuff in the objects, but I think this specific annotation should be excluded.

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

see above

...

[towolf]

Also, annotations are an uncontrolled field, where anything can go, very few contraints are placed on them. Just exposing them across the board is a very bold move.

[lilic]

Yes agreed, I am seeing those as well, we should filter the annotation_kubectl_kubernetes_io_last_applied_configuration out in the all the annotations. Unless anyone finds those useful? cc @tariq1890 @brancz

A full example of how the metric looks like with the annotation_kubectl_kubernetes_io_last_applied_configuration annotation included:

kube_deployment_annotations{namespace="default",deployment="hello-world",annotation_kubectl_kubernetes_io_last_applied_configuration="{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"hello-world\",\"namespace\":\"default\"},\"spec\":{\"replicas\":2,\"selector\":{\"matchLabels\":{\"run\":\"load-balancer-example\"}},\"template\":{\"metadata\":{\"labels\":{\"run\":\"load-balancer-example\"}},\"spec\":{\"containers\":[{\"image\":\"gcr.io/google-samples/node-hello:1.0\",\"name\":\"hello-world\",\"ports\":[{\"containerPort\":8080,\"protocol\":\"TCP\"}]}]}}}}\n",annotation_deployment_kubernetes_io_revision="1"} 1

[towolf]

Just one example, if you hardcode a password in env: then this would pop up in your Prometheus metrics. You shouldn't do that, bless your heart, but it could happen.

And there's probably no utility in having this field anyway other than to test the breaking point of Prometheus.

[tariq1890]

Your concerns are valid @towolf. Here are some options that I can think of:

I) Perhaps, we could disable annotation metrics by default in kube-state-metrics and provide users the knobs to enable the specific annotation metrics they need. One way of achieving this is the white black listing approach.

ii) We could choose to only expose certain annotation metrics(create specific metric families) that are if significance to Kubernetes and useful to the users.

[lilic]

I would say for now we filter out the annotation_kubectl_kubernetes_io_last_applied_configuration from annotation metric and backport this to the 1.7 branch and cut a v1.7.2 release. I don't think this annotation was meant to be in that metric in the first place. This would at least solve the immediate problem. WDYT @tariq1890?

As for the actual solution, it would probably end up in the next major release, I would go with something along the lines of number 2. What were you thinking, something similar to black/whitelist flag we have? Lets maybe open a new issue to discuss this.

[tariq1890]

Sounds good @LiliC . We can create the patch to filter that specific annotation metric out.

[lilic]

Think we can close this, as the PR to remove the annotation metrics was merged. Thanks for the issue!

[towolf]

Thanks for the fix :)