Kube-state-metrics: No labels in kube_node_labels possibly due to parser issues

[paulfantom]

What happened:

I tried using --labels-metric-allow-list flag to enable labels in kube_node_labels metric. Depending on quotation markings results were different (described below).

All deployments were made with following cli arguments, only last line was changed in test cases:

    spec:
      containers:
      - args:
        - --host=127.0.0.1
        - --port=8080
        - --telemetry-host=127.0.0.1
        - --telemetry-port=8081
        - --pod=$(POD_NAME)
        - --pod-namespace=$(POD_NAMESPACE)
        - --labels-metric-allow-list="nodes=["kubernetes.io/arch","gpu.infra/intel","network.infra/fast"]"

Test cases tried on a cluster:

1) all quoted

yaml line: - '--labels-metric-allow-list="nodes=["kubernetes.io/arch","gpu.infra/intel","network.infra/fast"]"'

ksm status: did not start

ksm logs:

invalid argument "\"nodes=[\"kubernetes.io/arch\",\"gpu.infra/intel\",\"network.infra/fast\"]\"" for "--labels-metric-allow-list" flag: invalid format, metric=[label1,label2,labeln...],metricN=[]

2) quoted labels only

yaml line: - --labels-metric-allow-list=nodes=["kubernetes.io/arch","gpu.infra/intel","network.infra/fast"]

ksm status: ksm started

metric output: kube_node_labels{container="kube-rbac-proxy-main", instance="10.42.5.210:8443", job="kube-state-metrics", node="node01"}

3) quoted all parameters to cli option

yaml line: - --labels-metric-allow-list="nodes=["kubernetes.io/arch","gpu.infra/intel","network.infra/fast"]"

ksm status: did not start

ksm logs:

invalid argument "\"nodes=[\"kubernetes.io/arch\",\"gpu.infra/intel\",\"network.infra/fast\"]\"" for "--labels-metric-allow-list" flag: invalid format, metric=[label1,label2,labeln...],metricN=[]

4) quoted array, but not labels

yaml line: - --labels-metric-allow-list="nodes=[kubernetes.io/arch,gpu.infra/intel,network.infra/fast]"

ksm status: did not start

ksm logs:

invalid argument "\"nodes=[kubernetes.io/arch,gpu.infra/intel,network.infra/fast]\"" for "--labels-metric-allow-list" flag: invalid format, metric=[label1,label2,labeln...],metricN=[]

5) no quotes

yaml line: - --labels-metric-allow-list=nodes=[kubernetes.io/arch,gpu.infra/intel,network.infra/fast]

ksm status: started

metric output: kube_node_labels{container="kube-rbac-proxy-main", instance="10.42.5.202:8443", job="kube-state-metrics", label_gpu_infra_intel="true", label_kubernetes_io_arch="amd64", label_network_infra_fast="true", node="node01"}

What you expected to happen:

kube_node_labels have additional labels and quotation marks don't affect ksm operations.
At a minimum, I would expect test case #5 and #2 to produce the same outcome.

How to reproduce it (as minimally and precisely as possible):

Deploy KSM to k8s environment. The problem doesn't occur when starting KSM from the command line

Anything else we need to know?:

Environment:

• kube-state-metrics version: 2.0.0-alpha.3
• Kubernetes version (use kubectl version): 1.19.4
• Cloud provider or hardware configuration: k3s
• Other info: Manifests used for ksm deployment are public and available at https://github.com/thaum-xyz/ankhmorpork/tree/master/apps/monitoring/ksm. Commit with all test cases is available at https://github.com/thaum-xyz/ankhmorpork/commit/1c5be5d5bac833c45b4e55996ef68e7863df8e5f

[lilic]

Thanks for the report! So if I understand correctly it did eventually work for you? With the use case number 5, you just want it to work with different ways? Can you describe the reason you want 2 to work in deployment flag, maybe I missed it, thanks! Or would just documenting 5 with an example be enough?

[paulfantom]

It is not uncommon to pass array parameters quoted, hence I would expect number 2 to work the same as 5.

For me personally, this is a bad UX, especially if case #2 fails silently, while #5 works flawlessly. Ideally, with perfect UX, all those cases would work the same for CLI as they work for YAML manifests. This is currently not the case, possibly due to how bash strips double-quotes from parameters. I suspect this is also why this issue slipped during testing. I also suspect that stripping all single quotes and double quotes before parsing parameter value would solve this issue without affecting ksm operations in any way.

[lilic]

Thanks for the feedback, right now we want to ship 2.0, so if I understand correctly it works just not in all possible scenarios. I would not call this a bug then? Maybe adding documentation for now on how it works in a Deployment manifest, to point people in the right direction.

Happy to come back to this once 2.0 is out, we were looking at actually adding an option to pass a config file, once users have more than a couple of labels. This should solve any of the above problems, correct?

[thomaspeitz]

For anybody not able to get this working:
- --labels-metric-allow-list=nodes=[nameoflabel] works
- --labels-metric-allow-list=nodes=["nameoflabel"] won't work.

Took me longer than I wished to get it working.

Same here. Expected that the second one would work!

[lilic]

PRs are always welcome for either docs https://github.com/kubernetes/kube-state-metrics/pull/1279 or for adding new suggested way of doing it! 🎉

[Drugoy]

Doesn't work for me.
When I save it like
- --labels-metric-allow-list=pods=[owner]
in the YAML, k8s alters it to
- '--labels-metric-allow-list=pods=[owner]'

So my full setup is:

spec.template.spec.containers.[0].args:
  - '--resources=pods'
  - >-
    --namespaces=myproj-dev,myproj-test,...
  - >-
    --metric-allowlist=kube_pod_start_time,kube_pod_created,kube_pod_status_ready,kube_pod_container_status_running,kube_pod_container_status_restarts_total
  - '--labels-metric-allow-list=pods=[owner]'.

ksm starts, but none of the metrics have owner label.

I then tried to investigate possible reasons: some pods didn't have that label.
I reduced the test to just 1 namespace where I also made sure that ALL of the existing pods have that label.
And yet, still, no labels in metrics.

[Drugoy]

Oh my, I just realized that this CLI arg merely adds labels to a set of specific metrics like kube_pod_labels, which I didn't have listed in my --metric-allowlist and thus noticed no difference...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale