Charts: [stable/prometheus-operator] kube-scheduler and kube-controller-manager override will append default label selector instead of replace.

[neoakris]

Background Context:
My kops cluster ships with kube-scheduler and kube-controller-manager pods
labeled:
k8s-app: kube-scheduler and k8s-app: kube-controller-manager

when I first deployed prometheus-operator
kubectl port-forward myrelease-prometheus-prometheus-oper-prometheus-0 9090:9090
http://localhost:9090/targets
monitoring/myrelease-prometheus-oper-kube-controller-manager/0 (0/0)
monitoring/myrelease-prometheus-oper-kube-scheduler/0 (0/0)

kubectl get endpoints -n=kube-system
showed for kube-controller-manager and kube-scheduler, because the services that prometheus-operator generated to generate endpoints to help prometheus with service discovery of endpoints to scrape metrics from had pod label selectors that were incorrect for my cluster. No problem right? I just need to override the selectors to be accurate for my environment.
No problem = wrong, Not the case due to a bug/unexpected behavior.

Describe the bug:
When I try to override the generated service's label selector to be accurate for my environment, instead of replacing the default label selector (expected behavior), the default label selector gets appended to my corrected label selector.

kubeScheduler:
  enabled: true
  service:
    port: 10251
    targetPort: 10251
    selector:
      k8s-app: kube-scheduler #This should match "kubectl get pods --show-labels -n=kube-system" 

kubeControllerManager:
  enabled: true
  service:
    port: 10252
    targetPort: 10252
    selector: 
      k8s-app: kube-controller-manager   #This should match "kubectl get pods --show-labels -n=kube-system" (if it doesn't prometheus will show 0/0 endpoints)

Version of Helm and Kubernetes:
Helm version = 2.13.1
Kubernetes version = 1.12.5

Which chart:
stable/prometheus-operator
I deployed v5.0.3 and v5.2.0(latest to date) both had the problem.

What happened:
My override label selector was appended to the default label selector
service's in kube-system
myrelease-prometheus-oper-kube-scheduler
myrelease-prometheus-oper-kube-controller-manager
will end up having both labels (my override label selector + the default)
component: kube-controller-manager && k8s-app: kube-controller-manager
component: kube-scheduler && k8s-app: kube-scheduler
and as a result, will fail to select the correct pods/have no endpoints.

What you expected to happen:
My override label selector should have replaced the default label selector
The resulting generated services should have only had the labels
k8s-app: kube-scheduler
k8s-app: kube-controller-manager

How to reproduce it (as minimally and precisely as possible):
Use the override file I mentioned above.

Anything else we need to know:
The following override will result in the desired behavior of replacing the default label, but this is hella unintuitive/not documented to my knowledge, I just happened to stumble upon it while searching through 10 related issues.

kubeScheduler:
  enabled: true
  service:
    port: 10251
    targetPort: 10251
    selector:
      k8s-app: kube-scheduler #This should match "kubectl get pods --show-labels -n=kube-system" 
      component: null #This is needed to remove the default label

kubeControllerManager:
  enabled: true
  service:
    port: 10252
    targetPort: 10252
    selector: 
      k8s-app: kube-controller-manager   #This should match "kubectl get pods --show-labels -n=kube-system" (if it doesn't prometheus will show 0/0 endpoints)
      component: null #This is needed to remove the default label, otherwise it'll get appended. 

[jdomag]

Actually i think it's not template related, but more likely helm issue. I've fixed it by editing service manually via kubectl.. ;(

[lorantonodipo]

Thanks for writing down the workaround, saved me a few hours

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[gmichels]

To me the override only worked using this notation:

  kubeControllerManager:
    service:
      selector:
        k8s-app: kube-controller-manager
        component: ""

[jdomag]

@gmichels worked for me as well, thanks!

[s2504s]

@gmichels
I have tried your suggestion but it does not work.
the key component must be null without quotes

  kubeControllerManager:
    service:
      selector:
        k8s-app: kube-controller-manager
        component: null

[gmichels]

True, it no longer works for me using the empty quotes, but it works using null.

As already mentioned, this is probably related to some updates on the Helm client. I don't know which version I was using back in October, but I'd guess something in the 2.13.x releases, and now I am using 2.15.2 and it only works with the null format.

[s2504s]

Even better :)
I've updated my helm to the version v2.16.1 and looks like components is no longer needed and I removed this key

  kubeControllerManager:
    service:
      selector:
        k8s-app: kube-controller-manager