Charts: [bitnami/prometheus-operator] telemetry port "already in use"

[tkam8]

Which chart:
prometheus-operator-0.19.1

Describe the bug
When starting kube-state-metrics, it wants to bind the telemetry port to 8081, but already in use by another service. Need a way to change this port.

I0607 07:29:54.237307       1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8081
2020/06/07 07:29:54 listen tcp 0.0.0.0:8081: bind: address already in use

To Reproduce
Steps to reproduce the behavior:

1. Install release: helm install prometheus-release1 --namespace monitoring bitnami/prometheus-operator
2. Issue kubectl get all -n monitoring , notice that the kube-state-metrics pod is in CrashLoopBackOff
3. Confirm pod logs

$ k logs prometheus-release1-kube-state-metrics-55bc7dc-7qbjl -n monitoring
        helm.sh/chart: kube-state-metrics-0.2.3
I0607 07:21:10.237270       1 main.go:89] Using collectors certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0607 07:21:10.237406       1 main.go:98] Using all namespace
I0607 07:21:10.237415       1 main.go:139] metric white-blacklisting: blacklisting the following items:
W0607 07:21:10.237448       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0607 07:21:10.242377       1 main.go:184] Testing communication with server
I0607 07:21:10.255251       1 main.go:189] Running with Kubernetes cluster version: v1.17. git version: v1.17.0. git tree state: clean. commit: 70132b0f130acc0bed193d9ba59dd186f0e634cf. platform: linux/amd64
I0607 07:21:10.255264       1 main.go:191] Communication with server successful
I0607 07:21:10.255360       1 main.go:225] Starting metrics server: 0.0.0.0:8080
I0607 07:21:10.255610       1 metrics_handler.go:96] Autosharding disabled
I0607 07:21:10.255979       1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0607 07:21:10.256362       1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8081
2020/06/07 07:21:10 listen tcp 0.0.0.0:8081: bind: address already in use

Expected behavior
A way to change the telemetry port

Version of Helm and Kubernetes:

• Output of helm version:

$ helm version
version.BuildInfo{Version:"v3.2.2", GitCommit:"a6ea66349ae3015618da4f547677a14b9ecc09b3", GitTreeState:"clean", GoVersion:"go1.13.12"}

• Output of kubectl version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Add any other context about the problem here.

[joancafom]

Hi @tkam8 ,

I have tried to reproduce your steps but I did not face the same issue. In my case, this is the logs that kube-state-metrics is providing:

I0608 08:08:30.050394       1 main.go:89] Using collectors certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0608 08:08:30.050467       1 main.go:98] Using all namespace
I0608 08:08:30.050473       1 main.go:139] metric white-blacklisting: blacklisting the following items:
W0608 08:08:30.050495       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0608 08:08:30.051962       1 main.go:184] Testing communication with server
I0608 08:08:30.056267       1 main.go:189] Running with Kubernetes cluster version: v1.18. git version: v1.18.1. git tree state: clean. commit: 7879fc12a63337efff607952a323df90cdc7a335. platform: linux/amd64
I0608 08:08:30.056282       1 main.go:191] Communication with server successful
I0608 08:08:30.056461       1 main.go:225] Starting metrics server: 0.0.0.0:8080
I0608 08:08:30.056738       1 metrics_handler.go:96] Autosharding disabled
I0608 08:08:30.056957       1 builder.go:146] Active collectors: certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses
I0608 08:08:30.057150       1 main.go:200] Starting kube-state-metrics self metrics server: 0.0.0.0:8081

It is worthwhile mentioning that I am using the default values.yml file. Just to try to bound the issue, are you proving a custom values.yml?

Thanks!

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[bluskool]

We had a similar issue with the kube-state-metrics pod. It was trying to bind to port 8080 on the node, but node-local-dns-cache was already bound to that port on every node. Setting hostNetwork: false in the kube-state-metrics values fixed the issue.

[joancafom]

Thanks for sharing your workaround!