Charts: [stable/prometheus-operator] Missing tls-proxy-secret when admission webhooks are disabled

[mikemowgli]

Bug description

When setting prometheusOperator.amissionWebhooks.enabled: false, the prometheus operator pod cannot start since it tries to mount the tls-proxy-secret that does not exist. When the admissionWebhooks are enabled, the secret is created from the job in templates/prometheus-operator/admission-webhooks/job-patch/job-createSecret.yaml. However, there is no test on the admissionWebhooks.enabled value when the secret is mounted in the pod.
I am not sure whether the good solution would be to

• either create the tls-proxy-secret regardless of the admissionWebhooks value (what's the relationship between this secret and the admission webhooks?)
• or add a test in prometheus-operator/deployment.yaml to check against the value of prometheusOperator.tlsProxy.enabled and prometheusOperator.amissionWebhooks.enabled

The error message yielded is:
MountVolume.SetUp failed for volume "tls-proxy-secret" : secrets "prom-h3-prometheus-operato-admission" not found

Workaround

Unfortunately, enabling the admission webhooks is not an option for me since I am stuck with Kubernetes v1.11 which fails to create the webhooks configurations because:

1. there's a missing clientConfig.caBundle setting in the admission webhooks configuration
2. the caBundle flag is required in k8s 1.11 but optional in 1.16
   The only workaround for now is to disable the tls proxy using the value prometheusOperator.tlsProxy.enabled: false. My understanding is that the prometheus operator will communicate in plain HTTP with the prometheus and alertmanager services. Is it correct?

Version of Helm and Kubernetes:
Helm 3.0.0
Kubernetes v1.11

Which chart:
stable/prometheus-operator

[vsliouniaev]

You can achieve what you are looking for with

--set prometheusOperator.admissionWebhooks.enabled=false 
--set prometheusOperator.tlsProxy.enabled=false

[mikemowgli]

I know, but is it normal that there's no tls proxy when the admission webhooks are disabled? Why are these two binded?

[vsliouniaev]

Kubernetes requires the admission hooks to be served through TLS, but there wasn't a reason to have a proxy before this. You will need to provide some kind of secret in the cluster for TLS to be meaningful for this proxy, otherwise you may as well disable it. The two are separate so you can provide your own secret if you don't want to use the helm chart's provisioning mechanism

[mikemowgli]

Ok I thought the TLS proxy was also used for the operator to communicate with Prometheus and Alertmanager.
If the proxy is only useful for the admission webhooks, then I don't think we need anything more.
I'm closing this one.

[krichter722]

The output of kubectl describe pod of a for-ever pending monitoring-prometheus-oper-operator pod:

Name:               monitoring-prometheus-oper-operator-7658df644-znds4
Namespace:          monitoring
Priority:           0
PriorityClassName:  <none>
Node:               gke-cluster-942c677d-70-default-pool-65c7aac2-424n/10.128.0.5
Start Time:         Mon, 02 Dec 2019 20:34:44 +0100
Labels:             app=prometheus-operator-operator
                    chart=prometheus-operator-8.3.0
                    heritage=Helm
                    pod-template-hash=7658df644
                    release=monitoring
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      ReplicaSet/monitoring-prometheus-oper-operator-7658df644
Containers:
  prometheus-operator:
    Container ID:
    Image:         quay.io/coreos/prometheus-operator:v0.34.0
    Image ID:
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      --kubelet-service=kube-system/monitoring-prometheus-oper-kubelet
      --logtostderr=true
      --localhost=127.0.0.1
      --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.34.0
      --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
      --config-reloader-cpu=100m
      --config-reloader-memory=25Mi
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from monitoring-prometheus-oper-operator-token-7pfnr (ro)
  tls-proxy:
    Container ID:
    Image:         squareup/ghostunnel:v1.4.1
    Image ID:
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      server
      --listen=:8443
      --target=127.0.0.1:8080
      --key=cert/key
      --cert=cert/cert
      --disable-authentication
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /cert from tls-proxy-secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from monitoring-prometheus-oper-operator-token-7pfnr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  tls-proxy-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  monitoring-prometheus-oper-admission
    Optional:    false
  monitoring-prometheus-oper-operator-token-7pfnr:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  monitoring-prometheus-oper-operator-token-7pfnr
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                                                         Message
  ----     ------       ----               ----                                                         -------
  Normal   Scheduled    4m1s               default-scheduler                                            Successfully assigned monitoring/monitoring-prometheus-oper-operator-7658df644-znds4 to gke-cluster-942c677d-70-default-pool-65c7aac2-424n
  Warning  FailedMount  118s               kubelet, gke-cluster-942c677d-70-default-pool-65c7aac2-424n  Unable to mount volumes for pod "monitoring-prometheus-oper-operator-7658df644-znds4_monitoring(cb1f8b93-153a-11ea-bf81-42010a8000d2)": timeout expired waiting for volumes to attach or mount for pod "monitoring"/"monitoring-prometheus-oper-operator-7658df644-znds4". list of unmounted volumes=[tls-proxy-secret]. list of unattached volumes=[tls-proxy-secret monitoring-prometheus-oper-operator-token-7pfnr]
  Warning  FailedMount  113s (x9 over 4m)  kubelet, gke-cluster-942c677d-70-default-pool-65c7aac2-424n  MountVolume.SetUp failed for volume "tls-proxy-secret" : secrets "monitoring-prometheus-oper-admission" not found

The workaround to specify prometheusOperator.tlsProxy.enabled=false works in my case.