Helm-charts: [prometheus-kube-stack] pod/kube-prometheus-stack-admission-create status NotReady error with Linkerd

[jjangga0214]

Describe the bug
After helm install ... on the cluster Linkerd is installed, pod/kube-prometheus-stack-admission-create-<randomAlphanumericString> is created and its status is always NotReady, no matter how much time I wait.

Version of Helm and Kubernetes:

Helm Version:

$ helm version
version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Kubernetes Version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-16T17:18:52Z", GoVersion:"go1.15.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Which chart: prometheus-kube-stack

Which version of the chart: 12.7.0

What happened:

I created a k8s cluster by kind, then installed Linkerd, then the chart. But something's stuck, and prometheus(and others) is not provisioned.

➜  capa-viewer git:(main) ✗ (☸ |kind-kind:default) kubectl get all
NAME                                               READY   STATUS     RESTARTS   AGE
pod/kube-prometheus-stack-admission-create-thfjk   1/2     NotReady   0          5m16s

NAME                                               COMPLETIONS   DURATION   AGE
job.batch/kube-prometheus-stack-admission-create   0/1           5m16s      5m16s

➜  capa-viewer git:(main) ✗ (☸ |kind-kind:default) kubectl logs pod/kube-prometheus-stack-admission-create-thfjk create                                         
W1225 03:05:22.273553       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"err":"secrets \"kube-prometheus-stack-admission\" not found","level":"info","msg":"no secret found","source":"k8s/k8s.go:106","time":"2020-12-25T03:05:22Z"}
{"level":"info","msg":"creating new secret","source":"cmd/create.go:23","time":"2020-12-25T03:05:22Z"}


➜  capa-viewer git:(main) ✗ (☸ |kind-kind:default) kubectl logs job.batch/kube-prometheus-stack-admission-create create 
W1225 03:05:22.273553       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"err":"secrets \"kube-prometheus-stack-admission\" not found","level":"info","msg":"no secret found","source":"k8s/k8s.go:106","time":"2020-12-25T03:05:22Z"}
{"level":"info","msg":"creating new secret","source":"cmd/create.go:23","time":"2020-12-25T03:05:22Z"}

➜  capa-viewer git:(main) ✗ (☸ |kind-kind:default) kubectl get secrets
NAME                                          TYPE                                  DATA   AGE
default-token-bsl8s                           kubernetes.io/service-account-token   3      100m
kube-prometheus-stack-admission               Opaque                                3      27m
kube-prometheus-stack-admission-token-6zn2p   kubernetes.io/service-account-token   3      6m48s
sh.helm.release.v1.kube-prometheus-stack.v1   helm.sh/release.v1                    1      6m49s

What you expected to happen: It's expected to 'just work'.

How to reproduce it (as minimally and precisely as possible):

$ kind create cluster
$ linkerd install | kubectl apply -f -
$ kubectl get all --watch -n linkerd # Wait until all of Linkerd's control plane gets ready
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: default
  annotations:
    linkerd.io/inject: enabled
EOF
$ helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack

Changed values of values.yaml (only put values which differ from the defaults):

Not changed.

The helm command that you execute and failing/misfunctioning:

For example:

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack

Helm values set after installation/upgrade:

Not changed.

[harrymitchinson]

Also experiencing this issue but managed to work around it by disabling Linkerd injection for the admission webhook while still keeping injection enabled for the namespace.

prometheusOperator:
  admissionWebhooks:
    patch:
      podAnnotations:
        linkerd.io/inject: disabled