Cert-manager: Error creating ClusterIssuer: the server is currently unable to handle the request

[petermikitsh]

Describe the bug:

I cannot create a ClusterIssuer on v0.7.0-alpha.1.

On a brand new Kubernetes cluster (on Digital Ocean):

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

kubectl create namespace cert-manager

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

helm install stable/cert-manager \
  --tls \
  --tls-ca-cert $(helm home)/ca.cert.pem \
  --tls-cert $(helm home)/helm.cert.pem \
  --tls-key $(helm home)/helm.key.pem \
  --version v0.6.6 \
  --namespace cert-manager \
  --name cert-manager \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer \
  --set "image.tag=v0.7.0-alpha.1" \
  --set "webhook.image.tag=v0.7.0-alpha.1"

cat <<EOF | kubectl create -f -
  {
    "apiVersion": "certmanager.k8s.io/v1alpha1",
    "kind": "ClusterIssuer",
    "metadata": {
      "name": "letsencrypt-prod",
      "namespace": "cert-manager"
    },
    "spec": {
      "acme": {
        "email": "REDACTED",
        "http01": {},
        "privateKeySecretRef": {
          "name": "letsencrypt-prod"
        },
        "server": "https://acme-v02.api.letsencrypt.org/directory"
      }
    }
  }
EOF

Output:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

Expected behaviour:

The ClusterIssue is created without error.

Steps to reproduce the bug:

See CLI steps above.

Anything else we need to know?:

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.13.1-do.2
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): DigitalOcean
• cert-manager version (e.g. v0.4.0): v0.7.0-alpha.1
• Install method (e.g. helm or static manifests): Helm

/kind bug

[petermikitsh]

More info:

$ kubectl get pods --namespace cert-manager
NAME                                    READY   STATUS             RESTARTS   AGE
cert-manager-744bc96879-6fg6g           1/1     Running            0          3m3s
cert-manager-webhook-86d8dfb849-vqjsr   0/1     CrashLoopBackOff   4          3m3s
cert-manager-webhook-ca-sync-q8vv7      0/1     Completed          0          3m3s

$ kubectl describe pods/cert-manager-webhook-86d8dfb849-vqjsr --namespace cert-manager
Name:               cert-manager-webhook-86d8dfb849-vqjsr
Namespace:          cert-manager
Priority:           0
PriorityClassName:  <none>
Node:               xenodochial-sanderson-uhic/10.136.37.236
Start Time:         Thu, 28 Feb 2019 14:33:54 -0800
Labels:             app=webhook
                    pod-template-hash=86d8dfb849
                    release=cert-manager
Annotations:        <none>
Status:             Running
IP:                 10.244.42.5
Controlled By:      ReplicaSet/cert-manager-webhook-86d8dfb849
Containers:
  webhook:
    Container ID:  docker://47b412d731297c8989bec5c8c43006f456b3038b8d1cd5c4347e6faf88ed615b
    Image:         quay.io/jetstack/cert-manager-webhook:v0.7.0-alpha.1
    Image ID:      docker-pullable://quay.io/jetstack/cert-manager-webhook@sha256:85931b8fd70ce90e00659339eba9a8ad842e7429c5e93472ecc0dca0a1208e5a
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=12
      --secure-port=6443
      --tls-cert-file=/certs/tls.crt
      --tls-private-key-file=/certs/tls.key
      --disable-admission-plugins=NamespaceLifecycle,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Initializers
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 28 Feb 2019 14:35:02 -0800
      Finished:     Thu, 28 Feb 2019 14:35:02 -0800
    Ready:          False
    Restart Count:  3
    Environment:
      POD_NAMESPACE:  cert-manager (v1:metadata.namespace)
    Mounts:
      /certs from certs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-webhook-token-9bdg7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-webhook-tls
    Optional:    false
  cert-manager-webhook-token-9bdg7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-manager-webhook-token-9bdg7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                                 Message
  ----     ------       ----               ----                                 -------
  Normal   Scheduled    88s                default-scheduler                    Successfully assigned cert-manager/cert-manager-webhook-86d8dfb849-vqjsr to xenodochial-sanderson-uhic
  Warning  FailedMount  80s (x5 over 87s)  kubelet, xenodochial-sanderson-uhic  MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-webhook-tls" not found
  Normal   Pulling      71s                kubelet, xenodochial-sanderson-uhic  pulling image "quay.io/jetstack/cert-manager-webhook:v0.7.0-alpha.1"
  Normal   Pulled       66s                kubelet, xenodochial-sanderson-uhic  Successfully pulled image "quay.io/jetstack/cert-manager-webhook:v0.7.0-alpha.1"
  Normal   Created      21s (x4 over 66s)  kubelet, xenodochial-sanderson-uhic  Created container
  Normal   Pulled       21s (x3 over 64s)  kubelet, xenodochial-sanderson-uhic  Container image "quay.io/jetstack/cert-manager-webhook:v0.7.0-alpha.1" already present on machine
  Normal   Started      20s (x4 over 66s)  kubelet, xenodochial-sanderson-uhic  Started container
  Warning  BackOff      8s (x6 over 62s)   kubelet, xenodochial-sanderson-uhic  Back-off restarting failed container

$ kubectl logs cert-manager-webhook-86d8dfb849-vqjsr --namespace=cert-manager -f
flag provided but not defined: -v
Usage of tls:
  -tls-cert-file string

Error: unknown flag: --disable-admission-plugins
Usage:
   [flags]

Flags:
      --alsologtostderr                                         log to standard error as well as files
      --audit-dynamic-configuration                             Enables dynamic audit configuration. This feature also requires the DynamicAuditing feature flag
      --audit-log-batch-buffer-size int                         The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-log-batch-max-size int                            The maximum size of a batch. Only used in batch mode. (default 1)
      --audit-log-batch-max-wait duration                       The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.
      --audit-log-batch-throttle-burst int                      Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
      --audit-log-batch-throttle-enable                         Whether batching throttling is enabled. Only used in batch mode.
      --audit-log-batch-throttle-qps float32                    Maximum average number of batches per second. Only used in batch mode.
      --audit-log-format string                                 Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json. (default "json")
      --audit-log-maxage int                                    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      --audit-log-maxbackup int                                 The maximum number of old audit log files to retain.
      --audit-log-maxsize int                                   The maximum size in megabytes of the audit log file before it gets rotated.
      --audit-log-mode string                                   Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "blocking")
      --audit-log-path string                                   If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      --audit-log-truncate-enabled                              Whether event and batch truncating is enabled.
      --audit-log-truncate-max-batch-size int                   Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      --audit-log-truncate-max-event-size int                   Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      --audit-log-version string                                API group and version used for serializing audit events written to log. (default "audit.k8s.io/v1")
      --audit-policy-file string                                Path to the file that defines the audit policy configuration.
      --audit-webhook-batch-buffer-size int                     The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-webhook-batch-max-size int                        The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-webhook-batch-max-wait duration                   The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      --audit-webhook-batch-throttle-burst int                  Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      --audit-webhook-batch-throttle-enable                     Whether batching throttling is enabled. Only used in batch mode. (default true)
      --audit-webhook-batch-throttle-qps float32                Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-webhook-config-file string                        Path to a kubeconfig formatted file that defines the audit webhook configuration.
      --audit-webhook-initial-backoff duration                  The amount of time to wait before retrying the first failed request. (default 10s)
      --audit-webhook-mode string                               Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "batch")
      --audit-webhook-truncate-enabled                          Whether event and batch truncating is enabled.
      --audit-webhook-truncate-max-batch-size int               Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      --audit-webhook-truncate-max-event-size int               Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      --audit-webhook-version string                            API group and version used for serializing audit events written to webhook. (default "audit.k8s.io/v1")
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authentication-tolerate-lookup-failure                  If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
      --authorization-always-allow-paths stringSlice            A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
      --enable-swagger-ui                                       Enables swagger ui on the apiserver at /swagger-ui
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. (default 1000)
      --kubeconfig string                                       kubeconfig file pointing at the 'core' kubernetes server.
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --log_file string                                         If non-empty, use this log file
      --logtostderr                                             log to standard error instead of files (default true)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names stringSlice                 List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      --requestheader-extra-headers-prefix stringSlice          List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers stringSlice                 List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers stringSlice              List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 443)
      --skip_headers                                            If true, avoid header prefixes in the log messages
      --stderrthreshold severity                                logs at or above this threshold go to stderr
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites stringSlice                           Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
      --tls-min-version string                                  Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
  -v, --v Level                                                 number for the log level verbosity (default 0)
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging

F0228 22:34:35.212372       1 cmd.go:42] unknown flag: --disable-admission-plugins

[munnerz]

You cannot use the v0.6 version of the chart with the v0.7 alpha releases - you'll need to either use a copy of the chart straight from the repository, or otherwise use the newly configured dedicated charts repo, where we publish alpha versions of Helm charts: https://hub.helm.sh/charts/jetstack/cert-manager/v0.7.0-alpha.1

We bumped the Kubernetes apimachinery dependencies in v0.7, which is why you're seeing this error 😄

[petermikitsh]

@munnerz Thank you for the suggestion. I edited my script use the new chart and the same error occurs:

kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

helm repo add jetstack https://charts.jetstack.io

kubectl create namespace cert-manager

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

helm install jetstack/cert-manager \
  --tls \
  --tls-ca-cert $(helm home)/ca.cert.pem \
  --tls-cert $(helm home)/helm.cert.pem \
  --tls-key $(helm home)/helm.key.pem \
  --version v0.7.0-alpha.1 \
  --namespace cert-manager \
  --name cert-manager \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer

cat <<EOF | kubectl create -f -
  {
    "apiVersion": "certmanager.k8s.io/v1alpha1",
    "kind": "ClusterIssuer",
    "metadata": {
      "name": "letsencrypt-prod",
      "namespace": "cert-manager"
    },
    "spec": {
      "acme": {
        "email": "REDACTED",
        "http01": {},
        "privateKeySecretRef": {
          "name": "letsencrypt-prod"
        },
        "server": "https://acme-v02.api.letsencrypt.org/directory"
      }
    }
  }
EOF

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

[scones]

what helped me:

do not create thge namespace before using it
do not label the namespace before using it

it's created and set up automagically for you and works then

[dbourcet]

I encountered the exact same issue, and also manage to fix it by not creating first the namespace (label it dit not help in any way). I can give full details if it helps.

[AndrewDryga]

If you ever had webhooks.enabled=true and changed it to false to workaround this issue then you need to manually delete a bunch of resources which left after you run helm del --purge cert-manager:

kubectl get APIService | grep "certmanager" | awk '{print $1;}' | xargs -I{} kubectl delete APIService {}
kubectl delete ClusterRole cert-manager-webhook-ca-sync
kubectl delete ConfigMap cert-manager-webhook-ca-sync --namespace cert-manager
kubectl get customresourcedefinitions | grep certmanager.k8s.io | awk '{print $1;}' | xargs -I{} kubectl delete customresourcedefinitions {}

And then add CRDS back and re-install cert-manager.

Also, you might need to kill cert-manager pod after you created your ClusterIssuer so that it takes into effect.

I must say that cert-manager went from easy to use in 0.5 to horrible developer experience in 0.6+. I would love to see any competitor project that keeps things simple and just works without dancing around it.

[rudolphjacksonm]

I must say that cert-manager went from easy to use in 0.5 to horrible developer experience in 0.6+. I would love to see any competitor project that keeps things simple and just works without dancing around it.

@AndrewDryga you're certainly not the only one who feels that way. It seems like after 0.5.0 everything broke down. Is there even a roadmap for this project anymore? I still haven't gotten an explanation as to why we have to install the CRDs separately to the helm chart. I've trie dthe steps you listed above and I still can't get 0.4.0 to install in our cluster after trying to rollback from 0.8.0.

[AndrewDryga]

@rudolphjacksonm you may also try to find if there is a leftover APIService once cert-manager is uninstalled, use kubectl get APIService and then delete if there is one.

[rudolphjacksonm]

@AndrewDryga thanks for the fast response on this! I've tried deleting the old APIService but still no luck. We can install cert-manager but applying the clusterIssuer fails: Error from server (InternalError): error when creating "cert-manager/cluster_issuer_internal.yaml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server could not find the requested resource

This is what that yaml file looks like, nothing unusual there:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: <secret name>

The only apiService that's installed as part of cert-manager is v1alpha1.certmanager.k8s.io--there's no sign of an admission apiService or anything in our cluster. I'm not sure what is left in the cluster that could be causing this, but it's insanely frustrating as you would think you could just run helm delete --purge and it would clean everything out. I've also ensured there are no CRDs left.

[AndrewDryga]

@rudolphjacksonm do you have certmanager.k8s.io/disable-validation=true label on namespace where you install cert-manager?

Try to do a completely clean run:

1. helm del --purge cert-manager
2. Remove leftovers:

kubectl get APIService | grep "certmanager" | awk '{print $1;}' | xargs -I{} kubectl delete APIService {}
kubectl delete ClusterRole cert-manager-webhook-ca-sync
kubectl delete ConfigMap cert-manager-webhook-ca-sync --namespace cert-manager
kubectl get customresourcedefinitions | grep certmanager.k8s.io | awk '{print $1;}' | xargs -I{} kubectl delete customresourcedefinitions {}

(I also manually checked resources in the namespace to make sure nothing is left.)

1. Make sure kubectl api-resources does not crash for you, if it does - it means something is left. Most likely - APIService.
2. Make sure the certmanager.k8s.io/disable-validation=true label is added to the namespace where you want to install cert-manager.
3. Check if you install valid CRDS (check the version in the URL, it should match the version you are installing). My version is v0.8.1 and command is kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml
4. Disable webhooks (I did not manage to make them work in my public cluster):

# values.yaml
webhook:
--  enabled: true
++  enabled: false

1. Install the chart to the labeled namespace via Helm.
2. Create ClusterIssuer
3. Make sure it works: kubectl get clusterissuer,issuer,certificates --all-namespaces

If it does not help, I don't know what to add :(.

[rudolphjacksonm]

@AndrewDryga thanks for all of that, that's really helpful.

We had that label on the namespace previously and it still didn't work, however this was before I tried removing the old apiservice, so it's possible that that needed to be re-added before we tried installing it again.

This was all being done on a dev cluster thankfully, so for the moment we've torn it down and are rebuilding it. If I still see the same behavior on a fresh cluster I'll run through all of those steps and let you know where I end up. Thanks again!