Cert-manager: Cannot create Vault ClusterIssuer, Service Unavailable

[RonnyMaas]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
I'm able to create vault Issuers but not vault ClusterIssuers.
With exact same yaml config except for issuer name (vault-issuer or vault-cluster-issuer) and issuer kind (Issuer or ClusterIssuer).

Expected behaviour:
If I'm able to create a vault Issuer I should also be able to create a vault ClusterIssuer

Steps to reproduce the bug:
Steps to reproduce the bug should be clear and easily reproducible to help people
gain an understanding of the problem.

kubectl apply -f vault-issuer.yaml -n cert-manager
issuer.certmanager.k8s.io/vault-issuer created

```
kubectl apply -f vault-cluster-issuer.yaml
Error from server (InternalError): error when creating "vault-cluster-issuer.yaml": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": Post https://kubernetes.default.svc:443/apis/admission.certmanager.k8s.io/v1beta1/clusterissuers?timeout=30s: Service Unavailable

Or:

ubectl apply -f vault-cluster-issuer.yaml -n cert-manager
Error from server (InternalError): error when creating "vault-cluster-issuer.yaml": Internal error occurred: failed calling webhook "clusterissuers.admission.certmanager.k8s.io": Post https://kubernetes.default.svc:443/apis/admission.certmanager.k8s.io/v1beta1/clusterissuers?timeout=30s: Service Unavailable

**Anything else we need to know?**:
I also tried to create ClusterIssuer in kube-system with no luck

diff vault-issuer.yaml vault-cluster-issuer.yaml
2c2

< kind: Issuer

kind: ClusterIssuer
4c4

< name: vault-issuer

name: vault-cluster-issuer

kubectl get ns cert-manager --show-labels
NAME STATUS AGE LABELS
cert-manager Active 16h certmanager.k8s.io/disable-validation=true,name=cert-ma
```nager

Environment details::

• Kubernetes version (e.g. v1.10.2): v1,14,1 Kubespray
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): Baremetal
• cert-manager version (e.g. v0.4.0): v0.7.2
• Install method (e.g. helm or static manifests): helm

/kind bug

[RonnyMaas]

I also checked if al needed firewall ports are open on master and nodes like 443, 6443, 8443 and they are.

[RonnyMaas]

cat vault-issuer.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: vault-issuer
spec:
  vault:
    auth:
      tokenSecretRef:
        name: cert-manager-vault-token
        key: token
    path: pki/sign/lxhost
    server: https://<vault servername>:8200
    caBundle: <base64 encoded vault ca>

[RonnyMaas]

Except from error "Service Unavailable" as reponse on executing kubectl command. No errors or events can be found in pod logging or logging on cluster hosts. Also no events for webhook service.

[RonnyMaas]

Forgot to mention that I also applied crd's before helm install
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

[munnerz]

This suggests to me that the webhook component has not been deployed properly. You can see some information and instructions on debugging these sorts of issues here in our docs: https://docs.cert-manager.io/en/latest/getting-started/troubleshooting.html

More information on the webhook, as well as how it works and is deployed, can also be found in the docs 😄 https://docs.cert-manager.io/en/latest/getting-started/webhook.html

[RonnyMaas]

Have read both (troubleshooting and webhook and even more) and checked everything before even creating this issue.
Have tried different versions of cert-manager 6.x, 7.x, 8.beta. But as I said no error logs or events for pods or other cert-manager resources. Have seen dozens of howtos where everyone does exact the same thing as I tried to do. But I don't have a clue why issue works and clusterissue doesn't.

[dcardozoo]

I had the same problem and also checked the troubleshooting pages and tested all what I could.

I found on the webhook logs this error which may be also something:

k logs cert-manager-webhook-64999fc84c-7wgdj
flag provided but not defined: -v
Usage of tls:
  -tls-cert-file string

W0523 16:38:11.476643       1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
I0523 16:38:11.483511       1 secure_serving.go:116] Serving securely on [::]:6443

weird is that I can create issuers just on the cert-manager namespace, outside not. That's may be also why I can not create clusterIssuers at all.

[knlambert]

I have the exact same problem.

Everything works fine if it's something scoped to the cert-manager namespace.
Otherwise, I have this kind of exception :

failed calling webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

[munnerz]

W0523 16:38:11.476643 1 authentication.go:296] Cluster doesn't provide requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.

This indicates your cluster isn't configure to properly perform API aggregation. I've not used kubespray, but it seems like you may need to upgrade to a newer version, or verify that it properly sets up the API aggregation CA when Kubernetes is deployed.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[RonnyMaas]

This still doesn't work.

Don't know what to do, check or test. Haven't had any feedback/suggestions (that I didn't already tried or checkt ) that kan help me find out what is going on or how to fix the problem.

Only thing I see or know is that cluster issuer doesn't work. No helpfull errors in logging.

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[munnerz]

This is due to issues with the webhook being deployed - we've improved this mechanism in v0.11, so if you could uninstall and reinstall the latest release and re-open this issue if you still see the same problem, that'd be great 😄

[jmreicha]

Seeing pretty much this same behavior in a test Digital Ocean DOKS cluster that was recently upgraded to 1.16 and using the v0.11 cert-manager configs. Everything seemed to have been working correctly prior to upgrading.

I don't seem to be able to look at control plane logs or any of its components so not really sure how to troubleshoot this. Any ideas or suggestions?

I can open a new issue if needed.

[jmreicha]

Temporarily "fixed" my issue by setting hostNetwork: true in my spec. I'm almost positive the behavior I'm seeing is a DO issue. Sorry to hijack the thread.

[artpar]

Facing the same issue on my cluster deployed on DO. @jmreicha can you please elaborate where did you add that hostNetwork: true. in the certmanager's webhook deployment yaml or the certmanager itself ?

[jmreicha]

@artpar cert-manager-webhook deployment.