Cert-manager: Certificate creation failure (AKS) - certificate signed by unknown authority

[shybbko]

Describe the bug:
Certificates are not created. The following errors are reported:

the certificate:
Error from server: conversion webhook for &{map[apiVersion:cert-manager.io/v1alpha2 kind:Certificate metadata:map[creationTimestamp:2020-05-13T17:30:48Z generation:1 name:xxx-tls namespace:test ownerReferences:[map[apiVersion:extensions/v1beta1 blockOwnerDeletion:true controller:true kind:Ingress name:xxx-ingress uid:6d73b182-bbce-4834-aee2-414d2b3aa802]] uid:d40bc037-aef7-4139-868f-bd615a423b38] spec:map[dnsNames:[xxx.test.domain.com] issuerRef:map[group:cert-manager.io kind:ClusterIssuer name:letsencrypt-prod] secretName:xxx-tls] status:map[conditions:[map[lastTransitionTime:2020-05-13T18:55:31Z message:Waiting for CertificateRequest "xxx-tls-1403681706" to complete reason:InProgress status:False type:Ready]]]]} failed: Post https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: x509: certificate signed by unknown authority

cert-manager-webhook container:
cert-manager 2020/05/15 14:22:58 http: TLS handshake error from 10.20.0.19:35350: remote error: tls: bad certificate

Where 10.20.0.19 is the IP of tunnelfront pod.

Expected behaviour:
Certificates are created without issues.

Steps to reproduce the bug:

• deploy cert-manager using static manifests
• deploy certificate request

Anything else we need to know?:
Debugging with https://cert-manager.io/docs/faq/acme/ sort of "fails" when trying to kubectl describe order... as kubectl describe certificaterequest... returns CSR contents with error (as above), but not the order ID.

The questions would be:

• how to force cert-manager to debug (display more info) regarding the certificate it does not trust?
• maybe the problem is occuring regularly and there is a known solution?
• what steps should be undertaken to debug the issue further?

Environment details::

• Kubernetes version: 1.15.10
• Cloud-provider/provisioner : Azure (AKS)
• cert-manager version: 0.14.3
• Install method: static manifests (see below) + cluster issuer (see below) + regular (not legacy)

cluster issuer:

```apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: x
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
azuredns:
clientID: x
clientSecretSecretRef:
name: cert-manager-stage
key: CLIENT_SECRET
subscriptionID: x
tenantID: x
resourceGroupName: dns-stage
hostedZoneName: x

the manifest:
```global:
  imagePullSecrets: []
  isOpenshift: false

  priorityClassName: ""
  rbac:
    create: true

  podSecurityPolicy:
    enabled: false

  logLevel: 2

  leaderElection:
    namespace: "kube-system"

replicaCount: 1

strategy: {}


image:
  repository: quay.io/jetstack/cert-manager-controller
  pullPolicy: IfNotPresent

  tag: v0.14.3

clusterResourceNamespace: ""

serviceAccount:
  create: true
  name:
  annotations: {}

extraArgs: []

extraEnv: []

resources: {}

securityContext:
  enabled: false
  fsGroup: 1001
  runAsUser: 1001

podAnnotations: {}

podLabels: {}

nodeSelector: {}

ingressShim:
  defaultIssuerName: letsencrypt-prod
  defaultIssuerKind: ClusterIssuer

prometheus:
  enabled: true
  servicemonitor:
    enabled: false
    prometheusInstance: default
    targetPort: 9402
    path: /metrics
    interval: 60s
    scrapeTimeout: 30s
    labels: {}


affinity: {}

tolerations: []

webhook:
  enabled: true
  replicaCount: 1

  strategy: {}

  podAnnotations: {}

  extraArgs: []

  resources: {}

  nodeSelector: {}

  affinity: {}

  tolerations: []

  image:
    repository: quay.io/jetstack/cert-manager-webhook
    pullPolicy: IfNotPresent
    tag: v0.14.3

  injectAPIServerCA: true

  securePort: 10250

cainjector:
  replicaCount: 1

  strategy: {}

  podAnnotations: {}

  extraArgs: []

  resources: {}

  nodeSelector: {}

  affinity: {}

  tolerations: []

  image:
    repository: quay.io/jetstack/cert-manager-cainjector
    pullPolicy: IfNotPresent
    tag: v0.14.3

[joaoh821C]

Having basically same problem here. Any luck?

[shybbko]

Nope, nothing. It would help greatly if I knew which cert the cert-manager is having problems with - Lets Encrypt's, tunnelfront's or AKS's. Sadly, I have no information on how to debug the cert-manager, as the docs are rather scarce.

[joaoh821C]

After fighting with this issue for maybe 2 weeks. I finally decided to just go with cert-manager 0.15.0 and it worked smoothly. So, at least based on my experience, 0.14.3 is broken.

[tmart-ops]

I had this issue with 0.14.2 on AKS. cert-manager 0.15.0 is working for me. Thanks @joaoh821C

[meyskens]

v0.15 got a big improvement on the TLS handling of the webhook. Can you let us know if that fixes it for everyone here? Thanks!

[shybbko]

I can confirm that the problem does not occur for v0.15.0. I'll be closing the issue then.

[yves-vogl]

Thos occurs with v0.15.1 again.

[loomsen]

We have that problem both with versions 0.15.0 and 0.15.1. GKE cluster version v1.15.9-gke.24. @schybbkoh Please reopen.

[shybbko]

I think it might be better to create a separate ticket since it's GKE, not AKS, but sure, the ticket is reopened.

[meyskens]

@lenhattan86 your error seems to not be relayed to cert-manager but rather another webhook you have installed, mutate-app-pass-paypal-com-v1-juno.

[lenhattan86]

@meyskens Right, I found out my mistake.

[Lirt]

This issue can also occur, if you install cert-manager into different namespace than cert-manager and you don't install CRDs using helm, but apply CRDs directly.

Reason is that CRDs linked in official installation guide have some hardcoded values:

1. Namespace in which to look for webhook service.
   
   
   
   • You can search for *.spec.conversion.webhookClientConfig.service.namespace.
   
   
   • Or just ctrl+f namespace: 'cert-manager' in official CRD resource (linked above).
   
   
2. Annotation for injecting CA from secret:
   
   
   
   • Search for metadata.annotations, where you will find cert-manager.io/inject-ca-from-secret: 'cert-manager/cert-manager-webhook-ca'.
   
   
   • Or just ctrl+f cert-manager.io/inject-ca-from-secret:.
   
   

[hzhou97]

@yves-vogl @loomsen Are you still having issues with this?
/area webhook
/priority awaiting-more-evidence

[hzhou97]

/kind bug

[loomsen]

@hzhou97 for me it's fine now. I reinstalled cert-manager cleanly and the problem vanished for me. But please note, this was on GKE not AKS.

[ankitsoni17-png]

I am getting this same error on EKS server. I am using helm3 to install cert-manager and Rancher.
Cert-Manager: v0.16.1
Kubernetes: v1.15.11-eks-af3caf
kubectl: v1.18.6

[meyskens]

Going to close this old issue as it seems the original question is fixed. I recommend others to check in the Kubernetes slack or file a new issue with the exact details.

/close

[jetstack-bot]

@meyskens: Closing this issue.

In response to this:

Going to close this old issue as it seems the original question is fixed. I recommend others to check in the Kubernetes slack or file a new issue with the exact details.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.