Cert-manager: http: TLS handshake error from 10.244.0.1:34746: remote error: tls: bad certificate

Created on 3 Dec 2019 · 8Comments · Source: jetstack/cert-manager

Describe the bug:
The following error is reported by the pod of the godaddy webhook when it is started

I1203 09:17:03.753396       1 secure_serving.go:116] Serving securely on [::]:443
I1203 09:17:12.680384       1 log.go:172] http: TLS handshake error from 10.244.0.1:34642: remote error: tls: bad certificate
I1203 09:17:22.492881       1 log.go:172] http: TLS handshake error from 10.244.0.1:34720: remote error: tls: bad certificate
I1203 09:17:24.033647       1 log.go:172] http: TLS handshake error from 10.244.0.1:34738: remote error: tls: bad certificate
I1203 09:17:25.333256       1 log.go:172] http: TLS handshake error from 10.244.0.1:34746: remote error: tls: bad certificate

When the API Server gets the request, it reports that x509: certificate signed by unknown authority

I1203 11:33:27.063391       1 controller.go:127] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
I1203 11:35:27.039257       1 controller.go:107] OpenAPI AggregationController: Processing item v1alpha1.acme.mycompany.com
E1203 11:35:27.045021       1 controller.go:114] loading OpenAPI spec for "v1alpha1.acme.mycompany.com" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: Error: 'x509: certificate signed by unknown authority'

Expected behaviour:
I expect that the pod of the godaddy-webhook will start without such error.

Question: Why the Trusted Certificate created by the cert-manager has been signed by an unknown authority then ?

Steps to reproduce the bug:

Install k8s 1.14
git clone github.com/inspectorioinc/cert-manager-webhook-godaddy && cd cert-manager-webhook-godaddy
Chnage the apiversion within the template/pki.yml file to use apiVersion: cert-manager.io/v1alpha2 instead of `apiVersion: certmanager.k8s.io/v1alpha1
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
helm init
... Wait a few moments till cert-manager is well installed
helm install --name godaddy-webhook --namespace cert-manager ./deploy/godaddy-webhook
kubectl logs -n cert-manager -lapp.kubernetes.io/instance=godaddy-webhook

Environment details::

Kubernetes version: 1.14.1
cert-manager version : 0.12.0
Install method: manifests

Additional info

Info about secret created and mounted to the pod of the godaddy webhook

kind: Secret
apiVersion: v1
metadata:
  name: godaddy-webhook-webhook-tls
  namespace: cert-manager
  selfLink: /api/v1/namespaces/cert-manager/secrets/godaddy-webhook-webhook-tls
  uid: a7e6dc5d-15ad-11ea-9e67-9600003838ca
  resourceVersion: '2304'
  creationTimestamp: '2019-12-03T09:16:57Z'
  annotations:
    cert-manager.io/alt-names: >-
      godaddy-webhook,godaddy-webhook.cert-manager,godaddy-webhook.cert-manager.svc
    cert-manager.io/certificate-name: godaddy-webhook-webhook-tls
    cert-manager.io/common-name: ''
    cert-manager.io/ip-sans: ''
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: godaddy-webhook-ca
    cert-manager.io/uri-sans: ''
data:
  ca.crt: >-
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURSRENDQWl5Z0F3SUJBZ0lRYW9yMVowQXFMUllNSUp5ZHpMNEIxVEFOQmdrcWhraUc5dzBCQVFzRkFEQkIKTVJVd0V3WURWUVFLRXd4alpYSjBMVzFoYm1GblpYSXhLREFtQmdOVkJBTVRIMk5oTG1kdlpHRmtaSGt0ZDJWaQphRzl2YXk1alpYSjBMVzFoYm1GblpYSXdIaGNOTVRreE1qQXpNRGt4TmpVM1doY05NalF4TWpBeE1Ea3hOalUzCldqQkJNUlV3RXdZRFZRUUtFd3hqWlhKMExXMWhibUZuWlhJeEtEQW1CZ05WQkFNVEgyTmhMbWR2WkdGa1pIa3QKZDJWaWFHOXZheTVqWlhKMExXMWhibUZuWlhJd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFSwpBb0lCQVFDd29KM3NoRitiajZjNmpMaXA3YmorZnZMNVl1ZFNiZEFHM1lYV3lBZUpHT1VNbnV3dXhOUWEvUGRWCi9hRUIzNHk3UmMzU21VY1RFNFhiLzYvT1BpVVNHR2Z4MkY3SFVYZEhFeVU1VzlyL2RockROY0JlRW5ldjh1bTYKRDdKb1B3RFZFVURkWEhLdCtGcStJYlVPb2xOYWtIWlNTWlpvRjdGemkxakNsc0JoRlVrZGpLNDRreUh1YzVxTgpzWnF1OE1iTVlOR3UybGkrak9PWWJKTHh2Rkw4aFppVVJyNFZaak40d3JMNEJRR1BONzIwZUlKMWhFajBWM0RiCm9zVnpYVFY4dkZvSjE4OTlDYVo0M3NWaXAvSXhrbi8vRUEyeE4xSnhxZ09CZGNWdjJKZkRsN0IrdG9tVFFxYi8Kc1EwZDR4TWl5ekgzcmlPWW52S0tNVVJucFBPWkFnTUJBQUdqT0RBMk1BNEdBMVVkRHdFQi93UUVBd0lDcERBVApCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCCkN3VUFBNElCQVFCZ1dUWVdzazBzVzd5LzREVjJXQ2QxdktCVFZxKzNLcTByUUNORnY4T0hoZjF4eTNPSlYxQVoKOFFWODBONS9QTm8vci9GNytnek0ydUsvVGNkN1FtV24wZkhxQlJXS0RjYUwrc2dIcjRvSFNxeVYxOXgyRDlLdQprSzRQRXpieUdkUHVDQUZuWlFHS1ZKdnA4b1RMU3VHSmUzK2ZjR2U4emZXV0lHVTUrclJtaFRJZE82K2wrYzg2Ck9YcVJDcTc2N1g3NkV4SkJPMFFFbDFYVk9kQ1hTVEhVK0NjZlA3dVlyS0VadU1xZk5tR3N6V2NRYS9DVWpqdDAKQWwwWXVxTEVZWUJVd2xFVWIwTXZCNlA4SklGWXVHTjI1d1Q1OFVxT2doZVU0V1B4Z0c3VnVjWjZhWlE5UXAraQpMWVV2V08xdFBVc29EQTlyYWttMTNOU1A1UDU0MjFLaQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.crt: >-
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURkakNDQWw2Z0F3SUJBZ0lSQUtYVHVzN0krWHpYN21QcVovaDdrR0l3RFFZSktvWklodmNOQVFFTEJRQXcKUVRFVk1CTUdBMVVFQ2hNTVkyVnlkQzF0WVc1aFoyVnlNU2d3SmdZRFZRUURFeDlqWVM1bmIyUmhaR1I1TFhkbApZbWh2YjJzdVkyVnlkQzF0WVc1aFoyVnlNQjRYRFRFNU1USXdNekE1TVRjd01sb1hEVEl3TVRJd01qQTVNVGN3Ck1sb3dGekVWTUJNR0ExVUVDaE1NWTJWeWRDMXRZVzVoWjJWeU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0MKQVE4QU1JSUJDZ0tDQVFFQTNhdjRvWUVjUzZla1BncnE4YUhsci83a3lZU2JJV0RRd1dqNzI3MHJnbUR5TGZvZwpuRElDN2ZNYTl1Mys4L1ZOTzRRUzUyM256MTY0UmxsZVY2d2orVFVnbG80RzN5RkFqN3JPVDRKb2ptU1dlODBICk9ML2xyd3JMWDZxZDBhdzJ0VFZCV2NLZnZCQkhpOHQyR1lkcU1yTm5lRG9OaTNTU093T3dOU0dJVnZrQmN6bFYKVmhmOHlhMUl6NmJuaUNaTlhJazQvdVRwU2VacUpod2xLMzlZbE9GZHU2amEvUTR1QmNuZGkwL1NHdXgzUjQ4eApqdVpVS3ZuU3BmazN4ZWpUYU02dlZIaU9aMk5LSlNqN09YdEJTNGkvS3M3K2tSUFBBRkNjOElzQlhIbmo1K3k1CkozcU5tWDBmSjNDSG8wc3VsTFE5bGVFOEFidnMvajBSYkRYSTV3SURBUUFCbzRHU01JR1BNQTRHQTFVZER3RUIKL3dRRUF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRBTUJnTlZIUk1CQWY4RUFqQUFNRm9HQTFVZApFUVJUTUZHQ0QyZHZaR0ZrWkhrdGQyVmlhRzl2YTRJY1oyOWtZV1JrZVMxM1pXSm9iMjlyTG1ObGNuUXRiV0Z1CllXZGxjb0lnWjI5a1lXUmtlUzEzWldKb2IyOXJMbU5sY25RdGJXRnVZV2RsY2k1emRtTXdEUVlKS29aSWh2Y04KQVFFTEJRQURnZ0VCQUJXU1JXWVBETVNkV3h0NXR6c1RqV0Zvb2hxc2toY2Z1SEVJeThTeklReFJEdFFNQUNZTAozcVJwd0VSQ1B2aVFHTFFKdndQOUxwaklVVUJuaUIzTGhtNlRlMnRYTytJcERvbUhYSlVrOFJoWkZKdEJKaDJvCjVCWTRkbFZ5bjVCY1JDcURrczlYTVM0TXljd04zL0tzZXZ6US84TEIxMTRoNStUcERSWHpMTUJCbjZLYkdySW4Ka0d5NlcxK2dvOWhieE42VlUrR3VXWUNoUTJPbDJYTWpVV1lXayt6WXdzUjQxVjdZZGJrNXRzdUdMRkZKMkJpawpIV3B4dlpydFJUcFQyTW5QTnM5dCtqR0lMU09OSDVUZFNOQkcwNkIyUHl0U2pXNnFEU0gvTnowbUdiTzdoMlFqCkx6NWg3WjdQQTlTZlh4amlrRmdJdUxTSVdSOGszY0RNRUNNPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: >-
    LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBM2F2NG9ZRWNTNmVrUGdycThhSGxyLzdreVlTYklXRFF3V2o3MjcwcmdtRHlMZm9nCm5ESUM3Zk1hOXUzKzgvVk5PNFFTNTIzbnoxNjRSbGxlVjZ3aitUVWdsbzRHM3lGQWo3ck9UNEpvam1TV2U4MEgKT0wvbHJ3ckxYNnFkMGF3MnRUVkJXY0tmdkJCSGk4dDJHWWRxTXJObmVEb05pM1NTT3dPd05TR0lWdmtCY3psVgpWaGY4eWExSXo2Ym5pQ1pOWElrNC91VHBTZVpxSmh3bEszOVlsT0ZkdTZqYS9RNHVCY25kaTAvU0d1eDNSNDh4Cmp1WlVLdm5TcGZrM3hlalRhTTZ2VkhpT1oyTktKU2o3T1h0QlM0aS9Lczcra1JQUEFGQ2M4SXNCWEhuajUreTUKSjNxTm1YMGZKM0NIbzBzdWxMUTlsZUU4QWJ2cy9qMFJiRFhJNXdJREFRQUJBb0lCQUMvWExaR3RpNjNJUUdvagpjUkUxSDUwY0pjUllHaUxhQWpPN3RKOTM5Q3Z2UUZVcjVsM1BiVkp0WlNSbHVPQnQwSDFIdzh5dXE1alZtRytXCldnWVI0TEdvQmFCNGhvTWhhZzJiQTJ1bGVXdEJ5amh0eDJFNENSbjkrbVpyTENSVVJucFZSQVZiTzU0YTVWWUcKVXphdlF2d3VPK1oyb1BXMzh0ZEp0SDNDL3VuRjdMTk11YlNFaTdNOTN5eWt3cHNDZmlXSG9ZYnhKMFdXbTNBSApVc0FyVVFULzFvbWtaZ2dLUWQ4dkl1eTRyd2l2YTAzNVUycG5URWg3S2pTV1k0c0ZCdzRva3drMlVPQ0d5YnRwCmFFcEdlZFVLYmR4M3Q0czUwcTBqaENYK2tYZ3ltT3M5b3V1RDNyY0ZicXdUTGtraSt1dngrblJlUk5EUUlvaW4KcXlSTEdnRUNnWUVBM2JvYkZlTTBFNk92LzZVUE9lQ0d1N2hVQ25EQ2hiRjJmbVJONjM4S09iR0RRUlZsT3JBSwpBQmM1V3pzR1NBYTllSnlTWWFiR0w0R0FsTEExU0tyTTJtVjhyOVMwU0tPRU85N3RLcGJpSkgvRU14Q0VlM1RsCjdHYU4zbXczc1BYQ2tGcUlwZjdERTQ2SFNHd1Y2bFNWa1NXWnR6Z2E5OThxbjlicjd6eElzS3NDZ1lFQS8rK3UKT3NyMVNFaVJPVmFLRk1FRDVGSGIzQXdQQk1iUGQ4ekNsMTI1SjBmWFBoSVZsanJFU2lRZTgxUnJ5Y08wYk1LbQpmUVFGcTFPZWpRWDR5RVprbjNpY3NQeXpsVWQvVGZIbnhCMnloYld1ZkZjQkxDM1IxU2RDYm1yS3grRnVDZ2p0CnFpa3NTWHZUa0ZWaVB2ZUU0OU1SdUtQODFUelV0MFpvaFpjeW9MVUNnWUFPa1hYS0lGMkJsTld4aEZUVnhxelQKbEs3ZzZMdS9NUVBLRmlrUWVnNWY4QXZoc05ndjNQVXJRRnRpWlNScWd4YmIzOTIyVU5EZ3VpK0tsWEZLenY0YQpzMlJYNFMvZlBlOG9SK05wVXZTVTFWZzh5cThKdFlxTjE4Y21nQkhBck1VVjJ2enpleU95RzBGRVB1WEp1YmtDCkNtaHBJMlNTT0xXQ1B3ck5jemFMZFFLQmdRQzhLa1JIVjJaa1lXM2I0UU1kNVJadWE3eWdHSUdHT29EWURFUFkKczRCb0g4cGhDVHl6dWJzTTQyZVo4S3hhcDdiajYyd3FKNlhmWTFnRFNLdXkyd1lwdEt4RzkxTTA0bDczT0czMApZdjBZancyY3Z2N09zMVU5U0dMditUWGpKSnNaWjUwTWFMNzF1OE04cjNuTzMzM2xYZzMrV1JxT3orMTRYWnlFCjh6ZTJyUUtCZ0FOb2ZybmNXTmRHeGNYVmhWN2R1QVYrdXBHcFM5OHdVUXhKcWt2L285Z1VPa09tRmtYN1p1TSsKR0dTb2Vna2JuQmRVTkR2UEV4UlRENERsZFBJN2ozaDRMVzhTdWJnNDFlamgyZ3hLT0dtZlpqOFZtK2NlN3cwZwoyQ21xc2tUWG1DYy9aVjU5TWtKQ241bXRNNy9oV1JDRjNzTHA5cXJyQ1lzTVJsSXlkYU94Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
type: kubernetes.io/tls

K8s ApiServer config

kind: Pod
apiVersion: v1
metadata:
  name: kube-apiserver-halkyon2
  namespace: kube-system
  labels:
    component: kube-apiserver
    tier: control-plane
  annotations:
    kubernetes.io/config.hash: fbe1e004bd8cab9068f3f4c63ffb6394
    kubernetes.io/config.mirror: fbe1e004bd8cab9068f3f4c63ffb6394
    kubernetes.io/config.seen: '2019-12-03T10:00:18.920677311+01:00'
    kubernetes.io/config.source: file
spec:
  volumes:
    - name: ca-certs
      hostPath:
        path: /etc/ssl/certs
        type: DirectoryOrCreate
    - name: etc-pki
      hostPath:
        path: /etc/pki
        type: DirectoryOrCreate
    - name: k8s-certs
      hostPath:
        path: /etc/kubernetes/pki
        type: DirectoryOrCreate
  containers:
    - name: kube-apiserver
      image: 'k8s.gcr.io/kube-apiserver:v1.14.1'
      command:
        - kube-apiserver
        - '--advertise-address=88.99.189.131'
        - '--allow-privileged=true'
        - '--authorization-mode=Node,RBAC'
        - '--client-ca-file=/etc/kubernetes/pki/ca.crt'
        - '--enable-admission-plugins=NodeRestriction'
        - '--enable-bootstrap-token-auth=true'
        - '--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt'
        - '--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt'
        - '--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key'
        - '--etcd-servers=https://127.0.0.1:2379'
        - '--insecure-port=0'
        - >-
          --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
        - '--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key'
        - '--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname'
        - '--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt'
        - '--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key'
        - '--requestheader-allowed-names=front-proxy-client'
        - '--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt'
        - '--requestheader-extra-headers-prefix=X-Remote-Extra-'
        - '--requestheader-group-headers=X-Remote-Group'
        - '--requestheader-username-headers=X-Remote-User'
        - '--secure-port=6443'
        - '--service-account-key-file=/etc/kubernetes/pki/sa.pub'
        - '--service-cluster-ip-range=10.96.0.0/12'
        - '--tls-cert-file=/etc/kubernetes/pki/apiserver.crt'
        - '--tls-private-key-file=/etc/kubernetes/pki/apiserver.key'
      resources:
        requests:
          cpu: 250m
      volumeMounts:
        - name: ca-certs
          readOnly: true
          mountPath: /etc/ssl/certs
        - name: etc-pki
          readOnly: true
          mountPath: /etc/pki
        - name: k8s-certs
          readOnly: true
          mountPath: /etc/kubernetes/pki

/kind bug

kinbug

Source

cmoulliard

Most helpful comment

You need to also update the 'injector' annotations to use the new API group:

https://github.com/inspectorioinc/cert-manager-webhook-godaddy/blob/f6e9c427e1a0f29e26a4931d93c007f0930872d9/deploy/godaddy-webhook/templates/apiservice.yaml#L8

Should be:

    cert-manager.io/inject-ca-from: "{{ .Release.Namespace }}/{{ include "godaddy-webhook.servingCertificate" . }}"

munnerz on 3 Dec 2019

👍3

All 8 comments

You need to also update the 'injector' annotations to use the new API group:

https://github.com/inspectorioinc/cert-manager-webhook-godaddy/blob/f6e9c427e1a0f29e26a4931d93c007f0930872d9/deploy/godaddy-webhook/templates/apiservice.yaml#L8

Should be:

    cert-manager.io/inject-ca-from: "{{ .Release.Namespace }}/{{ include "godaddy-webhook.servingCertificate" . }}"

munnerz on 3 Dec 2019

👍3

You need to also update the 'injector' annotations to use the new API group:

You saved a couple of precious hours ;-). Many thanks @munnerz
That works better

 kc logs -n cert-manager -lapp.kubernetes.io/instance=godaddy-webhook
I1203 16:32:07.439692       1 secure_serving.go:116] Serving securely on [::]:443

The log of the API Server reports such a message. Is it important ?

Trying to reach: 'https://10.101.56.140:443/openapi/v2', Header: map[]
I1203 16:30:11.686014       1 controller.go:127] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
I1203 16:32:07.403183       1 controller.go:107] OpenAPI AggregationController: Processing item v1alpha1.acme.mycompany.com
W1203 16:32:07.403261       1 handler_proxy.go:89] no RequestInfo found in the context
E1203 16:32:07.403311       1 controller.go:114] loading OpenAPI spec for "v1alpha1.acme.mycompany.com" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1203 16:32:07.403320       1 controller.go:127] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
I1203 16:32:13.847527       1 controller.go:107] OpenAPI AggregationController: Processing item v1alpha1.acme.mycompany.com
E1203 16:32:13.866137       1 controller.go:114] loading OpenAPI spec for "v1alpha1.acme.mycompany.com" failed with: OpenAPI spec does not exist
I1203 16:32:13.866186       1 controller.go:127] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
I1203 16:33:13.866736       1 controller.go:107] OpenAPI AggregationController: Processing item v1alpha1.acme.mycompany.com
E1203 16:33:13.871618       1 controller.go:114] loading OpenAPI spec for "v1alpha1.acme.mycompany.com" failed with: OpenAPI spec does not exist
I1203 16:33:13.871654       1 controller.go:127] OpenAPI AggregationController: action for item v1alpha1.acme.mycompany.com: Rate Limited Requeue.
I1203 16:34:11.669200       1 controller.go:107] OpenAPI AggregationController: Processing item v1alpha1.acme.mycompany.com

cmoulliard on 3 Dec 2019

I can process a certificate request

kc get certificate,certificaterequest,order,challenge -n godaddy                                
NAME                                      READY   SECRET            AGE
certificate.cert-manager.io/snowdrop-me   True    snowdrop-me-tls   2m8s

NAME                                                        READY   AGE
certificaterequest.cert-manager.io/snowdrop-me-2184923669   True    2m8s

NAME                                                           STATE   AGE
order.acme.cert-manager.io/snowdrop-me-2184923669-3161476526   valid   2m7s

cmoulliard on 3 Dec 2019

I have the same error message spamming the logs of my cert-manager-webhook container

"2020/01/16 19:03:20 http: TLS handshake error from 10.28.0.1:47956: remote error: tls: bad certificate

I'm lost how to debug this further. Neither the cert-manager container nor the cert-manager-cainjector container report any errors.

The certificate was successfully issued from 'https://acme-v02.api.letsencrypt.org/directory' using dns01 challenge from my GoogleCloud project.

I used cert-manager v0.10.0 for many months without any issues, but when the certificate would not renew it self 2 days ago I started investigating and found this error.

I reinstalled cert-manager using v0.13.0-alpha.0 and the certificate was reissued, but the error keeps logging every second or so.

Where can I find details about this error? What certificate is the bad one?

Thanks

lukas-alliado on 16 Jan 2020

👍2

@lukas-alliado I'm fairly sure that the issue is caused when the webhook tries to talk to the Kubernetes API. Presumably the API is serving up a bad cert. I've not seen a resolution to this issue yet.

Edit:

Looks like it uses mTLS and it's the apisservice (client) is providing an invalid client cert. Not sure how this could happen as it is automated.

lyndon160 on 30 Jan 2020

👍1

Have same issue. In webhook I see tls handshake error with an IP that is not currently in the cluster

SCLogo on 9 Mar 2020

I have the same:

Log from cert-manager-webhook pod:

I0326 11:19:51.463537 1 main.go:79] "msg"="enabling TLS as certificate file flags specified" | I0326 11:19:51.464154 1 server.go:131] "msg"="listening for insecure healthz connections" "address"=":6080" | I0326 11:19:51.464255 1 server.go:143] "msg"="listening for secure connections" "address"=":10250" | I0326 11:19:51.464291 1 server.go:165] "msg"="registered pprof handlers" | I0326 11:19:51.464626 1 tls_file_source.go:144] "msg"="detected private key or certificate data on disk has changed. reloading certificate" | 2020/03/26 11:19:54 http: TLS handshake error from 10.128.0.1:51630: remote error: tls: bad certificate | 2020/03/26 11:19:55 http: TLS handshake error from 10.128.0.1:51638: remote error: tls: bad certificate

Log from cert-manager pod:

I0326 11:21:18.568015 1 controller.go:144] cert-manager/controller/clusterissuers "msg"="finished processing work item" "key"="letsencrypt-prod" | I0326 11:21:44.433779 1 controller.go:138] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" | I0326 11:21:44.433944 1 controller.go:138] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca" | I0326 11:21:44.434309 1 controller.go:194] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="ca certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" | I0326 11:21:44.434324 1 controller.go:144] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca" | I0326 11:21:44.434407 1 controller.go:246] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="serving certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" | I0326 11:21:44.434429 1 controller.go:144] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-tls" | I0326 11:22:14.433893 1 controller.go:138] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" | I0326 11:22:14.434144 1 controller.go:138] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca" | I0326 11:22:14.434605 1 controller.go:194] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="ca certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" | I0326 11:22:14.434620 1 controller.go:144] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca" | I0326 11:22:14.434813 1 controller.go:246] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="serving certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" | I0326 11:22:14.434842 1 controller.go:144] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-tls"