Cert-manager: Document/link to guide on how to use Istio + ACME HTTP01

Created on 18 Jan 2020  ·  20Comments  ·  Source: jetstack/cert-manager

Hi
I have some issues to create a cert for istio.

Describe the bug:
Here is what istio suggests to do:

TELEMETRY_DOMAIN=

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: telemetry-gw-cert
  namespace: istio-system
spec:
  secretName: telemetry-gw-cert
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
  commonName: $TELEMETRY_DOMAIN
  dnsNames:
  - $TELEMETRY_DOMAIN
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - $TELEMETRY_DOMAIN

here is the tutorial:
https://istio.io/docs/tasks/observability/gateways/#option-1-secure-access-https

I installed cert-manger v0.13.0 using helm: see: https://cert-manager.io/docs/installation/kubernetes/

$ kubectl get crd
NAME                                      CREATED AT
adapters.config.istio.io                  2019-12-16T18:35:41Z
alertmanagers.monitoring.coreos.com       2020-01-15T18:32:40Z
attributemanifests.config.istio.io        2019-12-16T18:35:39Z
authorizationpolicies.rbac.istio.io       2019-12-19T01:24:42Z
authorizationpolicies.security.istio.io   2019-12-16T18:35:39Z
certificaterequests.cert-manager.io       2020-01-21T19:15:26Z
certificates.cert-manager.io              2020-01-21T19:15:26Z
challenges.acme.cert-manager.io           2020-01-21T19:15:26Z
clusterissuers.cert-manager.io            2020-01-21T19:15:27Z
clusterrbacconfigs.rbac.istio.io          2019-12-16T18:35:39Z
destinationrules.networking.istio.io      2019-12-16T18:35:39Z
envoyfilters.networking.istio.io          2019-12-16T18:35:40Z
gateways.networking.istio.io              2019-12-16T18:35:40Z
handlers.config.istio.io                  2019-12-16T18:35:41Z
httpapispecbindings.config.istio.io       2019-12-16T18:35:40Z
httpapispecs.config.istio.io              2019-12-16T18:35:40Z
instances.config.istio.io                 2019-12-16T18:35:41Z
issuers.cert-manager.io                   2020-01-21T19:15:27Z
meshpolicies.authentication.istio.io      2019-12-16T18:35:40Z
orders.acme.cert-manager.io               2020-01-21T19:15:27Z
podmonitors.monitoring.coreos.com         2020-01-15T18:32:41Z
policies.authentication.istio.io          2019-12-16T18:35:40Z
prometheuses.monitoring.coreos.com        2020-01-15T18:32:42Z
prometheusrules.monitoring.coreos.com     2020-01-15T18:32:44Z
quotaspecbindings.config.istio.io         2019-12-16T18:35:40Z
quotaspecs.config.istio.io                2019-12-16T18:35:40Z
rbacconfigs.rbac.istio.io                 2019-12-16T18:35:40Z
rules.config.istio.io                     2019-12-16T18:35:41Z
serviceentries.networking.istio.io        2019-12-16T18:35:41Z
servicemonitors.monitoring.coreos.com     2020-01-15T18:32:45Z
servicerolebindings.rbac.istio.io         2019-12-16T18:35:41Z
serviceroles.rbac.istio.io                2019-12-16T18:35:41Z
sidecars.networking.istio.io              2019-12-16T18:35:39Z
templates.config.istio.io                 2019-12-16T18:35:41Z
virtualservices.networking.istio.io       2019-12-16T18:35:41Z

$ helm ls
NAME                REVISION    UPDATED                     STATUS      CHART                       APP VERSION NAMESPACE   
cert-manager        1           Tue Jan 21 14:16:53 2020    DEPLOYED    cert-manager-v0.13.0        v0.13.0     cert-manager
istio               27          Sun Jan 19 06:04:44 2020    DEPLOYED    istio-1.4.0                 1.4.0       istio-system
kube2iam            1           Mon Dec 16 16:36:50 2019    DEPLOYED    kube2iam-2.1.0              0.10.7      kube-system 
metrics-server      1           Mon Dec 16 14:55:56 2019    DEPLOYED    metrics-server-2.8.8        0.3.5       kube-system 
prometheus-operator 2           Wed Jan 15 13:34:30 2020    DEPLOYED    prometheus-operator-8.5.9   0.34.0      monitoring

here is my clusterissuer and cert definition (my dnsNames was replaced with test.domain.com). The dnsName is in the AWS Route53 under CNAME entry, which points to the aws lbs.

cat <<EOF >./cert.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  annotations:
  name: letsencrypt-staging
spec:
  acme:
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: telemetry-gw-cert
  namespace: istio-system
spec:
  secretName: telemetry-gw-cert
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: test.domain.com
  dnsNames:
  - test.domain.com
EOF

Expected behaviour:
it should create the cert without an error

Steps to reproduce the bug:

kubectl apply -f ./cert.yaml

some debug:

$ kubectl get clusterissuer letsencrypt-staging
NAME                  READY   AGE
letsencrypt-staging   True    161m

$ kubectl get certificaterequest -n istio-system
NAME                           READY   AGE
telemetry-gw-cert-2903184431   False   159m

$ kubectl describe certificaterequest -n istio-system
Name:         telemetry-gw-cert-2903184431
Namespace:    istio-system
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: telemetry-gw-cert
              cert-manager.io/private-key-secret-name: telemetry-gw-cert
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"telemetry-gw-cert","namespace":"istio-s...
API Version:  cert-manager.io/v1alpha2
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2020-01-21T19:19:50Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  telemetry-gw-cert
    UID:                   405d640e-26ea-4389-9dea-05a950c876ee
  Resource Version:        34498318
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/istio-system/certificaterequests/telemetry-gw-cert-2903184431
  UID:                     93e01ceb-4fd8-479d-93e5-00c432f24f9e
Spec:
  Csr:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3N6Q0NBWnNDQVFBd09URVZNQk1HQTFVRUNoTU1ZMlZ5ZEMxdFlXNWhaMlZ5TVNBd0hnWURWUVFERXhkagpiWFl0ZFhNdFpXRnpkQzB5TG05d2RHOHVabUZwYkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQUtNUmJzdW5NVWRBelJaczR2Q1JsaG5DUmxqVHBLZVJxbmsrQi8xUm9GS2ZuWmVuSnpLVTZSeFYKMmJLVWRhelZKZFV3SDdocm5Wa3pBZ2NDWlc4TGpwQ3NHd3pxbDluUUM2V0FiaHR0cVFRYUducEF1Nkw3WktzLwprbUE1VkliVUFWZHlhRVZYNVFwYTU1T3lrKzFyaTVVcEhPSWowMVNGZ0JsK2Ric1pzSlF1UCtSSXFjNDlBRVBlCnNKOEVpZVloNE5uWGg0NjlOODRuWmVVdm11TExxald1d1A3VGphYnJpV2lpaEJJTXVTQmEyQUsrUWRYUDJBdGgKV281ME1NNHkyWjF1SjllNmRnM1AzcXV4NUs5aTk5SnU2MHZFYTUwaysvZ3Y0UWFTTnFZYStaKzNkRVZUQVNzTgpjbnZXdlJyQ0xvMTl0L2J1emJCVUhjZGpweWhwaWRzQ0F3RUFBYUExTURNR0NTcUdTSWIzRFFFSkRqRW1NQ1F3CklnWURWUjBSQkJzd0dZSVhZMjEyTFhWekxXVmhjM1F0TWk1dmNIUnZMbVpoYVd3d0RRWUpLb1pJaHZjTkFRRUwKQlFBRGdnRUJBRU1ZM3FFYjhENEFQZFlaNnhubmR5Qk1vSVlDNTZOWkp3QkZjVStHRms4Q2oyUDJrKzZaYVhDSwpFazduaXFwTHUyZGU4Sk1VR2NKZkgza05LRHVCUHhLeEIzK1BwN09ickRocjIzdjlVd1dMU25PTmJHN0pxR0UzCkNaU3p6UngxM3VWRFlreTh5QmdMTzl3N1RDMm50YTc2b2xCeWFpeGxDSlM0WXFoMnlFSlRzcWNERVlnNVZZZ0YKWHZFRFJMR3dEVTJUeHl2MWZTZXZhOXBQai8xSmJ4VnQ3RzhUSnc2RWRhMWFFaEExUC9RR2NFS3Q0aEU2WDhmSAorN0ZvL2hNWnFETWI5LzdPZDQzT1VjM3RaZ3dqNGgzcGg2MkNRb2RPdjVjekRSSXRHVUowS2ZHbjdTVTNpeWliCnNEVkJINlF6VzdhTnBhUlhyUUtMbDBHZHkwSkZSWEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-staging
Status:
  Conditions:
    Last Transition Time:  2020-01-21T19:19:50Z
    Message:               Waiting on certificate issuance from order istio-system/telemetry-gw-cert-2903184431-2058801590: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    <none>

$ kubectl get order -n istio-system
NAME                                      STATE     AGE
telemetry-gw-cert-2903184431-2058801590   pending   163m

$ kubectl get challenge -n istio-system
NAME                                                 STATE     DOMAIN                    AGE
telemetry-gw-cert-2903184431-2058801590-3026613874   pending   test.domain.com   163m

$ kubectl describe order -n istio-system
Name:         telemetry-gw-cert-2903184431-2058801590
Namespace:    istio-system
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: telemetry-gw-cert
              cert-manager.io/private-key-secret-name: telemetry-gw-cert
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cert-manager.io/v1alpha2","kind":"Certificate","metadata":{"annotations":{},"name":"telemetry-gw-cert","namespace":"istio-s...
API Version:  acme.cert-manager.io/v1alpha2
Kind:         Order
Metadata:
  Creation Timestamp:  2020-01-21T19:19:50Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  telemetry-gw-cert-2903184431
    UID:                   93e01ceb-4fd8-479d-93e5-00c432f24f9e
  Resource Version:        34498319
  Self Link:               /apis/acme.cert-manager.io/v1alpha2/namespaces/istio-system/orders/telemetry-gw-cert-2903184431-2058801590
  UID:                     2a1cc84c-59d8-44ba-99c5-37555c38437c
Spec:
  Common Name:  test.domain.com
  Csr:          LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3N6Q0NBWnNDQVFBd09URVZNQk1HQTFVRUNoTU1ZMlZ5ZEMxdFlXNWhaMlZ5TVNBd0hnWURWUVFERXhkagpiWFl0ZFhNdFpXRnpkQzB5TG05d2RHOHVabUZwYkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDCkFRb0NnZ0VCQUtNUmJzdW5NVWRBelJaczR2Q1JsaG5DUmxqVHBLZVJxbmsrQi8xUm9GS2ZuWmVuSnpLVTZSeFYKMmJLVWRhelZKZFV3SDdocm5Wa3pBZ2NDWlc4TGpwQ3NHd3pxbDluUUM2V0FiaHR0cVFRYUducEF1Nkw3WktzLwprbUE1VkliVUFWZHlhRVZYNVFwYTU1T3lrKzFyaTVVcEhPSWowMVNGZ0JsK2Ric1pzSlF1UCtSSXFjNDlBRVBlCnNKOEVpZVloNE5uWGg0NjlOODRuWmVVdm11TExxald1d1A3VGphYnJpV2lpaEJJTXVTQmEyQUsrUWRYUDJBdGgKV281ME1NNHkyWjF1SjllNmRnM1AzcXV4NUs5aTk5SnU2MHZFYTUwaysvZ3Y0UWFTTnFZYStaKzNkRVZUQVNzTgpjbnZXdlJyQ0xvMTl0L2J1emJCVUhjZGpweWhwaWRzQ0F3RUFBYUExTURNR0NTcUdTSWIzRFFFSkRqRW1NQ1F3CklnWURWUjBSQkJzd0dZSVhZMjEyTFhWekxXVmhjM1F0TWk1dmNIUnZMbVpoYVd3d0RRWUpLb1pJaHZjTkFRRUwKQlFBRGdnRUJBRU1ZM3FFYjhENEFQZFlaNnhubmR5Qk1vSVlDNTZOWkp3QkZjVStHRms4Q2oyUDJrKzZaYVhDSwpFazduaXFwTHUyZGU4Sk1VR2NKZkgza05LRHVCUHhLeEIzK1BwN09ickRocjIzdjlVd1dMU25PTmJHN0pxR0UzCkNaU3p6UngxM3VWRFlreTh5QmdMTzl3N1RDMm50YTc2b2xCeWFpeGxDSlM0WXFoMnlFSlRzcWNERVlnNVZZZ0YKWHZFRFJMR3dEVTJUeHl2MWZTZXZhOXBQai8xSmJ4VnQ3RzhUSnc2RWRhMWFFaEExUC9RR2NFS3Q0aEU2WDhmSAorN0ZvL2hNWnFETWI5LzdPZDQzT1VjM3RaZ3dqNGgzcGg2MkNRb2RPdjVjekRSSXRHVUowS2ZHbjdTVTNpeWliCnNEVkJINlF6VzdhTnBhUlhyUUtMbDBHZHkwSkZSWEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  Dns Names:
    test.domain.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-staging
Status:
  Authorizations:
    Challenges:
      Token:     cAZ1JaJq64scmW-qNDngbT6qlgdetF9zRhqSP5PUqnY
      Type:      http-01
      URL:       https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/34173287/WaVLxg
      Token:     cAZ1JaJq64scmW-qNDngbT6qlgdetF9zRhqSP5PUqnY
      Type:      dns-01
      URL:       https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/34173287/Nlf8rA
      Token:     cAZ1JaJq64scmW-qNDngbT6qlgdetF9zRhqSP5PUqnY
      Type:      tls-alpn-01
      URL:       https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/34173287/wMDdeg
    Identifier:  test.domain.com
    URL:         https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/34173287
    Wildcard:    false
  Finalize URL:  https://acme-staging-v02.api.letsencrypt.org/acme/finalize/12173117/71282728
  State:         pending
  URL:           https://acme-staging-v02.api.letsencrypt.org/acme/order/12173117/71282728
Events:          <none>

$ kubectl describe challenge -n istio-system
Name:         telemetry-gw-cert-2903184431-2058801590-3026613874
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1alpha2
Kind:         Challenge
Metadata:
  Creation Timestamp:  2020-01-21T19:19:51Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  telemetry-gw-cert-2903184431-2058801590
    UID:                   2a1cc84c-59d8-44ba-99c5-37555c38437c
  Resource Version:        34498341
  Self Link:               /apis/acme.cert-manager.io/v1alpha2/namespaces/istio-system/challenges/telemetry-gw-cert-2903184431-2058801590-3026613874
  UID:                     30d17ddd-486c-442a-a543-7c9f83f7e569
Spec:
  Authz URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/34173287
  Dns Name:   test.domain.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-staging
  Key:     cAZ1JaJq64scmW-qNDngbT6qlgdetF9zRhqSP5PUqnY.-uj5cYrN4sofy908xPW_mXZcYvTeJOYmkrvLFtqD4v4
  Solver:
    Http 01:
      Ingress:
        Class:  istio
  Token:        cAZ1JaJq64scmW-qNDngbT6qlgdetF9zRhqSP5PUqnY
  Type:         http-01
  URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/34173287/WaVLxg
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:        <none>

here the error log:

$ kubectl logs cert-manager-5fbd5d7bcd-nxckw -n cert-manager
I0121 19:16:56.118107       1 start.go:76] cert-manager "msg"="starting controller"  "git-commit"="6d9200f9d" "version"="v0.13.0"
W0121 19:16:56.118162       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0121 19:16:56.118972       1 controller.go:167] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["1.1.1.1:53","8.8.8.8:53"] 
I0121 19:16:56.119204       1 controller.go:130] cert-manager/controller "msg"="starting leader election"  
I0121 19:16:56.119391       1 metrics.go:202] cert-manager/metrics "msg"="listening for connections on" "address"="0.0.0.0:9402" 
I0121 19:16:56.119838       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-controller...
I0121 19:18:16.838963       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-controller
I0121 19:18:16.839269       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I0121 19:18:16.839285       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I0121 19:18:16.839294       1 controller.go:74] cert-manager/controller/clusterissuers "msg"="starting control loop"  
I0121 19:18:16.839308       1 controller.go:74] cert-manager/controller/ingress-shim "msg"="starting control loop"  
I0121 19:18:16.839311       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I0121 19:18:16.839345       1 controller.go:74] cert-manager/controller/issuers "msg"="starting control loop"  
I0121 19:18:16.839369       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="vault"
I0121 19:18:16.839451       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="ca"
I0121 19:18:16.839486       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="selfsigned"
I0121 19:18:16.839504       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I0121 19:18:16.839525       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="venafi"
I0121 19:18:16.839529       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I0121 19:18:16.839546       1 controller.go:74] cert-manager/controller/certificaterequests-issuer-ca "msg"="starting control loop"  
I0121 19:18:16.839565       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I0121 19:18:16.839594       1 controller.go:74] cert-manager/controller/certificaterequests-issuer-vault "msg"="starting control loop"  
I0121 19:18:16.839554       1 controller.go:74] cert-manager/controller/orders "msg"="starting control loop"  
I0121 19:18:16.839644       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I0121 19:18:16.839680       1 controller.go:74] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="starting control loop"  
I0121 19:18:16.839701       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I0121 19:18:16.839725       1 controller.go:74] cert-manager/controller/certificaterequests-issuer-venafi "msg"="starting control loop"  
I0121 19:18:16.839711       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificates" 
I0121 19:18:16.839745       1 controller.go:74] cert-manager/controller/certificates "msg"="starting control loop"  
I0121 19:18:16.839807       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="webhook-bootstrap" 
I0121 19:18:16.839826       1 controller.go:74] cert-manager/controller/webhook-bootstrap "msg"="starting control loop"  
I0121 19:18:16.940541       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="acme"
I0121 19:18:16.940562       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I0121 19:18:16.940576       1 controller.go:74] cert-manager/controller/challenges "msg"="starting control loop"  
I0121 19:18:16.940598       1 controller.go:101] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I0121 19:18:16.940618       1 controller.go:74] cert-manager/controller/certificaterequests-issuer-acme "msg"="starting control loop"  
I0121 19:18:17.039537       1 controller.go:129] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="monitoring/prometheus-operator-prometheus" 
I0121 19:18:17.039623       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="prometheus-operator-prometheus" "resource_namespace"="monitoring" 
I0121 19:18:17.039642       1 controller.go:135] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="monitoring/prometheus-operator-prometheus" 
I0121 19:18:17.040000       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-token-c2wd5" 
I0121 19:18:17.040011       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/istio.cert-manager-webhook" 
I0121 19:18:17.040008       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/istio.cert-manager" 
I0121 19:18:17.040022       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/default-token-xcfqs" 
I0121 19:18:17.040027       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-token-c2wd5" 
I0121 19:18:17.040032       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/istio.cert-manager" 
I0121 19:18:17.040032       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/istio.cert-manager-webhook" 
I0121 19:18:17.040031       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-cainjector-token-hvqtg" 
I0121 19:18:17.040041       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/default-token-xcfqs" 
I0121 19:18:17.040046       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/istio.default" 
I0121 19:18:17.040048       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/istio.cert-manager-cainjector" 
I0121 19:18:17.040044       1 controller.go:411] cert-manager/controller/webhook-bootstrap "msg"="existing Secret does not exist, creating new empty secret" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" 
I0121 19:18:17.040054       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-cainjector-token-hvqtg" 
I0121 19:18:17.040061       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/istio.default" 
I0121 19:18:17.040063       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/istio.cert-manager-cainjector" 
I0121 19:18:17.040043       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-token-8zb2s" 
I0121 19:18:17.040075       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-token-8zb2s" 
I0121 19:18:17.044793       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:17.044861       1 controller.go:190] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="regenerating new private key" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" 
I0121 19:18:17.045062       1 controller.go:411] cert-manager/controller/webhook-bootstrap "msg"="existing Secret does not exist, creating new empty secret" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
I0121 19:18:17.047433       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" 
E0121 19:18:17.047502       1 controller.go:230] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="error decoding CA private key" "error"="error decoding private key PEM block" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
E0121 19:18:17.047522       1 controller.go:131] cert-manager/controller/webhook-bootstrap "msg"="re-queuing item  due to error processing" "error"="error decoding private key PEM block" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:17.247859       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:17.248663       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:17.248785       1 controller.go:302] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="re-issuing webhook certificate" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" 
I0121 19:18:17.253104       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:17.253852       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:17.254193       1 controller.go:197] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="ca certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager" 
I0121 19:18:17.254214       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca" 
I0121 19:18:22.047677       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:22.048032       1 controller.go:249] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="regenerating new private key" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
I0121 19:18:22.398195       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:22.398465       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:22.398762       1 controller.go:302] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="re-issuing webhook certificate" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
I0121 19:18:22.404382       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:22.405231       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:18:22.405566       1 controller.go:255] cert-manager/controller/webhook-bootstrap/webhook-bootstrap/ca-secret "msg"="serving certificate already up to date" "resource_kind"="Secret" "resource_name"="cert-manager-webhook-tls" "resource_namespace"="cert-manager" 
I0121 19:18:22.405580       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-tls" 
I0121 19:19:41.493726       1 controller.go:129] cert-manager/controller/clusterissuers "msg"="syncing item" "key"="letsencrypt-staging" 
I0121 19:19:41.493974       1 setup.go:87] cert-manager/controller/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0121 19:19:41.760339       1 controller.go:129] cert-manager/controller/webhook-bootstrap "msg"="syncing item" "key"="cert-manager/letsencrypt-staging" 
I0121 19:19:41.760367       1 controller.go:135] cert-manager/controller/webhook-bootstrap "msg"="finished processing work item" "key"="cert-manager/letsencrypt-staging" 
I0121 19:19:41.761141       1 setup.go:168] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0121 19:19:41.761154       1 logger.go:90] Calling GetAccount
I0121 19:19:42.336604       1 logger.go:85] Calling CreateAccount
I0121 19:19:42.436134       1 setup.go:260] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0121 19:19:42.436150       1 conditions.go:92] Setting lastTransitionTime for Issuer "letsencrypt-staging" condition "Ready" to 2020-01-21 19:19:42.436146331 +0000 UTC m=+166.329230936
I0121 19:19:42.443177       1 controller.go:135] cert-manager/controller/clusterissuers "msg"="finished processing work item" "key"="letsencrypt-staging" 
I0121 19:19:42.443910       1 controller.go:129] cert-manager/controller/clusterissuers "msg"="syncing item" "key"="letsencrypt-staging" 
I0121 19:19:42.444223       1 setup.go:162] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0121 19:19:42.444244       1 controller.go:135] cert-manager/controller/clusterissuers "msg"="finished processing work item" "key"="letsencrypt-staging" 
I0121 19:19:46.760276       1 controller.go:129] cert-manager/controller/clusterissuers "msg"="syncing item" "key"="letsencrypt-staging" 
I0121 19:19:46.760620       1 setup.go:162] cert-manager/controller/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" 
I0121 19:19:46.760644       1 controller.go:135] cert-manager/controller/clusterissuers "msg"="finished processing work item" "key"="letsencrypt-staging" 
I0121 19:19:50.422182       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.422470       1 sync.go:367] cert-manager/controller/certificates "msg"="no existing CertificateRequest resource exists, creating new request..." "related_resource_kind"="Secret" "related_resource_name"="telemetry-gw-cert" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:50.430583       1 sync.go:379] cert-manager/controller/certificates "msg"="created certificate request" "related_resource_kind"="Secret" "related_resource_name"="telemetry-gw-cert" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "request_name"="telemetry-gw-cert-2903184431"
I0121 19:19:50.430677       1 conditions.go:155] Setting lastTransitionTime for Certificate "telemetry-gw-cert" condition "Ready" to 2020-01-21 19:19:50.430673042 +0000 UTC m=+174.323757655
I0121 19:19:50.431104       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431163       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431168       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431184       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431202       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431208       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431215       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431232       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.431281       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.444634       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.444656       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.444823       1 sync.go:385] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:50.444890       1 sync.go:497] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "state"=""
I0121 19:19:50.446544       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:50.446689       1 sync.go:77] cert-manager/controller/orders "msg"="Creating new ACME order as status.url is not set" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:50.446698       1 conditions.go:200] Setting lastTransitionTime for CertificateRequest "telemetry-gw-cert-2903184431" condition "Ready" to 2020-01-21 19:19:50.446694169 +0000 UTC m=+174.339778774
I0121 19:19:50.446714       1 sync.go:190] cert-manager/controller/orders "msg"="order URL not set, submitting Order to ACME server" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:50.446737       1 sync.go:196] cert-manager/controller/orders "msg"="build set of domains for Order" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" "domains"=["test.domain.com"]
I0121 19:19:50.446746       1 logger.go:40] Calling CreateOrder
I0121 19:19:50.456332       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.456356       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.456528       1 sync.go:385] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:50.456600       1 sync.go:497] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "state"=""
I0121 19:19:50.456709       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.457161       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457223       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457233       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457234       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457249       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457263       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.457307       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457331       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457339       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457448       1 sync.go:385] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:50.457549       1 sync.go:497] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "state"="Pending"
I0121 19:19:50.457659       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.457793       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.457813       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.458033       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="telemetry-gw-cert-2903184431-2058801590" "related_resource_namespace"="istio-system" "resource_kind"="CertificateRequest" "resource_name"="telemetry-gw-cert-2903184431" "resource_namespace"="istio-system" 
I0121 19:19:50.464197       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464263       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464273       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464289       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464297       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464309       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464337       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464339       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:50.464357       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464374       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464374       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464536       1 sync.go:385] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:50.464590       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="telemetry-gw-cert-2903184431-2058801590" "related_resource_namespace"="istio-system" "resource_kind"="CertificateRequest" "resource_name"="telemetry-gw-cert-2903184431" "resource_namespace"="istio-system" 
I0121 19:19:50.464622       1 sync.go:497] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "state"="Pending"
I0121 19:19:50.464642       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:50.464728       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:51.114189       1 sync.go:211] cert-manager/controller/orders "msg"="submitted Order to ACME server" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.114213       1 sync.go:56] cert-manager/controller/orders "msg"="updating Order resource status" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.122008       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.122179       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.122208       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.122310       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="telemetry-gw-cert-2903184431-2058801590" "related_resource_namespace"="istio-system" "resource_kind"="CertificateRequest" "resource_name"="telemetry-gw-cert-2903184431" "resource_namespace"="istio-system" 
I0121 19:19:51.122323       1 sync.go:92] cert-manager/controller/orders "msg"="Fetching Authorizations from ACME server as status.authorizations contains unpopulated authorizations" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.122329       1 logger.go:75] Calling GetAuthorization
I0121 19:19:51.128365       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129041       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129042       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-ca "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129052       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-vault "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129069       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-venafi "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129071       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129109       1 controller.go:129] cert-manager/controller/certificates "msg"="syncing item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:51.129136       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-ca "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129139       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129139       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-venafi "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129150       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129347       1 sync.go:385] cert-manager/controller/certificates "msg"="validating existing CSR data" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" 
I0121 19:19:51.129380       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="telemetry-gw-cert-2903184431-2058801590" "related_resource_namespace"="istio-system" "resource_kind"="CertificateRequest" "resource_name"="telemetry-gw-cert-2903184431" "resource_namespace"="istio-system" 
I0121 19:19:51.129443       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.129503       1 sync.go:497] cert-manager/controller/certificates "msg"="CertificateRequest is not in a final state, waiting until CertificateRequest is complete" "related_resource_kind"="CertificateRequest" "related_resource_name"="telemetry-gw-cert-2903184431" "related_resource_namespace"="istio-system" "resource_kind"="Certificate" "resource_name"="telemetry-gw-cert" "resource_namespace"="istio-system" "state"="Pending"
I0121 19:19:51.129652       1 controller.go:135] cert-manager/controller/certificates "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert" 
I0121 19:19:51.219310       1 sync.go:56] cert-manager/controller/orders "msg"="updating Order resource status" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.226560       1 controller.go:129] cert-manager/controller/certificaterequests-issuer-acme "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.226909       1 acme.go:178] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="telemetry-gw-cert-2903184431-2058801590" "related_resource_namespace"="istio-system" "resource_kind"="CertificateRequest" "resource_name"="telemetry-gw-cert-2903184431" "resource_namespace"="istio-system" 
I0121 19:19:51.226970       1 controller.go:135] cert-manager/controller/certificaterequests-issuer-acme "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431" 
I0121 19:19:51.227320       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.227341       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.227435       1 logger.go:95] Calling HTTP01ChallengeResponse
I0121 19:19:51.227508       1 sync.go:129] cert-manager/controller/orders "msg"="Creating additional Challenge resources to complete Order" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.234495       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:51.234523       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:51.234553       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.234575       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:51.234681       1 logger.go:95] Calling HTTP01ChallengeResponse
I0121 19:19:51.234729       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:51.234751       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.159166       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:52.159188       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.159302       1 logger.go:70] Calling GetChallenge
I0121 19:19:52.159310       1 logger.go:95] Calling HTTP01ChallengeResponse
I0121 19:19:52.159371       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:52.159399       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.258080       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.258245       1 logger.go:95] Calling HTTP01ChallengeResponse
I0121 19:19:52.258301       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:52.258319       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.259038       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:52.259062       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:52.259262       1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.269163       1 service.go:55] cert-manager/controller/challenges/http01/ensureService "msg"="creating HTTP01 challenge solver service" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.300472       1 ingress.go:111] cert-manager/controller/challenges/http01/ensureIngress "msg"="creating HTTP01 challenge solver ingress" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.318891       1 controller.go:129] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="istio-system/cm-acme-http-solver-hnvxn" 
I0121 19:19:52.318947       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="cm-acme-http-solver-hnvxn" "resource_namespace"="istio-system" 
I0121 19:19:52.318955       1 controller.go:135] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="istio-system/cm-acme-http-solver-hnvxn" 
I0121 19:19:52.319970       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="test.domain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-8ql4d" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.320041       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="test.domain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-72842" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.320099       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="test.domain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-hnvxn" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
E0121 19:19:52.385463       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.393511       1 controller.go:129] cert-manager/controller/orders "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.393644       1 logger.go:95] Calling HTTP01ChallengeResponse
I0121 19:19:52.393691       1 sync.go:179] cert-manager/controller/orders "msg"="No action taken" "resource_kind"="Order" "resource_name"="telemetry-gw-cert-2903184431-2058801590" "resource_namespace"="istio-system" 
I0121 19:19:52.393708       1 controller.go:135] cert-manager/controller/orders "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590" 
I0121 19:19:52.394966       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:52.394992       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:19:52.395157       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="test.domain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-8ql4d" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.395208       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="test.domain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-72842" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.395250       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="test.domain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-hnvxn" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
E0121 19:19:52.435027       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:19:52.435064       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:20:02.385634       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:20:02.385961       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="test.domain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-8ql4d" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:02.386028       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="test.domain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-72842" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:02.386068       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="test.domain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-hnvxn" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
E0121 19:20:02.425699       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:02.425755       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:20:12.425872       1 controller.go:129] cert-manager/controller/challenges "msg"="syncing item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874" 
I0121 19:20:12.426101       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="test.domain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-8ql4d" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:12.426151       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="test.domain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-72842" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:12.426193       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="test.domain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-hnvxn" "related_resource_namespace"="istio-system" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
E0121 19:20:12.464591       1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="test.domain.com" "resource_kind"="Challenge" "resource_name"="telemetry-gw-cert-2903184431-2058801590-3026613874" "resource_namespace"="istio-system" "type"="http-01" 
I0121 19:20:12.464631       1 controller.go:135] cert-manager/controller/challenges "msg"="finished processing work item" "key"="istio-system/telemetry-gw-cert-2903184431-2058801590-3026613874"

Anything else we need to know?:

Environment details::

  • Kubernetes version (e.g. v1.10.2):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): 
aws kops
$ kops version
Version 1.15.0
  • istio: 1.4.0
  • cert-manager version (e.g. v0.4.0): v0.12.0
  • Install method (e.g. helm or static manifests): helm

My aws Kops cluster definition:

$ kops get cluster
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2019-12-16T18:17:45Z"
  generation: 4
  name: kops-us-east-2.bla.bla
spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "sts:AssumeRole"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
  api:
    dns: {}
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://mm-kops-bla-bla-state-store/kops-us-east-2.bla.bla
  dnsZone: bla.bla
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.15.6
  masterInternalName: api.internal.kops-us-east-2.bla.bla
  masterPublicName: api.kops-us-east-2.bla.bla
  networkCIDR: 172.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - xxx.xxx.xxx.x/22
    subnets:
  - cidr: 172.20.32.0/19
    name: us-east-2a
    type: Public
    zone: us-east-2a
  - cidr: 172.20.64.0/19
    name: us-east-2b
    type: Public
    zone: us-east-2b
  topology:
    dns:
      type: Public
    masters: public
    nodes: public
areacme help wanted kindocumentation prioritimportant-longterm
Source
picture Arnold1
👍10 👀1

Most helpful comment

I'm facing the same issue on cert-manager 0.13, Kubernetes 1.17 on bare-metal cluster with a vendor LB dispatching traffic across 3 kube masters and 3 workers.

Something very weird when I try to debug the challenge from the outside world - it's an ingress for kubernetes dashboard. I get the path used by the solver and check with cURL (every time both tests are made with the exact same solver):
Without https: 

❯ curl -XGET -ikL http://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Fri, 07 Feb 2020 21:11:07 GMT
content-length: 19

404 page not found

With https:

curl -XGET -ikL https://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 200 
server: nginx/1.17.7
date: Fri, 07 Feb 2020 21:10:56 GMT
content-type: text/plain; charset=utf-8
content-length: 87
cache-control: no-cache, no-store, must-revalidate
strict-transport-security: max-age=15724800; includeSubDomains
proxy-body-size: 999m

THE_ACTUAL_KEY_DAMN_IT

I thought disabling ssl_redirect (which is now default) in Ingress Nginx, disabling it on the ingress: still no luck. The dashboard can be accessed without https and without being redirected, but the solver still behaves this strange way.

Also that's not coming from my external LB: the exact same behavior happens when I'm hitting the solver's Service from a Pod in the same namespace.

picture superbiche on 7 Feb 2020
👍2 👀1

All 20 comments

@davidraleigh

Arnold1 picture Arnold1 on 18 Jan 2020

I'm facing the same issue on cert-manager 0.13, Kubernetes 1.17 on bare-metal cluster with a vendor LB dispatching traffic across 3 kube masters and 3 workers.

Something very weird when I try to debug the challenge from the outside world - it's an ingress for kubernetes dashboard. I get the path used by the solver and check with cURL (every time both tests are made with the exact same solver):
Without https: 

❯ curl -XGET -ikL http://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Fri, 07 Feb 2020 21:11:07 GMT
content-length: 19

404 page not found

With https:

curl -XGET -ikL https://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 200 
server: nginx/1.17.7
date: Fri, 07 Feb 2020 21:10:56 GMT
content-type: text/plain; charset=utf-8
content-length: 87
cache-control: no-cache, no-store, must-revalidate
strict-transport-security: max-age=15724800; includeSubDomains
proxy-body-size: 999m

THE_ACTUAL_KEY_DAMN_IT

I thought disabling ssl_redirect (which is now default) in Ingress Nginx, disabling it on the ingress: still no luck. The dashboard can be accessed without https and without being redirected, but the solver still behaves this strange way.

Also that's not coming from my external LB: the exact same behavior happens when I'm hitting the solver's Service from a Pod in the same namespace.

superbiche picture superbiche on 7 Feb 2020
👍2 👀1

I'm having the same problem, and I noticed that the acme-challenge url works only with a double slash. Maybe this helps tracking a typo somewhere in code, or maybe it's just something caused by my configuration: 

curl http://<hostname>/.well-known/acme-challenge/<TOKEN>
# default backend - 404%                                                                                                                 

curl http://<hostname>//.well-known/acme-challenge/<TOKEN>
# ZAPiasdasq5H9s_lcoIZ-qJMAYPVrATWgsadbNfgv-nS6VNnSls.ov5L-JPaOsVrAvpvmuNUsadsaJINqFVW0ToS8asdsalLK7R1pVA_0%
sadym picture sadym on 16 Feb 2020
👀2

I can confirm this.
Which one was the last working cert-manager version?
Edit:
Actually... I have hat it working with 0.13.1 (I think) some days ago. Not sure what I changed since then.

mindrunner picture mindrunner on 28 Feb 2020

I think it because nginx ingress hsts includeSubdomains is enabled by default. I only get the error when i have an ingress on the root domain and an subdomain or an www server alias. Can others confirm they have the same?

Stijn98s picture Stijn98s on 28 Feb 2020

Duplicate https://github.com/jetstack/cert-manager/issues/2517

mindrunner picture mindrunner on 28 Feb 2020

With regard to double slash in the path name that @sadym mentioned, I implemented a rewrite rule in a server snippet to test if this fixed the issue.

It did seem to force the request to resolve for a single slash, but it did not fix this issue for me.

k8s - v1.17.3
cert-manager - v0.13.1

kkurzhal picture kkurzhal on 9 Mar 2020

k8s - v1.15.9
cert-manager - v0.13.1
nginx-ingress - v0.4.3

I was able to solve this issue by downgrading the version of nginx-ingress. After losing 8 hours of my life, I'm now on v0.30.0 and it worked immediately. I do not think this is an error with cert-manager.

jonathan-graf picture jonathan-graf on 12 Mar 2020

With istio 1.5.0 the k8s default ingress doesn't work as expected, there is no route created.

chrstnwhlrt picture chrstnwhlrt on 13 Mar 2020

@chrstnwhlrt

With istio 1.5.0 the k8s default ingress doesn't work as expected, there is no route created.

did you find a way to work around that?

Rested picture Rested on 13 Mar 2020

@chrstnwhlrt

With istio 1.5.0 the k8s default ingress doesn't work as expected, there is no route created.

did you find a way to work around that?

Not yet, but I will take a look into it in the coming days. I redeployed istio 1.5.0 using istioctl but I originally came with the helm deployment method. I think some of the options are not handled at all (everything concerning .k8singress. doesn't seem to make a difference within the generated manifest, but is validated and accepted).

chrstnwhlrt picture chrstnwhlrt on 13 Mar 2020

@chrstnwhlrt

With istio 1.5.0 the k8s default ingress doesn't work as expected, there is no route created.

did you find a way to work around that?

Not yet, but I will take a look into it in the coming days. I redeployed istio 1.5.0 using istioctl but I originally came with the helm deployment method. I think some of the options are not handled at all (everything concerning _.k8singress._ doesn't seem to make a difference within the generated manifest, but is validated and accepted).

What worked for me was adding 

values:
  global:
    k8sIngress:
      enabled: true
      enableHttps: true
      gatewayName: ingressgateway

to my IstioOperator manifest (not using istioctl but am on 1.5.0). So maybe

istioctl manifest apply --set values.global.k8sIngress.enabled=true
# etc

would do the trick?

Rested picture Rested on 13 Mar 2020

This issue is a little hard to follow over time, and I know that there are issues when using Istio with ACME HTTP01 as cert-manager can only solve HTTP01 challenges using Ingress resources and does not manage VirtualService resources. There is an existing issue here to track us better supporting Istio natively.

In the meantime, if someone does have HTTP01 with Istio working with Ingress resources today, adding a page in our tutorials section in the documentation would really help others to get this setup. We could also include a troubleshooting & known issues section there which could explain common things to look out for.

/kind documentation
/area acme
/priority backlog
/help

munnerz picture munnerz on 24 Apr 2020
👀1

@munnerz:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

This issue is a little hard to follow over time, and I know that there are issues when using Istio with ACME HTTP01 as cert-manager can only solve HTTP01 challenges using Ingress resources and does not manage VirtualService resources. There is an existing issue here to track us better supporting Istio natively.

In the meantime, if someone does have HTTP01 with Istio working with Ingress resources today, adding a page in our tutorials section in the documentation would really help others to get this setup. We could also include a troubleshooting & known issues section there which could explain common things to look out for.

/kind documentation
/area acme
/priority backlog
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot picture jetstack-bot on 24 Apr 2020

I can see the originally linked doc here too: https://istio.io/docs/tasks/observability/gateways/#option-1-secure-access-https it may be worthwhile re-opening this issue over on the Istio tracker/discuss forum, as I'm not too familiar with how the setup between Istio and cert-manager works with HTTP01!

munnerz picture munnerz on 24 Apr 2020

The solution that worked for us is specifying FQDN with the spec.servers.host on the custom Istio Gateway used for the endpoint traffic, while keeping * on the istio-autogenerated-k8s-ingress gateway host setting.

This way auto-generated gateway serves the native K8s Ingress resources that the cert-manager uses for HTTP-01, while the custom gateway serves VirtualService resources with the certs generated by the cert-manager.

alexivkin picture alexivkin on 29 Apr 2020
👀1 👍1

@superbiche tnx for pointing me in right way:

I'm facing the same issue on cert-manager 0.13, Kubernetes 1.17 on bare-metal cluster with a
Without https:

❯ curl -XGET -ikL http://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 404 Not Found

With https:

curl -XGET -ikL https://kube-dashboard.solar.kissmy.cloud/.well-known/acme-challenge/TOKEN
HTTP/1.1 200 

THE_ACTUAL_KEY_DAMN_IT

I'm facing same issue
Seems I found cause of this problem but unfortunately no permanent solution

kubectl -n istio-system edit gateways.networking.istio.io istio-autogenerated-k8s-ingress

and change http port protocol from HTTP2 to HTTP

spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

Damn my http challenges works and I'm do not use k8s ingress for any other purposes and my all other gateways have specified FQDN's so all looks fine
But I'm pretty sure that editing autogenerated stuff are incorrect :(
I feel that solution can be on certmanager side around ingress and svc templating but still no success

ij3net picture ij3net on 14 May 2020

What worked for me was adding

values:
  global:
    k8sIngress:
      enabled: true
      enableHttps: true
      gatewayName: ingressgateway

to my IstioOperator manifest (not using istioctl but am on 1.5.0). So maybe

istioctl manifest apply --set values.global.k8sIngress.enabled=true
# etc

would do the trick?

I can confirm this has worked for me.

I had to add the following to my istioctl command:

    --set values.global.k8sIngress.enabled=true \
    --set values.global.k8sIngress.enableHttps=true \
    --set values.global.k8sIngress.gatewayName=ingressgateway

Then the certificates, which were stuck for many hours before this, were immediately generated.

luislhl picture luislhl on 3 Jun 2020

I am facing the same issue. Can anyone please help me. The http01 challenge via Istio is returning 405 error

yaminisridaran picture yaminisridaran on 30 Oct 2020

I was able to get http01 working with Istio and made a minimal repo with instructions here:
https://github.com/abatilo/istio-https-grpc

Full disclosure: there's a referral link in the README that helps me make money. Apologies if that's not alright.

abatilo picture abatilo on 28 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jbartus picture jbartus  ·  4Comments
Stono picture Stono  ·  3Comments
f-f picture f-f  ·  4Comments
matthew-muscat picture matthew-muscat  ·  4Comments
howardjohn picture howardjohn  ·  3Comments