Cert-manager: ACME Solver Pod

Created on 17 Apr 2019  Â·  19Comments  Â·  Source: jetstack/cert-manager

c:\Development\kubeclient>
We are using a third party tool to manage the containers in our environment. Also we have our own ACME server
I have created a below Cluster Issuer which registers the account successfully
kubectl describe clusterissuer

Name: xxxxxtestcluster
Namespace:
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"xxxxxtestcluster"},"spec":{"acme":{"email":"abc@...
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2019-04-17T04:16:25Z
Generation: 1
Resource Version: 14382539
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/xxxxxtestcluster
UID: 907e5e6e-60c7-11e9-88ad-00505608c441
Spec:
Acme:
Email: [email protected]
Http 01:
Private Key Secret Ref:
Name: xxxxxcluster
Server: http://xxxx.xxxx.xxxx.net:8080/directory
Skip TLS Verify: true
Status:
Acme:
Uri: http://xxxx.xxxx.xxxx.net:8080/account/HfymrXpSl-LUrsozLg_jTgAAAWopgpnB
Conditions:
Last Transition Time: 2019-04-17T04:16:26Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events:

The ingress yaml looks like below
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/cluster-issuer: xxxtestcluster
ingress.kubernetes.io/ssl-redirect: "false"
creationTimestamp: 2019-04-17T04:25:14Z
generation: 8
labels:
nirmata.io/application.name: i360
nirmata.io/ingress.name: ui
name: ui
namespace: sanjaytest
resourceVersion: "14459424"
selfLink: /apis/extensions/v1beta1/namespaces/sanjaytest/ingresses/ui
uid: cbfc05b9-60c8-11e9-b3a8-00505608c440
spec:
backend:
serviceName: i360ui
servicePort: 80
rules:

  • host: sanjayi360.dev.xxx.xxxx.xxxx
    http:
    paths:

    • backend:

      serviceName: i360ui

      servicePort: 80

      path: /

    • backend:

      serviceName: i360ui

      servicePort: 80

      path: /admin

      tls:

  • hosts:

    • sanjayi360.dev.xxx.xxxx.xxxx

      secretName: pk

      status:

      loadBalancer:

      ingress:

    • {}

      The cert-manager controller is printing "Waiting for Order to transition before attempting to issue Certificate."

Also our acme server when it sends get request for http challenge validation gets null response from http://sanjayi360.dev.xxx.xxx.xxxx/.well-known/acme-challenge/yIcT029DF7KBnGOrFy_8LzMEJLh4u344CBIZS-tWCXT6_QEng3ln7JFoK57RTdpTJetorgAAAWorNDwe

I am not sure if our ingress is setup properly with all the required configuration. I checked our events for acme-solver pod but could not find any running/terminated instance

areacme triagneeds-information triagsupport
Source
picture san360

All 19 comments

You should be able to get more information on what's going on here by looking at the Order resource that cert-manager has created: http://docs.cert-manager.io/en/latest/reference/orders.html

There should then be corresponding Challenge resources too.

If this does not help, please could you attach a copy of your log output from cert-manager? 😄

munnerz picture munnerz on 24 Apr 2019

Also we have our own ACME server

Is this a custom-written ACME server, or a Boulder deployment? Have you validated that your own ACME server is 'compliant' by issuing a certificate with another client (certbot, etc?)

munnerz picture munnerz on 24 Apr 2019

The acme server used by us is built on https://gitlab.com/ID4me/Acme/ with minor changes (Hooking it with our internal CA, changes to ACME server). we have tested this implementation with acme.sh, certbot client, acme4j and it works.

san360 picture san360 on 24 Apr 2019

If you can follow the aforementioned debugging steps first, and then include a copy of your logs here, I'll be able to help dig in 😄

munnerz picture munnerz on 24 Apr 2019

C:\Users\singhsau>kubectl describe orders --all-namespaces
Name: pk-1744134006
Namespace: sanjaytest
Labels: acme.cert-manager.io/certificate-name=pk
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Order
Metadata:
Creation Timestamp: 2019-04-24T14:31:20Z
Generation: 1
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: pk
UID: a0c241fc-669d-11e9-9fc2-00505608c43f
Resource Version: 16111711
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/orders/pk-1744134006
UID: a0cc00ee-669d-11e9-9fc2-00505608c43f
Spec:
Config:
Domains:
sanjayi360.dev.xxx.xxxxxx.xxxx
Http 01:
Csr: MIICxTCCAa0CAQAwQjEVMBMGA1UEChMMY2VydC1tYW5hZ2VyMSkwJwYDVQQDEyBzYW5qYXlpMzYwLmRldi5uaWJyLm5vdmFydGlzLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKVmtFvJ1P2RroQM/4ReQiSQ7XVmmxTYw6xrReQUIIgBu1DpFN2n0k+fjTev4uMwoHiAya2tScrYNYOoR/GmDG7FakDWzeriCtoF4kSfpOAqdlsJ5C0EqCgCbr9C9YM2+eQ0xPaLrsT/apiNDXuZ8EFixFBZQzvWk6EYPGpvik2EmSKimuVTfgkFiYqsYw4fyGapmDVIlhdoWUM6SeHx8ONNpvumRn33iTEgZnwzNGzIQc2LbtpXfNLEeGqMJFm1ovhXWSkbDuVvLLUzH8r1Zbr8FWO2hS+G28ibabFVUWNKZolCBAFIYnYE4sUjje61BX7J7XBfHFNhoNSkhXCHSY0CAwEAAaA+MDwGCSqGSIb3DQEJDjEvMC0wKwYDVR0RBCQwIoIgc2FuamF5aTM2MC5kZXYubmlici5ub3ZhcnRpcy5uZXQwDQYJKoZIhvcNAQELBQADggEBACPtdUU1qFrB5VCIEmgglf/7Gxrm/4V0Gq3VmNLYfmYjQdptNFlK+PB1vT8nmgao37+ibATksoNGFINIlHwbP9AlqTIIyz7l9vj3tD0pUQScc5ws8eprafLYiEvp/PvPsm2jtrtyYFuH1DeauyBd96HyOAuc1rfxmSd2WAA/iuu9/qCnyVqXmmcIBsfW2kLPHNhVDW0US4zIUJCElBmo+ybGPt/3M+d6sf75/jx6d3HwUfmSyXmUL9uOYzmpak70aHfYcEpXm/0pv3ABW/weTeRFZa9tMnnmDjxqCJANj1tX5kIq0uvXM8IIkypqZoEpWEOri9jU6nDR4c5E4Pqdc+0=
Dns Names:
sanjayi360.dev.xxx.xxxxxx.xxxx
Issuer Ref:
Kind: ClusterIssuer
Name: xxxxxtestcluster
Status:
Challenges:
Authz URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/authz/927csNVUC6dLwK6z40lhLwAAAWpPwht6
Config:
Http 01:
Dns Name: sanjayi360.dev.xxx.xxxxxx.xxxx
Issuer Ref:
Kind: ClusterIssuer
Name: xxxxxtestcluster
Key: u4fCSAclH-O_roJZo2jtLmSrSItc6bk0vzc2vpa08P6gJ4i_Lh1O-zDAkcQANJN0kJbLfAAAAWpPwht5.J1ak5i0EDBwmlAMGeMAdccU78cij7QSHxwsOCRdUO7k
Token: u4fCSAclH-O_roJZo2jtLmSrSItc6bk0vzc2vpa08P6gJ4i_Lh1O-zDAkcQANJN0kJbLfAAAAWpPwht5
Type: http-01
URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/chall/4lSpSJNqQrWVJhQLNLRYBgAAAWpPwhuB
Wildcard: false
Finalize URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/order/order/Z6THw5anCGlg_OzxOfcMbQAAAWpPwhtV/finalize
State: pending
URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/order/Z6THw5anCGlg_OzxOfcMbQAAAWpPwhtV
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 1m cert-manager Created Challenge resource "pk-1744134006-0" for domain "sanjayi360.dev.xxx.xxxxxx.xxxx"

C:\Users\singhsau>kubectl describe challenges --all-namespaces
Name: pk-1744134006-0
Namespace: sanjaytest
Labels: acme.cert-manager.io/order-name=pk-1744134006
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Challenge
Metadata:
Creation Timestamp: 2019-04-24T14:31:22Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 1
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: pk-1744134006
UID: a0cc00ee-669d-11e9-9fc2-00505608c43f
Resource Version: 16111710
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/challenges/pk-1744134006-0
UID: a2017abd-669d-11e9-9fc2-00505608c43f
Spec:
Authz URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/authz/927csNVUC6dLwK6z40lhLwAAAWpPwht6
Config:
Http 01:
Dns Name: sanjayi360.dev.xxx.xxxxxx.xxxx
Issuer Ref:
Kind: ClusterIssuer
Name: xxxxxtestcluster
Key: u4fCSAclH-O_roJZo2jtLmSrSItc6bk0vzc2vpa08P6gJ4i_Lh1O-zDAkcQANJN0kJbLfAAAAWpPwht5.J1ak5i0EDBwmlAMGeMAdccU78cij7QSHxwsOCRdUO7k
Token: u4fCSAclH-O_roJZo2jtLmSrSItc6bk0vzc2vpa08P6gJ4i_Lh1O-zDAkcQANJN0kJbLfAAAAWpPwht5
Type: http-01
URL: http://xxxxx-sld4088.xxx.xxxxxx.xxxx:8080/chall/4lSpSJNqQrWVJhQLNLRYBgAAAWpPwhuB
Wildcard: false
Status:
Presented: false
Processing: false
Reason:
State: invalid
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 2m cert-manager Challenge scheduled for processing

C:\Users\singhsau>kubectl describe certificates -n sanjaytest
Name: pk
Namespace: sanjaytest
Labels:
Annotations:
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-04-24T14:31:20Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: ui
UID: cbfc05b9-60c8-11e9-b3a8-00505608c440
Resource Version: 16111689
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/certificates/pk
UID: a0c241fc-669d-11e9-9fc2-00505608c43f
Spec:
Acme:
Config:
Domains:
sanjayi360.dev.xxx.xxxxxx.xxxx
Http 01:
Dns Names:
sanjayi360.dev.xxx.xxxxxx.xxxx
Issuer Ref:
Kind: ClusterIssuer
Name: xxxxxtestcluster
Secret Name: pk
Status:
Conditions:
Last Transition Time: 2019-04-24T14:31:20Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 4m cert-manager Created Order resource "pk-1744134006"

C:\Users\singhsau>kubectl logs cert-manager-67f64fc697-p2gzb -n cert-manager -f
I0424 14:31:20.299300 1 start.go:76] cert-manager "level"=0 "msg"="starting controller" "git-commit"="3a747cda7547496d812966d8e6725bf1a59140b6" "version"="canary"
I0424 14:31:20.301439 1 controller.go:154] cert-manager/controller/build-context "level"=0 "msg"="configured acme dns01 nameservers" "nameservers"=["10.10.0.10:53"]
I0424 14:31:20.301708 1 controller.go:118] cert-manager/controller "level"=0 "msg"="starting leader election"
I0424 14:31:20.303080 1 leaderelection.go:217] attempting to acquire leader lease cert-manager/cert-manager-controller...
I0424 14:31:20.303841 1 metrics.go:147] cert-manager/metrics "level"=0 "msg"="listening for connections on" "address"="0.0.0.0:9402"
I0424 14:31:20.342951 1 leaderelection.go:227] successfully acquired lease cert-manager/cert-manager-controller
I0424 14:31:20.345178 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="certificates"
I0424 14:31:20.345216 1 controller.go:126] cert-manager/controller/certificates "level"=0 "msg"="starting control loop"
I0424 14:31:20.346237 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="ingress-shim"
I0424 14:31:20.346188 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="clusterissuers"
I0424 14:31:20.346296 1 controller.go:107] cert-manager/controller/clusterissuers "level"=0 "msg"="starting control loop"
I0424 14:31:20.347530 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="issuers"
I0424 14:31:20.347614 1 controller.go:109] cert-manager/controller/issuers "level"=0 "msg"="starting control loop"
I0424 14:31:20.348141 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="orders"
I0424 14:31:20.348192 1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="challenges"
E0424 14:31:20.352210 1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk-1744134006\" not found" "related_resource_kind"="Order" "related_resource_name"="pk-1744134006" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk-1744134006-0" "resource_namespace"="sanjaytest"
I0424 14:31:20.445426 1 controller.go:132] cert-manager/controller/certificates "level"=0 "msg"="synced all caches for control loop"
I0424 14:31:20.445595 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:20.445636 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I0424 14:31:20.446223 1 sync.go:285] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="42610h51m17.553816089s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0424 14:31:20.446263 1 sync.go:285] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="7570h51m22.55377926s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-webhook-tls" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="cert-manager-webhook-webhook-tls" "resource_namespace"="cert-manager"
I0424 14:31:20.446543 1 controller.go:178] ingress-shim controller: syncing item 'amit-test/amit-ingres'
I0424 14:31:20.446587 1 sync.go:64] Not syncing ingress amit-test/amit-ingres as it does not contain necessary annotations
I0424 14:31:20.446597 1 controller.go:184] ingress-shim controller: Finished processing work item "amit-test/amit-ingres"
I0424 14:31:20.446614 1 controller.go:178] ingress-shim controller: syncing item 'aqua/aquasec'
I0424 14:31:20.446640 1 sync.go:64] Not syncing ingress aqua/aquasec as it does not contain necessary annotations
I0424 14:31:20.446650 1 controller.go:184] ingress-shim controller: Finished processing work item "aqua/aquasec"
I0424 14:31:20.446658 1 controller.go:178] ingress-shim controller: syncing item 'xxxxx-next/next-predev-ingress'
I0424 14:31:20.446666 1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/ui'
I0424 14:31:20.446672 1 sync.go:64] Not syncing ingress xxxxx-next/next-predev-ingress as it does not contain necessary annotations
I0424 14:31:20.446679 1 controller.go:184] ingress-shim controller: Finished processing work item "xxxxx-next/next-predev-ingress"
I0424 14:31:20.446691 1 controller.go:178] ingress-shim controller: syncing item 'nirmata-monitoring/grafana'
I0424 14:31:20.446704 1 sync.go:64] Not syncing ingress nirmata-monitoring/grafana as it does not contain necessary annotations
I0424 14:31:20.446712 1 controller.go:184] ingress-shim controller: Finished processing work item "nirmata-monitoring/grafana"
I0424 14:31:20.446789 1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="xxxxxtestcluster"
I0424 14:31:20.446640 1 controller.go:178] ingress-shim controller: syncing item 'loadtest/ui'
I0424 14:31:20.446977 1 sync.go:64] Not syncing ingress loadtest/ui as it does not contain necessary annotations
I0424 14:31:20.446989 1 controller.go:184] ingress-shim controller: Finished processing work item "loadtest/ui"
I0424 14:31:20.446619 1 controller.go:178] ingress-shim controller: syncing item 'knime-dev/knimedev'
I0424 14:31:20.447025 1 sync.go:64] Not syncing ingress knime-dev/knimedev as it does not contain necessary annotations
I0424 14:31:20.447036 1 controller.go:184] ingress-shim controller: Finished processing work item "knime-dev/knimedev"
I0424 14:31:20.447213 1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="xxxxxcluster" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="xxxxxtestcluster" "resource_namespace"=""
I0424 14:31:20.447292 1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="xxxxxtestcluster"
I0424 14:31:20.448073 1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:20.448348 1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0424 14:31:20.448385 1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I0424 14:31:20.448401 1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:20.448423 1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="sanjaytest/xxxxxtest"
I0424 14:31:20.448481 1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I0424 14:31:20.448553 1 setup.go:156] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="xxxxx" "related_resource_namespace"="sanjaytest" "resource_kind"="Issuer" "resource_name"="xxxxxtest" "resource_namespace"="sanjaytest"
I0424 14:31:20.448585 1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="sanjaytest/xxxxxtest"
I0424 14:31:20.450069 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:20.456872 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I0424 14:31:20.458229 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:20.463690 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
E0424 14:31:20.464432 1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk-1744134006\" not found" "related_resource_kind"="Order" "related_resource_name"="pk-1744134006" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk-1744134006-0" "resource_namespace"="sanjaytest"
I0424 14:31:20.464545 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
E0424 14:31:20.464656 1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"pk-1744134006-0\" not found"
I0424 14:31:20.464759 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:20.475515 1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/ui"
I0424 14:31:20.475622 1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/ui'
I0424 14:31:20.475691 1 sync.go:180] Certificate "pk" for ingress "ui" already exists
I0424 14:31:20.475758 1 sync.go:183] Certificate "pk" for ingress "ui" is up to date
I0424 14:31:20.475787 1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/ui"
I0424 14:31:20.475540 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk"
I0424 14:31:20.497769 1 conditions.go:154] Setting lastTransitionTime for Certificate "pk" condition "Ready" to 2019-04-24 14:31:20.476309837 +0000 UTC m=+0.199537034
I0424 14:31:20.522821 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:20.523133 1 logger.go:38] Calling CreateOrder
I0424 14:31:20.558077 1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/ui'
I0424 14:31:20.558137 1 sync.go:180] Certificate "pk" for ingress "ui" already exists
I0424 14:31:20.558200 1 sync.go:183] Certificate "pk" for ingress "ui" is up to date
I0424 14:31:20.558209 1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/ui"
I0424 14:31:20.559326 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk"
I0424 14:31:20.559363 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk"
I0424 14:31:20.559959 1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="pk-1744134006" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="pk" "resource_namespace"="sanjaytest"
I0424 14:31:20.560083 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk"
I0424 14:31:22.352883 1 logger.go:73] Calling GetAuthorization
I0424 14:31:22.528621 1 logger.go:93] Calling HTTP01ChallengeResponse
I0424 14:31:22.539453 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:22.539561 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk"
I0424 14:31:22.539643 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:22.539912 1 sync.go:274] Need to create 1 challenges
I0424 14:31:22.540210 1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="pk-1744134006" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="pk" "resource_namespace"="sanjaytest"
I0424 14:31:22.540344 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk"
I0424 14:31:22.549062 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:22.549363 1 sync.go:323] Waiting for all challenges for order "pk-1744134006" to enter 'valid' state
I0424 14:31:22.549441 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:22.549493 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:22.549645 1 sync.go:274] Need to create 0 challenges
I0424 14:31:22.549691 1 sync.go:323] Waiting for all challenges for order "pk-1744134006" to enter 'valid' state
I0424 14:31:22.549732 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:22.549820 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:23.458149 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:23.458387 1 logger.go:68] Calling GetChallenge
I0424 14:31:23.458152 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:23.458771 1 sync.go:274] Need to create 0 challenges
I0424 14:31:23.458796 1 sync.go:323] Waiting for all challenges for order "pk-1744134006" to enter 'valid' state
I0424 14:31:23.458832 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.204923 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:24.205234 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:24.205280 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.206148 1 sync.go:274] Need to create 0 challenges
I0424 14:31:24.206167 1 logger.go:43] Calling GetOrder
I0424 14:31:24.212583 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:24.213249 1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:24.213414 1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006-0"
I0424 14:31:24.468032 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.468086 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.468311 1 sync.go:274] Need to create 0 challenges
I0424 14:31:24.468336 1 logger.go:43] Calling GetOrder
I0424 14:31:24.468405 1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk"
I0424 14:31:24.468933 1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="pk-1744134006" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="pk" "resource_namespace"="sanjaytest"
I0424 14:31:24.469118 1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk"
I0424 14:31:24.722769 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.722815 1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-1744134006"
I0424 14:31:24.723042 1 sync.go:274] Need to create 0 challenges
I0424 14:31:24.723062 1 logger.go:43] Calling GetOrder
I0424 14:31:24.955863 1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-1744134006"
I0424 14:31:25.379164 1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:25.379793 1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0424 14:31:25.379879 1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0424 14:31:25.379977 1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="sanjaytest/xxxxxtest"
I0424 14:31:25.380041 1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="xxxxxtestcluster"
I0424 14:31:25.380396 1 setup.go:156] cert-manager/controller/issuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="xxxxx" "related_resource_namespace"="sanjaytest" "resource_kind"="Issuer" "resource_name"="xxxxxtest" "resource_namespace"="sanjaytest"
I0424 14:31:25.380475 1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="xxxxxcluster" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="xxxxxtestcluster" "resource_namespace"=""
I0424 14:31:25.380548 1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="xxxxxtestcluster"
I0424 14:31:25.380619 1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="sanjaytest/xxxxxtest"

san360 picture san360 on 24 Apr 2019

Do you need some more information?

san360 picture san360 on 30 Apr 2019

i did a little more debugging and found out that challenge response is not resolved by ingress but its directed to application container. We are using haproxy ingress controller and have set below annotation

certmanager.k8s.io/cluster-issuer=letsencrypt-prd
ingress.kubernetes.io/ssl-redirect=false

Am i missing something?

san360 picture san360 on 7 May 2019 
C:\Users\singhsau>**kubectl describe certificates -n sanjaytest**
Name:         23467
Namespace:    sanjaytest
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-05-08T16:40:16Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  i360
    UID:                   e4434e9b-700b-11e9-afe8-00505608c441
  Resource Version:        19376424
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/certificates/23467
  UID:                     f55dd4d3-71af-11e9-bf21-00505608c440
Spec:
  Acme:
    Config:
      Domains:
        sanjayi360.dev.xxx.xxxxxx.xxxx
      Http 01:
  Dns Names:
    sanjayi360.dev.xxx.xxxxxx.xxxx
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prd
  Secret Name:  23467
Status:
  Conditions:
    Last Transition Time:  2019-05-08T16:40:16Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age              From          Message
  ----    ------              ----             ----          -------
  Normal  Generated           3m               cert-manager  Generated new private key
  Normal  GenerateSelfSigned  3m               cert-manager  Generated temporary self signed certificate
  Normal  OrderCreated        2m (x2 over 3m)  cert-manager  Created Order resource "23467-3100399849"


C:\Users\singhsau>**kubectl describe orders -n sanjaytest**
Name:         23467-3100399849
Namespace:    sanjaytest
Labels:       acme.cert-manager.io/certificate-name=23467
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Order
Metadata:
  Creation Timestamp:  2019-05-08T16:40:44Z
  Generation:          1
  Owner References:
    API Version:           certmanager.k8s.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  23467
    UID:                   f55dd4d3-71af-11e9-bf21-00505608c440
  Resource Version:        19376509
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/orders/23467-3100399849
  UID:                     064be038-71b0-11e9-bf21-00505608c440
Spec:
  Config:
    Domains:
      sanjayi360.dev.xxx.xxxxxx.xxxx
    Http 01:
  Csr:  MIICxTCCAa0CAQAwQjEVMBMGA1UEChMMY2VydC1tYW5hZ2VyMSkwJwYDVQQDEyBzYW5qYXlpMzYwLmRldi5uaWJyLm5vdmFydGlzLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL1XIUI3f847Eql5U0t7vmvb4Y0YzZNdbJa3nXNbc+4TB6H5SlhpwtWzZssr1HLkGFy9wbU1f+NcSymVsn9ilAAu+WW9QnEkYHUphtqmK2UpV8krEI+FslG2iuM9IDDXVGoyO7AlQPa8bPQrg+wQSAtYtJYGtmBwjFcU2ez4DyMbWH0qTr+QApJPcpZC9+6z9NWyZ6S0fLFgY/6fd/UKMF1+SUbKIRl4mytF3anM5nXAsyntM0Y/gPb6UiMyvfvdddvd3hm4DnEWTiyEaFXtXsNdZaOTb65oyFG16f+50DhjtClLdhmiB9olJth25ArUNPDKikmTS/N/RCqEq96/dPMCAwEAAaA+MDwGCSqGSIb3DQEJDjEvMC0wKwYDVR0RBCQwIoIgc2FuamF5aTM2MC5kZXYubmlici5ub3ZhcnRpcy5uZXQwDQYJKoZIhvcNAQELBQADggEBAHlduY3xlgnlPDle0AQwnW/21jmmq9EeXvgtfW3AtT6OcdW3pQLCTqsYTEz1WDnvEbOScxT9czf5eNd9qYJ/Aw6TwD2fRUAkU0AIbPg6NO6F2pOLXy62UAgnqG3vOdNxpazzsZl1F1iEBu2tEIdYm1wpuO9Hb6S6COKdIHYXoOvaeDPgU+HHxQ0c7yjdSMeBX8s2dyrfgB5lGy+VNPLQWD3zJDOiNwjBOW2+Hc6Gdb3UyGGyhF9fs8LRUOn8HcF9Pvn/HXaQHK/dKBr17tXK8bdcDcAb545MBJROvBultWTXEMeZQDwk/6agnGzypkI8s3zVd1q87wC6cZBa0UolWng=
  Dns Names:
    sanjayi360.dev.xxx.xxxxxx.xxxx
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prd
Status:
  Challenges:
    Authz URL:  http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/authz/qGddexbFJCFDfrLHwuRcdwAAAWqYUZXb
    Config:
      Http 01:
    Dns Name:  sanjayi360.dev.xxx.xxxxxx.xxxx
    Issuer Ref:
      Kind:      ClusterIssuer
      Name:      letsencrypt-prd
    Key:         srVmzKyD-UGq-CXC3JUSdj50rq847FehZ5QxzwuWYyPiUgvBPeSK-JHshn6r2VTRxBa_RAAAAWqYUZXb.xklneKsllP8_5YuR7gG0IiZYtXmYZU7QjJEqwBtsHKc
    Token:       srVmzKyD-UGq-CXC3JUSdj50rq847FehZ5QxzwuWYyPiUgvBPeSK-JHshn6r2VTRxBa_RAAAAWqYUZXb
    Type:        http-01
    URL:         http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/chall/2pXUs9TB8c7QbBVQzXf6ggAAAWqYUZXc
    Wildcard:    false
  Finalize URL:  http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/order/order/x8C0QYX5sHkfLEN1gGVXmQAAAWqYUZXU/finalize
  State:         pending
  URL:           http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/order/x8C0QYX5sHkfLEN1gGVXmQAAAWqYUZXU
Events:
  Type    Reason   Age              From          Message
  ----    ------   ----             ----          -------
  Normal  Created  3m (x2 over 3m)  cert-manager  Created Challenge resource "23467-3100399849-0" for domain "sanjayi360.dev.xxx.xxxxxx.xxxx"

C:\Users\singhsau>**kubectl describe challenges -n sanjaytest**
Name:         23467-3100399849-0
Namespace:    sanjaytest
Labels:       acme.cert-manager.io/order-name=23467-3100399849
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2019-05-08T16:40:53Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           certmanager.k8s.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  23467-3100399849
    UID:                   064be038-71b0-11e9-bf21-00505608c440
  Resource Version:        19376538
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/sanjaytest/challenges/23467-3100399849-0
  UID:                     0be21743-71b0-11e9-bf21-00505608c440
Spec:
  Authz URL:  http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/authz/qGddexbFJCFDfrLHwuRcdwAAAWqYUZXb
  Config:
    Http 01:
  Dns Name:  sanjayi360.dev.xxx.xxxxxx.xxxx
  Issuer Ref:
    Kind:    ClusterIssuer
    Name:    letsencrypt-prd
  Key:       srVmzKyD-UGq-CXC3JUSdj50rq847FehZ5QxzwuWYyPiUgvBPeSK-JHshn6r2VTRxBa_RAAAAWqYUZXb.xklneKsllP8_5YuR7gG0IiZYtXmYZU7QjJEqwBtsHKc
  Token:     srVmzKyD-UGq-CXC3JUSdj50rq847FehZ5QxzwuWYyPiUgvBPeSK-JHshn6r2VTRxBa_RAAAAWqYUZXb
  Type:      http-01
  URL:       http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/chall/2pXUs9TB8c7QbBVQzXf6ggAAAWqYUZXc
  Wildcard:  false
Status:
  Presented:   false
  Processing:  false
  Reason:
  State:       invalid
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Started  3m    cert-manager  Challenge scheduled for processing

C:\Users\singhsau>**kubectl describe ingress -n sanjaytest**
Name:             i360
Namespace:        sanjaytest
Address:
Default backend:  i360ui:80 (10.244.4.16:80)
TLS:
  23467 terminates sanjayi360.dev.xxx.xxxxxx.xxxx
Rules:
  Host                              Path  Backends
  ----                              ----  --------
  sanjayi360.dev.xxx.xxxxxx.xxxx
                                    /   i360ui:80 (10.244.4.16:80)
Annotations:
  ssl-redirect:  false
Events:
  Type    Reason             Age   From                Message
  ----    ------             ----  ----                -------
  Normal  UPDATE             15m   ingress-controller  Ingress sanjaytest/i360
  Normal  CreateCertificate  4m    cert-manager        Successfully created Certificate "23467"
san360 picture san360 on 8 May 2019 
C:\Users\singhsau>kubectl logs cert-manager-67f64fc697-sv72l -n cert-manager -f --v=4
I0508 16:40:11.946982       1 start.go:76] cert-manager "level"=0 "msg"="starting controller"  "git-commit"="3a747cda7547496d812966d8e6725bf1a59140b6" "version"="canary"
I0508 16:40:11.951578       1 controller.go:154] cert-manager/controller/build-context "level"=0 "msg"="configured acme dns01 nameservers" "nameservers"=["10.10.0.10:53"]
I0508 16:40:11.951801       1 controller.go:118] cert-manager/controller "level"=0 "msg"="starting leader election"
I0508 16:40:11.952619       1 metrics.go:147] cert-manager/metrics "level"=0 "msg"="listening for connections on" "address"="0.0.0.0:9402"
I0508 16:40:11.953158       1 leaderelection.go:217] attempting to acquire leader lease  cert-manager/cert-manager-controller...
I0508 16:40:11.980333       1 leaderelection.go:227] successfully acquired lease cert-manager/cert-manager-controller
I0508 16:40:11.982082       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="challenges"
I0508 16:40:11.982232       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="orders"
I0508 16:40:11.982362       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="certificates"
I0508 16:40:11.982420       1 controller.go:126] cert-manager/controller/certificates "level"=0 "msg"="starting control loop"
I0508 16:40:11.982457       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="ingress-shim"
I0508 16:40:11.982501       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="clusterissuers"
I0508 16:40:11.982525       1 controller.go:107] cert-manager/controller/clusterissuers "level"=0 "msg"="starting control loop"
I0508 16:40:11.984458       1 controller.go:93] cert-manager/controller "level"=0 "msg"="starting controller" "controller"="issuers"
I0508 16:40:11.984497       1 controller.go:109] cert-manager/controller/issuers "level"=0 "msg"="starting control loop"
E0508 16:40:11.996526       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"12345-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="12345-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="12345-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.997667       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk-3628299734\" not found" "related_resource_kind"="Order" "related_resource_name"="pk-3628299734" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk-3628299734-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.997841       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"sanjayi360http-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="sanjayi360http-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="sanjayi360http-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.997888       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"test-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="test-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="test-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.997926       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"testingi360-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="testingi360-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="testingi360-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.997960       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"test234-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="test234-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="test234-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.998120       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"wwww-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="wwww-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="wwww-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.998174       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"11-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="11-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="11-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.998210       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"12-3164661215\" not found" "related_resource_kind"="Order" "related_resource_name"="12-3164661215" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="12-3164661215-0" "resource_namespace"="sanjaytest"
E0508 16:40:11.998647       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"1234-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="1234-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="1234-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.000307       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"45657-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="45657-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="45657-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.000366       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk1-3628299734\" not found" "related_resource_kind"="Order" "related_resource_name"="pk1-3628299734" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk1-3628299734-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.000448       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:12.082496       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/12345-3100399849-0"
I0508 16:40:12.082588       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/sanjayi360http-3100399849-0"
I0508 16:40:12.082620       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/testingi360-3100399849-0"
I0508 16:40:12.082496       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-3628299734-0"
I0508 16:40:12.083949       1 controller.go:178] ingress-shim controller: syncing item 'nirmata-monitoring/grafana'
I0508 16:40:12.083986       1 sync.go:64] Not syncing ingress nirmata-monitoring/grafana as it does not contain necessary annotations
I0508 16:40:12.083995       1 controller.go:184] ingress-shim controller: Finished processing work item "nirmata-monitoring/grafana"
I0508 16:40:12.084011       1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/i360'
I0508 16:40:12.085314       1 controller.go:132] cert-manager/controller/certificates "level"=0 "msg"="synced all caches for control loop"
I0508 16:40:12.085367       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:12.086098       1 sync.go:285] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="42272h42m25.913943397s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0508 16:40:12.086059       1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-selfsign"
I0508 16:40:12.086339       1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:12.086350       1 controller.go:178] ingress-shim controller: syncing item 'amit-test/amit-ingres'
I0508 16:40:12.086421       1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-selfsign"
I0508 16:40:12.086450       1 controller.go:184] ingress-shim controller: Finished processing work item "amit-test/amit-ingres"
I0508 16:40:12.086473       1 controller.go:178] ingress-shim controller: syncing item 'aqua/aquasec'
I0508 16:40:12.086483       1 sync.go:64] Not syncing ingress aqua/aquasec as it does not contain necessary annotations
I0508 16:40:12.086490       1 controller.go:184] ingress-shim controller: Finished processing work item "aqua/aquasec"
I0508 16:40:12.086500       1 controller.go:178] ingress-shim controller: syncing item 'knime-dev/knimedev'
I0508 16:40:12.086513       1 sync.go:64] Not syncing ingress knime-dev/knimedev as it does not contain necessary annotations
I0508 16:40:12.086519       1 controller.go:184] ingress-shim controller: Finished processing work item "knime-dev/knimedev"
I0508 16:40:12.086529       1 controller.go:178] ingress-shim controller: syncing item 'loadtest/ui'
I0508 16:40:12.086537       1 sync.go:64] Not syncing ingress loadtest/ui as it does not contain necessary annotations
I0508 16:40:12.086548       1 controller.go:184] ingress-shim controller: Finished processing work item "loadtest/ui"
I0508 16:40:12.086564       1 controller.go:178] ingress-shim controller: syncing item 'nibr-ecs-predev/next-predev-ingress'
I0508 16:40:12.086579       1 sync.go:64] Not syncing ingress nibr-ecs-predev/next-predev-ingress as it does not contain necessary annotations
I0508 16:40:12.086586       1 controller.go:184] ingress-shim controller: Finished processing work item "nibr-ecs-predev/next-predev-ingress"
I0508 16:40:12.086780       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I0508 16:40:12.086795       1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0508 16:40:12.086858       1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:12.087089       1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="letsencrypt-prd"
I0508 16:40:12.087323       1 sync.go:285] cert-manager/controller/certificates "level"=0 "msg"="certificate scheduled for renewal" "duration_until_renewal"="7232h42m30.912731291s" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-webhook-tls" "related_resource_namespace"="cert-manager" "resource_kind"="Certificate" "resource_name"="cert-manager-webhook-webhook-tls" "resource_namespace"="cert-manager"
I0508 16:40:12.087345       1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prd" "resource_namespace"=""
I0508 16:40:12.087381       1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="letsencrypt-prd"
I0508 16:40:12.087420       1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="letsencrypt-dev"
I0508 16:40:12.087595       1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-developer" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-dev" "resource_namespace"=""
I0508 16:40:12.087636       1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="letsencrypt-dev"
I0508 16:40:12.082592       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/test-3100399849-0"
I0508 16:40:12.100213       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/12345-3100399849-0"
I0508 16:40:12.100285       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/test234-3100399849-0"
I0508 16:40:12.100603       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-3628299734-0"
I0508 16:40:12.100634       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/wwww-3100399849-0"
E0508 16:40:12.100870       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"12345-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="12345-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="12345-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.100939       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk-3628299734\" not found" "related_resource_kind"="Order" "related_resource_name"="pk-3628299734" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk-3628299734-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.104780       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"sanjayi360http-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="sanjayi360http-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="sanjayi360http-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:12.104928       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/sanjayi360http-3100399849-0"
I0508 16:40:12.104960       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/11-3100399849-0"
E0508 16:40:12.105824       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"testingi360-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="testingi360-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="testingi360-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:12.106869       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/testingi360-3100399849-0"
I0508 16:40:12.106929       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/12-3164661215-0"
I0508 16:40:12.192489       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:12.391693       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-webhook-tls"
I0508 16:40:12.597949       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/test-3100399849-0"
I0508 16:40:12.598012       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/1234-3100399849-0"
E0508 16:40:12.598600       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"test-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="test-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="test-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:12.797848       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"test234-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="test234-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="test234-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:12.798199       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/test234-3100399849-0"
I0508 16:40:12.798282       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/45657-3100399849-0"
I0508 16:40:12.996249       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/wwww-3100399849-0"
I0508 16:40:12.996306       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk1-3628299734-0"
E0508 16:40:12.996747       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"wwww-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="wwww-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="wwww-3100399849-0" "resource_namespace"="sanjaytest"
E0508 16:40:13.197134       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"11-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="11-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="11-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:13.198386       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/11-3100399849-0"
I0508 16:40:13.198462       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0" 
I0508 16:40:13.397534       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/12-3164661215-0"
I0508 16:40:13.397610       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/12345-3100399849-0"
E0508 16:40:13.397657       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"12345-3100399849-0\" not found"
E0508 16:40:13.397664       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"12-3164661215\" not found" "related_resource_kind"="Order" "related_resource_name"="12-3164661215" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="12-3164661215-0" "resource_namespace"="sanjaytest"
I0508 16:40:13.397714       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/12345-3100399849-0"
I0508 16:40:13.397742       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk-3628299734-0"
E0508 16:40:13.397774       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"pk-3628299734-0\" not found"
I0508 16:40:13.397812       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk-3628299734-0"
I0508 16:40:13.397829       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/sanjayi360http-3100399849-0"
E0508 16:40:13.397851       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"sanjayi360http-3100399849-0\" not found"
I0508 16:40:13.397893       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/sanjayi360http-3100399849-0"
I0508 16:40:13.397931       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/testingi360-3100399849-0"
E0508 16:40:13.397949       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"testingi360-3100399849-0\" not found"
I0508 16:40:13.397964       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/testingi360-3100399849-0"
I0508 16:40:13.397979       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/test-3100399849-0"
E0508 16:40:13.398014       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"test-3100399849-0\" not found"
I0508 16:40:13.398029       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/test-3100399849-0"
I0508 16:40:13.398045       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/test234-3100399849-0"
E0508 16:40:13.398092       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"test234-3100399849-0\" not found"
I0508 16:40:13.398107       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/test234-3100399849-0"
I0508 16:40:13.398132       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/wwww-3100399849-0"
E0508 16:40:13.398174       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"wwww-3100399849-0\" not found"
I0508 16:40:13.398188       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/wwww-3100399849-0"
I0508 16:40:13.398201       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/11-3100399849-0"
E0508 16:40:13.398216       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"11-3100399849-0\" not found"
I0508 16:40:13.398227       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/11-3100399849-0"
I0508 16:40:13.398255       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/12-3164661215-0"
E0508 16:40:13.398270       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"12-3164661215-0\" not found"
I0508 16:40:13.398281       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/12-3164661215-0"
E0508 16:40:13.598933       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"1234-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="1234-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="1234-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:13.599195       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/1234-3100399849-0"
I0508 16:40:13.599255       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/1234-3100399849-0"
E0508 16:40:13.599276       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"1234-3100399849-0\" not found"
I0508 16:40:13.599288       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/1234-3100399849-0"
I0508 16:40:13.794861       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/45657-3100399849-0"
I0508 16:40:13.794932       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/45657-3100399849-0"
E0508 16:40:13.794987       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"45657-3100399849-0\" not found"
I0508 16:40:13.795002       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/45657-3100399849-0"
E0508 16:40:13.795080       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"45657-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="45657-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="45657-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:13.995860       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk1-3628299734-0"
I0508 16:40:13.996132       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/pk1-3628299734-0"
E0508 16:40:13.996201       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"pk1-3628299734-0\" not found"
I0508 16:40:13.996217       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/pk1-3628299734-0"
E0508 16:40:13.996287       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"pk1-3628299734\" not found" "related_resource_kind"="Order" "related_resource_name"="pk1-3628299734" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="pk1-3628299734-0" "resource_namespace"="sanjaytest"
E0508 16:40:14.194480       1 controller.go:135] cert-manager/controller/orders/handleOwnedResource "msg"="error getting order referenced by resource" "error"="order.certmanager.k8s.io \"sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849\" not found" "related_resource_kind"="Order" "related_resource_name"="sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Challenge" "resource_name"="sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0" "resource_namespace"="sanjaytest"
I0508 16:40:14.194769       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0"
I0508 16:40:14.194796       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0" 
E0508 16:40:14.194817       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0\" not found"
I0508 16:40:14.194832       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/sanjayi360.dev.xxx.xxxxxx.xxxx-3100399849-0"
I0508 16:40:16.107857       1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/i360"
I0508 16:40:16.107903       1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/i360'
I0508 16:40:16.107907       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:16.107924       1 sync.go:180] Certificate "23467" for ingress "i360" already exists
I0508 16:40:16.107939       1 sync.go:183] Certificate "23467" for ingress "i360" is up to date
I0508 16:40:16.107949       1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/i360"
I0508 16:40:16.107957       1 conditions.go:154] Setting lastTransitionTime for Certificate "23467" condition "Ready" to 2019-05-08 16:40:16.107939137 +0000 UTC m=+4.177253765
E0508 16:40:16.108163       1 pki.go:41] cert-manager/controller/certificates "msg"="failed to retrieve secret" "error"="secret \"23467\" not found" "related_resource_kind"="Secret" "related_resource_name"="23467" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:16.468818       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:16.468865       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:16.469272       1 conditions.go:154] Setting lastTransitionTime for Certificate "23467" condition "Ready" to 2019-05-08 16:40:16.469265287 +0000 UTC m=+4.538579937
I0508 16:40:16.469986       1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/i360'
I0508 16:40:16.470012       1 sync.go:180] Certificate "23467" for ingress "i360" already exists
I0508 16:40:16.470041       1 sync.go:183] Certificate "23467" for ingress "i360" is up to date
I0508 16:40:16.470049       1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/i360"
I0508 16:40:17.023306       1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="letsencrypt-dev"
I0508 16:40:17.023720       1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-developer" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-dev" "resource_namespace"=""
I0508 16:40:17.023800       1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="letsencrypt-dev"
I0508 16:40:17.027106       1 controller.go:150] cert-manager/controller/clusterissuers "level"=0 "msg"="syncing resource" "key"="letsencrypt-prd"
I0508 16:40:17.027506       1 setup.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prd" "resource_namespace"=""
I0508 16:40:17.027575       1 controller.go:156] cert-manager/controller/clusterissuers "level"=0 "msg"="finished processing work item" "key"="letsencrypt-prd"
I0508 16:40:17.027929       1 controller.go:153] cert-manager/controller/issuers "level"=0 "msg"="syncing item" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:17.028612       1 setup.go:73] cert-manager/controller/issuers/setup "level"=0 "msg"="signing CA verified" "related_resource_kind"="Secret" "related_resource_name"="cert-manager-webhook-ca" "related_resource_namespace"="cert-manager" "resource_kind"="Issuer" "resource_name"="cert-manager-webhook-ca" "resource_namespace"="cert-manager"
I0508 16:40:17.028723       1 controller.go:159] cert-manager/controller/issuers "level"=0 "msg"="finished processing work item" "key"="cert-manager/cert-manager-webhook-ca"
I0508 16:40:20.484354       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:20.484609       1 logger.go:38] Calling CreateOrder
E0508 16:40:20.495523       1 controller.go:169] cert-manager/controller/certificates "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.certmanager.k8s.io \"23467\": the object has been modified; please apply your changes to the latest version and try again" "key"="sanjaytest/23467"
I0508 16:40:20.495607       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:20.496067       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:20.504831       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:20.504994       1 controller.go:178] ingress-shim controller: syncing item 'sanjaytest/i360'
I0508 16:40:20.505022       1 sync.go:180] Certificate "23467" for ingress "i360" already exists
I0508 16:40:20.505050       1 sync.go:183] Certificate "23467" for ingress "i360" is up to date
I0508 16:40:20.505060       1 controller.go:184] ingress-shim controller: Finished processing work item "sanjaytest/i360"
I0508 16:40:20.505093       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:20.505575       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:20.505688       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:22.238025       1 logger.go:73] Calling GetAuthorization
I0508 16:40:22.545896       1 logger.go:93] Calling HTTP01ChallengeResponse
I0508 16:40:22.564985       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:22.565605       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:22.565843       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:22.566055       1 sync.go:274] Need to create 1 challenges
I0508 16:40:22.566286       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:22.566445       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:22.571716       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:22.571786       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:22.572005       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:22.572078       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:22.572105       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:22.572343       1 sync.go:274] Need to create 0 challenges
I0508 16:40:22.572369       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:22.572445       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:23.093494       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:23.093675       1 logger.go:68] Calling GetChallenge
I0508 16:40:23.093795       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:23.094041       1 sync.go:274] Need to create 0 challenges
I0508 16:40:23.094158       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:23.094246       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:23.946160       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:23.946646       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:23.946925       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:23.947492       1 sync.go:274] Need to create 0 challenges
I0508 16:40:23.947858       1 logger.go:43] Calling GetOrder
I0508 16:40:23.953137       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:23.953168       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:23.953273       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:24.223671       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:24.224247       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:24.223671       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:24.224445       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:24.224626       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:24.224866       1 sync.go:274] Need to create 0 challenges
I0508 16:40:24.224969       1 logger.go:43] Calling GetOrder
I0508 16:40:24.474826       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:25.495746       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:25.496352       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:25.496518       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:44.489044       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:44.489509       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
E0508 16:40:44.489580       1 controller.go:222] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.certmanager.k8s.io \"23467-3100399849\" not found"
I0508 16:40:44.489596       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:44.503003       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:44.503486       1 logger.go:38] Calling CreateOrder
I0508 16:40:44.504067       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:44.504217       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:44.505314       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:44.505628       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:44.508089       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:44.521265       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:44.521596       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
E0508 16:40:44.521629       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"23467-3100399849-0\" not found"
I0508 16:40:44.521668       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:44.795683       1 logger.go:73] Calling GetAuthorization
I0508 16:40:45.062328       1 logger.go:93] Calling HTTP01ChallengeResponse
I0508 16:40:45.074114       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.074289       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.074180       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:45.074717       1 sync.go:274] Need to create 1 challenges
I0508 16:40:45.075091       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:45.075422       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:45.082612       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.082770       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.082701       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:45.083026       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.083150       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.083384       1 sync.go:274] Need to create 0 challenges
I0508 16:40:45.083545       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:45.083679       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.106143       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.106417       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.106488       1 sync.go:274] Need to create 0 challenges
I0508 16:40:45.106507       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:45.106535       1 logger.go:68] Calling GetChallenge
I0508 16:40:45.106553       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.635769       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.635868       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.635939       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.636633       1 sync.go:274] Need to create 0 challenges
I0508 16:40:45.636652       1 logger.go:43] Calling GetOrder
I0508 16:40:45.645453       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.645508       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.645561       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:45.909917       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.910080       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:45.910253       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467"
I0508 16:40:45.910600       1 sync.go:274] Need to create 0 challenges
I0508 16:40:45.910642       1 logger.go:43] Calling GetOrder
I0508 16:40:45.911084       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="23467-3100399849" "related_resource_namespace"="sanjaytest" "resource_kind"="Certificate" "resource_name"="23467" "resource_namespace"="sanjaytest"
I0508 16:40:45.911438       1 controller.go:173] cert-manager/controller/certificates "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467"
I0508 16:40:46.176286       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.608084       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.608246       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:53.608434       1 sync.go:274] Need to create 0 challenges
I0508 16:40:53.608452       1 logger.go:43] Calling GetOrder
I0508 16:40:53.625839       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:53.625887       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
E0508 16:40:53.625928       1 controller.go:234] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.certmanager.k8s.io \"23467-3100399849-0\" not found"
I0508 16:40:53.625945       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:53.865544       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.865588       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.865775       1 sync.go:274] Need to create 1 challenges
I0508 16:40:53.881986       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:53.882041       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:53.882618       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:53.882666       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.882717       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:53.882958       1 sync.go:274] Need to create 0 challenges
I0508 16:40:53.882980       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:53.883004       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.118155       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.118292       1 logger.go:68] Calling GetChallenge
I0508 16:40:54.119619       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.120026       1 sync.go:274] Need to create 0 challenges
I0508 16:40:54.120045       1 sync.go:323] Waiting for all challenges for order "23467-3100399849" to enter 'valid' state
I0508 16:40:54.120102       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.667009       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.667154       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.667024       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.667651       1 sync.go:274] Need to create 0 challenges
I0508 16:40:54.667669       1 logger.go:43] Calling GetOrder
I0508 16:40:54.673510       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.673814       1 controller.go:209] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.673954       1 controller.go:215] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849-0"
I0508 16:40:54.929015       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.929088       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="sanjaytest/23467-3100399849"
I0508 16:40:54.929435       1 sync.go:274] Need to create 0 challenges
I0508 16:40:54.929461       1 logger.go:43] Calling GetOrder
I0508 16:40:55.194387       1 controller.go:204] cert-manager/controller/orders "level"=0 "msg"="finished processing work item" "key"="sanjaytest/23467-3100399849"
san360 picture san360 on 8 May 2019

It looks like your Challenge resource is in an invalid state. I'm not sure why this isn't triggering the Order to be marked as failed, nor why there is no Reason field to explain why it's in this state:

Status:
  Presented:   false
  Processing:  false
  Reason:
  State:       invalid
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Started  3m    cert-manager  Challenge scheduled for processing

This invalid state implies that the ACME server is responding with invalid when querying the 'authorization URL': http://xxxx-xxxxx.xxx.xxxxxx.xxxx:8080/authz/qGddexbFJCFDfrLHwuRcdwAAAWqYUZXb

Can you query that endpoint yourself and glean any more information as to why the authz is invalid?

Deleting the Order resource will also trigger a new order to be created, which may then be successful (assuming that for some reason cert-manager is 'stuck' and not handling this 'invalid' state, as it should)

munnerz picture munnerz on 10 May 2019

If my understanding is correct this is due to the fact that ACME server gets 404 error for challenge response i.e. GET call to client /.well-known/acme-challenge/ is not successful. i have tried deleting the order resource and it fails at the same point.

san360 picture san360 on 13 May 2019

Hey @san360 - has this issue been resolved now?

munnerz picture munnerz on 3 Jul 2019

I had this issue with "State: Invalid",
and then I realized we have enabled ip restriction within the configmap for ingress
"server-snippets: 'allow 10.0.0.0/8; allow 192.168.0.0/16; deny all;"
So i removed it temporarily, then applied the ingress and it was able to get a letsencypt cert.

aaronrryan picture aaronrryan on 20 Sep 2019

Since there hasn't been any more comments on this I'm going to close. If there are still problems please feel free to reopen.
/close

JoshVanL picture JoshVanL on 16 Oct 2019

@JoshVanL: Closing this issue.

In response to this:

Since there hasn't been any more comments on this I'm going to close. If there are still problems please feel free to reopen.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jetstack-bot picture jetstack-bot on 16 Oct 2019
👎3

@san360 were you able to solve this issue? I am facing the same

knrt10 picture knrt10 on 23 Jun 2020

@san360 were you able to solve this issue? I am facing the same

we gave up as the underlying supported acme version was draft 12 (cert-manager) and we had draft 16 version running.

san360 picture san360 on 24 Jun 2020

we gave up as the underlying supported acme version was draft 12 (cert-manager) and we had draft 16 version running.

I’m not sure this is correct - we aim to match the final RFC (8555). If there are specific divergences from this that you’re aware of, could you open an issue to track these?

munnerz picture munnerz on 24 Jun 2020

Given we test against Pebble in ‘strict mode’, as well as against Let’s Encrypt (which does have a few notable divergences) and both pass without special modification, and are RFC 8555 compliant, I strongly suspect there’s some other issue here and it’s not related to our spec compliance 🙂

munnerz picture munnerz on 24 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dontreboot picture dontreboot  Â·  3Comments
jakubknejzlik picture jakubknejzlik  Â·  3Comments
timblakely picture timblakely  Â·  4Comments
Stono picture Stono  Â·  3Comments
apetheriotis picture apetheriotis  Â·  3Comments