Cert-manager: No configured Challenge Solvers for ACME Prod only
Using the following configuration, I am able to _successfully_ issue certificates using the ACME staging API:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: [email protected]
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- dns01:
cloudflare:
email: [email protected]
apiKeySecretRef:
name: cloudflare-secret
key: apikey
However, when then moving to use the production ACME API, using the configuration below, I am left with the error Failed to determine a valid solver configuration for the set of domains on the order: no configured challenge solvers can be used for this challenge
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: [email protected]
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the DNS-01 challenge provider
solvers:
- dns01:
cloudflare:
email: [email protected]
apiKeySecretRef:
name: cloudflare-secret
key: apikey
FWIW: I did review this upgrade page, which describes the error message I'm receiving, however given that I'm using the updated syntax, I do not believe it applies.
vanpyrzericj
All 15 comments
Same problem here, only we're using http instead of dns
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: [email protected]
privateKeySecretRef:
name: letsencrypt-prod
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx
No challenges are created, the order does not complete and the cert-manager shows logs about missing challenges
k8s logs cert-manager-745fcfddc4-8kb9q -n cert-manager
E0207 16:28:12.945941 1 sync.go:111] cert-manager/controller/orders "msg"="Failed to determine the list of Challenge resources needed for the Order" "error"="no configured challenge solvers can be used for this challenge" "resource_kind"="Order" "resource_name"="wildcard-certv2-2132113776-130921517"
I also see an error message that says there is a missing annotation in my Ingress:
I0207 16:43:53.262599 1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="XXXXl" "resource_namespace"="XXXX"
My Ingress resource:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
name: myIngress
namespace: myNamespace
spec:
rules:
- host: '*.my.site.com'
http:
paths:
- backend:
serviceName: myService
servicePort: 8080
tls:
- hosts:
- '*.my.site.com'
secretName: wildcard-cert
status:
loadBalancer:
ingress:
- {}
pwolthausen
on 7 Feb 2020
I am also hitting this issue. I am wondering if we hit the rate-limit of let's encrypt production api.
nchowinf
on 12 Feb 2020
Same problem with the latest v0.13.0 helm chart. This works with staging issuer, but not production:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: default
spec:
acme:
email: [email protected]
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
route53:
region: us-west-2
hostedZoneID: [READACTED]
role: arn:aws:iam::[READACTED]:role/[READACTED]
The order itself:
Name: ingress-tls-cert-xxxxx-xxxxxx
Namespace: default
Annotations: cert-manager.io/certificate-name: ingress-tls-cert
cert-manager.io/private-key-secret-name: ingress-tls-cert
API Version: acme.cert-manager.io/v1alpha2
Kind: Order
Metadata:
Creation Timestamp: 2020-02-12T18:24:21Z
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: CertificateRequest
Name: ingress-tls-cert-xxxxxxxxxx
UID: xxxxxxxxx-4ef0-42a7-b202-xxxxxxxxx
Resource Version: 42775452
Self Link: /apis/acme.cert-manager.io/v1alpha2/namespaces/default/orders/ingress-tls-cert-xxxxxx-xxxxxxxxxxxx
UID: xxxxxx-a72d-4df8-ab12-xxxxxxxx
Spec:
Csr: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ21EQ0NBWUFDQVFBd0Z6RVZNQk1HQTFVRUNoTU1ZMlZ5ZEMxdFlXNWhaMlZ5TUlJQklqQU5CZ2txaGtpRwo5dzBCQVFFRkFBT0NBUThBTUlJQkN....RVUVTVC0tLS0tCg==
Dns Names:
domain1.org
*.domain1.org
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Status:
Authorizations:
Challenges:
Token: xxxxxxxxxxxxxxxxxxxxx
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxx/xxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxx
Wildcard: false
Challenges:
Token: xxxxxxxxxxx
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxx
Wildcard: true
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/xxxxxxxxxxx/xxxxxxxxxxx
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxxxx/xxxxxxxxxxx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Solver 23m cert-manager Failed to determine a valid solver configuration for the set of domains on the Order: no configured challenge solvers can be used for this challenge
The error reported by the order:
```
Failed to determine a valid solver configuration
for the set of domains on the Order: no configured challenge solvers can be used for
this challenge
````
demisx
on 12 Feb 2020
I've solved the problem on my end. My issue was that I was switching from http01 to dns01 challenge in prod and the Order kept creating with http-01 challenge at the top of the list:
$ kubectl describe order
---
...
Status:
Authorizations:
Challenges:
Token: xxxxxxxxxxxxxxxxxxxx
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxxxxxxxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxxx
Wildcard: false
Challenges:
Token: xxxxxxxxxxxxxxxxxxxx
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxxxxxxxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxxx
Wildcard: true
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/xxxxxxxxxxxxxxxxxxxx
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxxxxxxxxxxxxx
So, I have deleted the letsencrypt secrets and the current order and then the certificate got reissued within next 5 mins or so.
kubectl -n cert-manager delete secret letsencrypt-prod
kubectl -n cert-manager delete secret letsencrypt-stg
kubectl delete order ingress-tls-cert-xxxxx-xxxxxx
If you are switching from http01 to dns01 and things still don't work, I recommend removing all resources related to cert-manager (helm release, crd, secrets, orders, crs, etc.) and then redeploying it clean. There is a good chance something left behind is interfering with your new setup. That was my case at least.
demisx
on 12 Feb 2020
You shouldn't need to completely uninstall cert-manager when changing challenge types - deleting the pending/failed CertificateRequest resources should trigger the process to restart, including selecting the appropriate solver.
You can read more information on how to configure multiple different solvers on a single Issuer/ClusterIssuer here: https://cert-manager.io/docs/configuration/acme/#adding-multiple-solver-types
That said, your included status block here:
Status:
Authorizations:
Challenges:
Token: xxxxxxxxxxxxxxxxxxxx
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxxxxxxxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxxx
Wildcard: false
Challenges:
Token: xxxxxxxxxxxxxxxxxxxx
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/xxxxxxxxxxxxxxxxxxxx
Identifier: domain1.org
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxxxxxxxxxxxxxxxxx
Wildcard: true
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/xxxxxxxxxxxxxxxxxxxx
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/xxxxxxxxxxxxxxxxxxxx
appears to show that only the HTTP01 challenge type is being allowed for the non-wildcard version of the domains you are validating. This is unexpected as DNS01 validation should also be allowed here... Ultimately that's what is causing your issue I think, and in a quick test locally I'm able to reproduce a similar status output (however in my case, as I also have a configured HTTP01 solver it succeeds in obtaining the certificate, but performs one challenge via HTTP01 and the other via DNS01).
/priority important-soon
/kind bug
/area acme
/assign
munnerz
on 23 Apr 2020
With further testing, when changing my Certificate to reference an Issuer pointing to Let's Encrypt's staging environment, I am seeing different challenges offered:
authorizations:
- challenges:
- token: xxx
type: dns-01
url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/xxx/xxx
identifier: example.com
url: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/xxx
wildcard: true
- challenges:
- token: xxx
type: http-01
url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/xxx/xxx
- token: xxx
type: dns-01
url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/xxx/xxx
- token: xxx
type: tls-alpn-01
url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/xxx/xxx
identifier: example.com
url: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/xxx
wildcard: false
I'm not certain why there's different results here.. @jsha do you know any reason why the production endpoint would not be allowing DNS01/TLS-ALPN challenges for the non-wildcard version of a domain, when the staging environment does?
munnerz
on 23 Apr 2020
My guess is that you're looking at a challenge that is either already
invalid from a previous attempt, or already valid (via authz reuse). As of
a few months ago, when a challenge is attempted, we remove all the other
challenges from that authz.
jsha
on 23 Apr 2020
/unassign
/cc @JoshVanL
munnerz
on 24 Apr 2020
My guess is that you're looking at a challenge that is either already
invalid from a previous attempt, or already valid (via authz reuse)
I think you are right. We don't actually check the Status field when we first fetch the Authorization object, which means we will always attempt to create a Challenge resource even if the Authorization is already valid.
As of a few months ago, when a challenge is attempted, we remove all the other challenges from that authz.
This is probably why we didn't originally see this when implementing this and making it the default behaviour in v0.11 馃槄
This is something we want to fix in the short term for v0.15 and I'm working on a patch to that effect (simply recording the initialState so we can skip creating Challenges if they aren't required).
I think it has raised the need for us to reconsider our Challenge resource type. It'd be better if we had an Authorization resource type which could perform this check and immediately mark itself as valid. This would also make it clearer as to what is going wrong when a 'solving configuration' actually cannot be selected too. This will require a bit more work, and is ultimately a (minor) breaking API change. I think we should target this before cutting v1beta1, and once done, I think this is a good signal that we can push ahead with our beta API release.
So..
v0.15: fixes this issue
v0.16: _properly_ fixes this issue with some underlying improvements to the way we do things to make all this better in future 馃槃
@jsha you mention invalid from a previous attempt here - if the Challenge is now invalid from a previous attempt, presumably re-creating the Order resource will then return a new set of Authorization objects which do contain the failed challenge type, so the challenge can actually be retried in the event of a real failure (i.e. due to a false-positive on our self check due to some external network configuration)
munnerz
on 1 May 2020
I was seeing symptoms similar to the ones described above using v0.14.1 deployed via helm.
Orders were created and remain in pending, a secret was issued using ACME signer, and no challenge resource was every created.
Using DNS proof via Route53, no http proof.
Via helm, I upgraded to v0.15.0-beta.0 and the challenge was created properly and certificates issued as expected.
:+1:
warmfusion
on 2 May 2020
We are having issue with letsencrypt prod, the tls-secret that it creates should not have tls.crt empty, ca.crt shows empty too in working clusters so I think that may be fine. However, the empty tls.crt field is definitively the one causing Nginx to use fake certificate.
cluster-issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
Install the cert manager helm chart
helm install
cert-manager
--namespace ingress-public
--version v0.13.0
jetstack/cert-manager
Help needed!
Raghubalam
on 1 Sep 2020
Having the same issue on chart version v1.0.2. I tried the fix @demisx but it didn't help. Any suggestions?
benjick
on 6 Oct 2020
I'm facing also the same problem on chart version V1.0.2 .
using DNS01 Challenge Provider
kubernetes 1.13
Any suggestions?
yassinetimdouine
on 16 Oct 2020
V1.1.0 I completely deleted everything, but the error is the same
kubectl describe orders --namespace dev dev-tls-pljng-3804290353
Name: dev-tls-pljng-3804290353
Namespace: dev
Labels: <none>
Annotations: cert-manager.io/certificate-name: dev-tls
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: dev-tls-ll6lh
API Version: acme.cert-manager.io/v1
Kind: Order
Metadata:
Creation Timestamp: 2020-12-15T01:33:19Z
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:ownerReferences:
.:
k:{"uid":"0ee6deb4-ef21-48ee-9555-48c16c8bc732"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:status:
.:
f:authorizations:
f:finalizeURL:
f:state:
f:url:
Manager: controller
Operation: Update
Time: 2020-12-15T01:33:21Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: CertificateRequest
Name: dev-tls-pljng
UID: 0ee6deb4-ef21-48ee-9555-48c16c8bc732
Resource Version: 2374449
Self Link: /apis/acme.cert-manager.io/v1/namespaces/dev/orders/dev-tls-pljng-3804290353
UID: e88fe97e-33fd-487a-8b37-a2bd9c867ea9
Spec:
Dns Names:
newdev.domen.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2tUQ0NBWGtDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU9uRQpOOHJaZ2s0ai8vY2YrWEZHcXcydzU1RmhJSFZKSEQ0Z3BYRlBQSTRMN0ZzZlBHcWk1YnZrbE9KWFozUXVteGNICmhVQWFKSk91Y1RtYVpDeHF6VUFWOWoyNmVJdHgrcnZNWFREWWhyblluYzVrbktNV3dxWEdKWE9VWWljV1NRczAKeHhzdy9tUmNuVUQ1WU1mMjJVbDhwbUl6SFQyWFFGenh2M05MbGx3eUp1TGdqdkxvZHp3cE8rS3RnL0tRSzl2WgpmMXRtNm51MFBYWHV0c2tpcjhia2FEOUFKYWdEVnUxVVpWMEVKMnpvMGtnVmFGTDlyWnUzZGNJNUZxRCtIWHA3CjNzMXhDRnZJR01tWWVEYzlHVTk1NkhRUkxOUzBwQWRkUFROaTM1WTNuWk5LNWpkOWdYdWswRmNMNDNvdS9YMmsKeEcvS1RpWFY1TFpkZW1Gemw3VUNBd0VBQWFCTU1Fb0dDU3FHU0liM0RRRUpEakU5TURzd0xBWURWUjBSQkNVdwpJNEloYm1WM1pHVjJMbk5qYjNWMGN6TnJMbUZ3YVM1ellXNXVZV052WkdVdVkyOXRNQXNHQTFVZER3UUVBd0lGCm9EQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFwck9nTWFpMkJNaXd2aUozeEdyaXRpL1UzVUN5ZkRwOElyTWUKYVF3KzYvY2RtOG9KdkZWTHpYMW5TRWJ0TU0raVB1d3lLTGwrWFlSeFd0M2pKdFFzenNrK0hGU3VjQzlhTjZKQQpCWVBreTNxUjdEL2dWWlpmMGhKT1l3cWVIc3JZMVYwcnpOdGlXcnd2VUFNdDMzbGlKMmZLNldQMTF5d3NXMVYzCjBGV05XelV6R0RJejRoZjNFRUlLTnVzRXBzTVlTMS9rVVVMRHlPT1dYUTNEZEJJZFdVUjNDYnU1VFIwVVp6dFcKQVNPZkxnK0xnRHZXSnpLVGtDZ1JKTnlQSElPQzFPTlBIemgzZDFteWNFczc4d2dMK1ZmUGFMMXdjV2U1YkErdQpuN1VnV0w0RUtQMmtMQkprSEd3VmRUdHh5dDlkNHZmWFJaY2x0VENLUWZEcGZDcUhLQT09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Status:
Authorizations:
Challenges:
Token: WfQjHCsr9UKPEDpG2uMZDC76fAK01bATE5JweB_958g
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9317936764/RGEQVA
Token: WfQjHCsr9UKPEDpG2uMZDC76fAK01bATE5JweB_958g
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9317936764/aLrDrA
Token: WfQjHCsr9UKPEDpG2uMZDC76fAK01bATE5JweB_958g
Type: tls-alpn-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/9317936764/h7RVUw
Identifier: newdev.domen.com
Initial State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/9317936764
Wildcard: false
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/106371774/6747042986
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/106371774/6747042986
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Solver 54s cert-manager Failed to determine a valid solver configuration for the set of domains on the Order: no configured challenge solvers can be used for this challenge
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: [email protected]
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
cloudflare:
email: [email protected]
apiKeySecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsNames:
- '*.host.com'
md5-b3d61684e0262634649c41c92a310270
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: dev
name: dev-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
nginx.ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/enable-access-log: "true"
spec:
tls:
- hosts:
- newdev.host.com
# - scouts.idap.pro
secretName: dev-tls
rules:
- host: newdev.host.com
http:
paths:
- path: /
backend:
serviceName: scouts-frontend-dev
servicePort: 80
Titan7777
on 15 Dec 2020
Same Issue, not shure what I did wrong. Cert-Manager 1.1.0
ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: 'letsencryt-prod'
spec:
acme:
email: [email protected]
preferredChain: ''
privateKeySecretRef:
name: letsencrypt-prod
server: 'https://acme-v02.api.letsencrypt.org/directory'
solvers:
- dns01:
cloudflare:
apiKeySecretRef:
key: apikey
name: cloudflare-apikey-secret
email: [email protected]
selector:
dnsNames:
- 'mydomain.com'
- '*.mydomain.com'
- 'mydomain.xyz'
- '*.mydomain.xyz'
matchLabels:
use-cloudflare-solver: 'true'
Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: 'mycert'
namespace: cattle-system
spec:
dnsNames:
- 'subdomain.mydomain.com'
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
secretName: mycertificate
MSandro
on 10 Jan 2021
Related issues
jbartus
路
4Comments
kragniz
路
4Comments
caiobegotti
路
4Comments
f-f
路
4Comments
matthew-muscat
路
4Comments
Most helpful comment
I've solved the problem on my end. My issue was that I was switching from
http01todns01challenge in prod and the Order kept creating withhttp-01challenge at the top of the list:So, I have deleted the letsencrypt secrets and the current order and then the certificate got reissued within next 5 mins or so.
If you are switching from http01 to dns01 and things still don't work, I recommend removing all resources related to
cert-manager(helm release, crd, secrets, orders, crs, etc.) and then redeploying it clean. There is a good chance something left behind is interfering with your new setup. That was my case at least.