Cert-manager: Use DigiCert ACME endpoint

[rvennam-lbg]

Is your feature request related to a problem? Please describe.
We are doing a POC with new DigiCert ACME (beta) implementation, but not able to use cert-manager with DigiCert.

Describe the solution you'd like
DigiCert as per their current ACME implementation does the domain validation in their CertCentral portal and not look for any additional authorizations. When an Issuer / ClusterIssuer is created with no solver tied to it for authorization it fails during certificate request.

Describe alternatives you've considered
We reached out to DigiCert regarding this authorization check and are awaiting a response.

Additional context
Is this a feature which can be accommodated by cert-manger?

Environment details (if applicable):

• Kubernetes version (e.g. v1.10.2): v1.13 & v1.14
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): GKE
• cert-manager version (e.g. v0.4.0): v0.12.0
• Install method (e.g. helm or static manifests): Helm

/kind feature

[munnerz]

So to confirm, when the CreateOrder endpoint is called, an Order resource with no Challenges specified on it is created?

What will the response look like if a user created an Order but does not have appropriate authorization? Do you return a challenge object? If so, what is the type of this challenge object?

[rvennam-lbg]

@munnerz From what I understand of the DigiCert ACME implementation they have a middle layer called EnrollMe, which looks for the domains already registered to the requesters account i.e. the ACME endpoint generated from the CertCentral portal and then return a signed certificate back.

From what I was informed there is no authorization challenge as the endpoint created is bound to the domain and it should be considered as a secret as anyone with the valid endpoint can requests those certificates, which I get is not as secure as the existing implementation.

[munnerz]

Hm, based on the document here: https://docs.digicert.com/certificate-tools/acme-user-guide/ it appears DigiCert treat the actual directory URL as a secret value and otherwise perform no auth. I'm not quite sure how the account private key comes into play here either...

This seems a bit weird and off-spec. We will endeavour to support the RFC specification as much as possible. If DigiCert implement an endpoint that _isn't_ RFC compliant, they are a bit stuck in the water (as that's the whole point of an RFC :)). If we can identify more specifically what is an issue here, it may be possible that it is compatible, but it's hard to tell without using the service myself.

cc @cpu @jsha is this something you are aware of? I'm not sure myself, but it seems like there's something a bit off going on here. Not sure if it is a concern or something noteworthy :)

[cpu]

_(FYI: Speaking just as an individual here. I don't work for Let's Encrypt/ISRG anymore)_

@munnerz I've seen that docs page about the directory URL before and it lacks enough detail for me to say confidently what's going on but I find it puzzling/troubling. Overall it does give the impression the directory URL is a capability URL or some other kind of secret bound to out of band domain authorizations. If that's true I think it represents a large design mistake. Most ACME clients log the directory URL liberally (or keep it in world readable config files, etc) and there's no indication it should be anything but a well known public value. This scenario is exactly what external account binding is for (for example, see how Sectigo implemented RFC 8555 integration for their CA). Overall this entire implementation seems difficult to automate, perhaps diluting one of the primary goals of the protocol :-/

It would help to see more detailed logs from @rvennam-lbg but I would expect an order to always have an "authorizations" array of URLs (per https://tools.ietf.org/html/rfc8555#section-7.1.3) it is a required field. I _think_ specifying an empty array would be fine, provided the order is returned with "status": "valid", e.g. indicating the CA requires no challenges. In my opinion it would be better in this case to return authorizations for each of the identifiers in the order, each with "status": "valid" as well (and an empty "challenges" array) but I don't think RFC 8555 is strict enough to require this.

Hope that helps!

[rvennam-lbg]

@munnerz I did set cert-manager to debug level 5 but am not seeing the exact requests is there anything more with respect to getting the information requested above?

[rvennam-lbg]

@munnerz / @cpu Apart from this issue, do you know if cert-manager was used with any commercial vendors with their ACME implementations currently. This might help in understanding the route for us going ahead.

[munnerz]

We have confirmed that the Sectigo ACME endpoint, which utilises 'External Account Bindings' in order to associate ACME accounts with corresponding customer accounts, does work :)

@munnerz I did set cert-manager to debug level 5 but am not seeing the exact requests is there anything more with respect to getting the information requested above?

There isn't (currently) a way to force the request/response bodies for the ACME client to be logged - this may be a good addition to help debugging this sort of thing in future, but it'll require some changes to the underlying ACME library we use (in golang.org/x/crypto/acme).

Hope that helps!

[rvennam-lbg]

Thanks @munnerz on the integration we are working with DigiCert for fixes and will keep this thread updated if you will like me to (or) close this issue

[gregsidelinger]

I'm seeing the same behavior with Keyfactors ACME endpoint. The response does not have any challenges as it uses a whitelist and externalAccountBinding to know if it can issue the cert.

When I use certbot to talk to the same endpoint it is happy to finish the order process without any challenges. Here is an example of how I'm using certbot to order the cert.

certbot certonly --standalone --server https://keyfactor.example.com/ACME -d sample.example.com

[munnerz]

@gregsidelinger can you provide log output or any information about the error/issue you're seeing here?

[gregsidelinger]

Here are the logs for the solver being picked.

I0324 19:22:31.659655       1 controller.go:144] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/my-app-tls-1453198217-754478407"
I0324 19:22:31.659672       1 controller.go:138] cert-manager/controller/orders "msg"="syncing item" "key"="cert-manager/my-app-tls-1453198217-754478407"
I0324 19:22:31.659687       1 metrics.go:385] cert-manager/metrics "msg"="incrementing controller sync call count"  "controllerName"="orders"
I0324 19:22:31.659761       1 sync.go:130] cert-manager/controller/certificaterequests-issuer-acme "msg"="invoking sign function as existing certificate does not exist" "resource_kind"="CertificateRequest" "resource_name"="my-app-tls-1453198217" "resource_namespace"="cert-manager"
I0324 19:22:31.659790       1 sync.go:108] cert-manager/controller/orders "msg"="Computing list of Challenge resources that need to exist to complete this Order" "resource_kind"="Order" "resource_name"="my-app-tls-1453198217-754478407" "resource_namespace"="cert-manager"
I0324 19:22:31.659810       1 util.go:137] cert-manager/controller/orders/challengeSpecForAuthorization "msg"="cannot use solver as the ACME authorization does not allow solvers of this type" "resource_kind"="Order" "resource_name"="my-app-tls-1453198217-754478407" "resource_namespace"="cert-manager"
E0324 19:22:31.659830       1 sync.go:111] cert-manager/controller/orders "msg"="Failed to determine the list of Challenge resources needed for the Order" "error"="no configured challenge solvers can be used for this challenge" "resource_kind"="Order" "resource_name"="my-app-tls-1453198217-754478407" "resource_namespace"="cert-manager"
I0324 19:22:31.659863       1 sync.go:53] cert-manager/controller/orders "msg"="skipping updating resource as new status == existing status" "resource_kind"="Order" "resource_name"="my-app-tls-1453198217-754478407" "resource_namespace"="cert-manager"
I0324 19:22:31.659874       1 controller.go:144] cert-manager/controller/orders "msg"="finished processing work item" "key"="cert-manager/my-app-tls-1453198217-754478407"
I0324 19:22:31.659831       1 acme.go:201] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="acme Order resource is not in a ready state, waiting..." "related_resource_kind"="Order" "related_resource_name"="my-app-tls-1453198217-754478407" "related_resource_namespace"="cert-manager" "resource_kind"="CertificateRequest" "resource_name"="my-app-tls-1453198217" "resource_namespace"="cert-manager"
I0324 19:22:31.659921       1 event.go:281] Event(v1.ObjectReference{Kind:"Order", Namespace:"cert-manager", Name:"my-app-tls-1453198217-754478407", UID:"ce8e0ad7-6e04-11ea-b79b-0242ac110008", APIVersion:"acme.cert-manager.io/v1alpha2", ResourceVersion:"89610027", FieldPath:""}): type: 'Warning' reason: 'Solver' Failed to determine a valid solver configuration for the set of domains on the Order: no configured challenge solvers can be used for this challenge

Here is my sanitized issuer. I replace real values with example.com

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: keyfactor
spec:
  acme:
    email: [email protected]
    server: https://keyfactor.example.com/ACME
    skipTLSVerify: True
    externalAccountBinding:
      keyID: KEY
      keySecretRef:
        name: acme
        key: secret
      keyAlgorithm: HS256
    privateKeySecretRef:
      name: acme-key
    # Keyfactor uses a whitelist of what it will approve so this may not be needed
    # Enable the HTTP-01 challenge provider
    solvers:
    # An empty 'selector' means that this solver matches all domains
    - selector: {}
      http01:
        ingress:
          class: nginx

Here is my cert

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: my-app-tls
spec:
  secretName: my-app-tls
  issuerRef:
    name: keyfactor
    kind: ClusterIssuer
  dnsNames:
  - sample.example.com

I've played around with and without different solvers. Was hoping that by having the ingress solver things would work but that was before I dug into the challenge processing code.

[munnerz]

Ah, I think this could be related to https://github.com/jetstack/cert-manager/issues/2494#issuecomment-618503293 - can you provide the output of kubectl get order -o yaml whilst an Order is in progress?

[munnerz]

@gregsidelinger any update? Is this issue resolved when using v0.15.0?

[gregsidelinger]

@gregsidelinger any update? Is this issue resolved when using v0.15.0?

I have not tried with 0.15. I actually ended up creating a custom issuer to talk to the keyfactor api and sign a cert. The team that manages the keyfactor was going to want extra metadata about the cert passed to keyfactor which ACME will not support so I'm not sure how far beyond a POC I would have gotten.