Cert-manager: Accept both HTTP and HTTPS requests in acmesolver's listener

[sillva]

Description: I have an ALB ingress that propagates the load to backend as HTTPS, with the following annotation.

"alb.ingress.kubernetes.io/backend-protocol": "HTTPS"

The issue here is that when cert-manager tries to renew the certificate, it does correctly patch the ingress and spawns both service and pod. However, it expects HTTP connections so that I end up with a bad gateway obviously.

Is there a way to make the solver pod also expect HTTPS connections?

I looked at the documentation but it seems one is only allowed to changed small details of the pod template.

spec *ACMEChallengeSolverHTTP01IngressPodSpec* 
PodSpec defines overrides for the HTTP01 challenge solver pod. Only the 'nodeSelector', 'affinity' and 'tolerations' fields are supported currently. All other fields will be ignored.

Suggestions are welcomed.

[munnerz]

We'd need to modify the actual listener to allow for serving over HTTPS as well as HTTP - this is fairly reasonable in order to overcome limitations in Ingress controller configuration, and we could use a self-signed certificate to serve with.

Are there any requirements around the certificate's validity/signer? Would it be okay for the acmesolver to serve with an untrusted certificate?

Additionally, I think we'll want to come up with some way to serve both HTTP AND HTTPS on port 80 AND 443 (i.e. serve both on both) - I'm pretty sure this is possible as I've seen other software doing it, it may just require some extra magic in our listener. This should make it possible for you to use this annotation without any additional changes to the podTemplate 😄

[sillva]

I do not see any issues in having an untrusted certificate on the acme resolver.

[mcambal]

Same issue here with Azure Appgw Ingress Controller. This change would be highly appreciate. We can't create and renew certificates then.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[InamAli144]

Same issue here.

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[munnerz]

/remove-lifecycle rotten

[irbekrm]

/assign

[maelvls]

Update: my comment is worthless... I did not pay attention to the actual question:

Is there a way to make the solver pod also expect HTTPS connections?

Apologies for my side-tracked answer!

My side-tracked comment

Would it be possible to use the alb.ingress.kubernetes.io/actions.* feature of the ALB controller?

In order for the ingress-shim to support the ALB controller's redirect feature, we would have to add a way to configure additional rules for the ingress. I detail what I mean by that in the remaining of this comment.

Supporting ALB controller HTTPS redirect for the challenge Ingress object

Usually, the ingress created for the challenge would look like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  labels:
    acme.cert-manager.io/http-domain: "446694490"
    acme.cert-manager.io/http-token: "930942483"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-cj5v2
  namespace: default
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-4gn4q
          servicePort: 8089
        path: /.well-known/acme-challenge/U-egeadUm0n-HqQAfk94FCHSLlcHXaye2ZJ0kBy-ql0
        pathType: ImplementationSpecific

If we wanted to support the ALB controller HTTPS redirect using annotations, we would use some annotations (from this example):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
  labels:
    acme.cert-manager.io/http-domain: "446694490"
    acme.cert-manager.io/http-token: "930942483"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-cj5v2
  namespace: default
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-4gn4q
          servicePort: 8089
        path: /.well-known/acme-challenge/U-egeadUm0n-HqQAfk94FCHSLlcHXaye2ZJ0kBy-ql0
        pathType: ImplementationSpecific
      - path: /*                      # ⚠️ Needed for the annotation to work
        backend:                      # ⚠️
          serviceName: ssl-redirect   # ⚠️
          servicePort: use-annotation # ⚠️

Would it make sense to support additional rules?

Now, would it make sense for cert-manager to support additional rules similarly as to how it supports additional annotations for the challenge ingress object?

If you ask me, I would say no... additional rules feels "too much" customization.

I think the ALB controller should at least have a simple annotation for HTTP → HTTPS redirection, like what the GKE ingress controller does as detailed by @thockin in https://github.com/jetstack/cert-manager/issues/3549. The GKE controller annotation looks like this:

kubernetes.io/ingress.allow-http: "false"

I guess we could ping the ALB controller team to ask if they plan on supporting such a simple annotation?

[munnerz]

We should not have any ingress-controller specific behaviour IMO. Allowing users to provide additional annotations to be added to objects we create _does_ make sense (similar to pods).

That said, I don't think what you've described here is actually relevant to this issue. The annotation in question is "alb.ingress.kubernetes.io/backend-protocol": "HTTPS" - which tells Azure's load balancer to send HTTPS requests to the backend, rather than expecting to speak HTTP to the backend.

The issue here is that the acmesolver only listens with HTTP, so when an HTTPS request is sent to it, naturally clients will error and fail.

As I noted in the comment above - we can resolve this ticket by having the http.Server used in the acmesolver binary also serve HTTPS as well as HTTP, both on the same port (by inspecting the first few bytes of an incoming request and detecting if it is an HTTPS request, and handling it appropriately if so).

[munnerz]

Regards to your actual comment though - I'm not sure why it'd be useful for us to do this given we don't really mind if the user (Let's Encrypt) requests the challenge endpoint over HTTP or HTTPS - with the way we do things right now, the Ingress will not configure TLS, so the LB will listen on port 80. We could redirect them to 443, but it'd bring no benefits.

Having TLS used for backend connections from the load balancer (i.e. this ticket) _does_ have value if you need to require all internal traffic to be encrypted. But from Let's Encrypt<>LB I don't think so (given we'd only be using a self signed certificate anyway, and Let's Encrypt doesn't actually verify this certificate either).

[irbekrm]

We had a short discussion about this issue earlier with the team- there is a concern that having the solver accept HTTPS traffic would be against the ACME spec (even though it would only be between the load balancer and the backend). Also (and related to the first point) that someone might see our implementation of the solver that accepts HTTPS and reuse it in a different scenario (i.e without an Ingress)

[munnerz]

I don't think it's against the ACME spec at all - how traffic traverses from the LB to the backend isn't really a concern of Let's Encrypt. If it _were_ against the spec, then any kind of service mesh would be problematic to run too.

that someone might see our implementation of the solver that accepts HTTPS and reuse it in a different scenario (i.e without an Ingress)

I'm also not too sure on this one either - why would someone choose to use it without an Ingress because it accepts HTTPS traffic? Let's Encrypt specifically requires port 80 to be listening and will make the initial request to the HTTP port only (and follow redirects to HTTPS if returned).

[irbekrm]

Linking a discussion about this issue on cert-manager Kubernetes Slack channel.

Note: testing this could be problematic as we currently only run e2e tests on GCP whilst the issue is linked to ALB/Azure load balancer.

[munnerz]

I think testing the actual behaviour of "can make an HTTPS and HTTP request to the same endpoint" would probably suffice - we don't really provide a contract around whether an ingress controller will work - rather, we build features that various ingress controllers can work with when configured by users.. it's a fine line 😅