Cert-manager: Support VirtualService resources for HTTP01 solving for better Istio support

[sneko]

Bugs should be filed for issues encountered whilst operating cert-manager.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.

Describe the bug:
Hi everyone,

I'm stuggling a new time with renewing my certificates while using Istio. My ACME solvers are all created (pods+services+ingresses) except that ingresses are not reachable. I can see they don't have any IP linked to them, but it seems normal since that's the Istio ingressgateway that is supposed to "simulate" them.

My certificate uses:

    - http01:
        ingressClass: istio

and even by enabling k8sIngress property when reinstalling Istio:

global:
  k8sIngress:
    enabled: true
    gatewayName: ingressgateway
    enableHttps: false

... those solvers are still unreachable.

I'm looking for hours on forums, but there is no clear explanation how to use Istio with cert-manager for HTTP01. What can be find is about using DNS01 but I cannot in my case.

I guess I miss something?

Thank you,

Expected behaviour:
Having my ACME solvers reachable from the outside.

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.10.x
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): Kops
• cert-manager version (e.g. v0.4.0): v0.7.2
• Install method (e.g. helm or static manifests): Helm

/kind bug

[rfevang]

Some custom way of directing traffic would be sufficient I think, right now cert-manager makes ingress resources, but they don't work well with Isito's new traffic management structures. Maybe a template type thing could work as an option in addition to ingressClass and ingressName?

A VirtualService is needed for Istio, right now I don't see any way of making one myself, as the name of the service to redirect to is unknown. Actually just adding an option to specify the name would likely be enough, but I don't know if that's feasible?

How about allowing a ConfigMap that specifies how to generate the ingress resource?

[munnerz]

I think cert-manager itself can probably 'learn' Istio a bit, similar to how we support multiple DNS01 providers in our own codebase.

If someone familiar with Istio could document exactly what needs creating in order to replicate our Ingress options into Istio-gateway equivalents (it'd be awesome if you could provide a set of example manifests too 😄) we can then begin drawing up what the API needs to look like within cert-manager.

That said, I think this issue is specifically about Istio's k8sIngress option, which allows Istio to understand Ingress resources instead of just Gateway resources, which should technically work today (although I've not tried it myself).

[spekr]

This is an excellent read if you want Cert-manager to work with Istio. https://medium.com/@gregoire.waymel/istio-cert-manager-lets-encrypt-demystified-c1cbed011d67
I have not been able to get it working though. I fear it has to do with the istio-sidecar being absent in the acme-challenge-solver pods. (We have mtls enabled.) This annotation is hardcoded in certmanager. Hope the article helps you.

[rfevang]

Istio wants a VirtualService and Gateway instead of Ingress. I set up a test issuer and certificate like this:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    email: [email protected]
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: test-issuer-account-key
    http01: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-cert
  namespace: cert-test
spec:
  secretName: testing-cert
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: test.mydomain.com
  dnsNames:
  - test.mydomain.com
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - test.mydomain.com

cert-manager then creates a challenge pod and service. It also creates an Ingress resource to route to the service like this:

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: istio
      nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
    creationTimestamp: 2019-05-21T13:30:39Z
    generateName: cm-acme-http-solver-
    generation: 1
    labels:
      certmanager.k8s.io/acme-http-domain: "2182023630"
      certmanager.k8s.io/acme-http-token: "481627430"
    name: cm-acme-http-solver-jkbn8
    namespace: cert-test
    ownerReferences:
    - apiVersion: certmanager.k8s.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Challenge
      name: test-cert-3923790336-0
      uid: 9f1de09f-7bcc-11e9-962f-42010af0004a
    resourceVersion: "221800953"
    selfLink: /apis/extensions/v1beta1/namespaces/cert-test/ingresses/cm-acme-http-solver-jkbn8
    uid: 9fad9db5-7bcc-11e9-962f-42010af0004a
  spec:
    rules:
    - host: test.mydomain.com
      http:
        paths:
        - backend:
            serviceName: cm-acme-http-solver-5txgr
            servicePort: 8089
          path: /.well-known/acme-challenge/1-SB0YcK7xC4bGaEfxb9-Sqf0_Ia-TVCIDhv_23-1cs
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

To convert that ingress to something that works with standard Istio, we need a HTTP gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public-http
  namespace: cert-test
spec:
  selector:
    app: istio-ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
     hosts:
     - test.mydomain.com

A lot of Istio deployments already have an HTTP gateway defined, in that case cert-manager need to use the existing one, in a similar way that it allows ingresses to reuse an ingress controller by specifying ingressClass.

The other thing needed (and the Ingress equivalent) is a VirtualService. I wrote one copying the generated names from the Ingress above, it looks like this:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: cm-acme-http-solver-jkbn8
  namespace: cert-test
spec:
  hosts:
  - test.mydomain.com
  gateways:
  - public-http
  http:
  - match:
    - uri:
        exact: /.well-known/acme-challenge/1-SB0YcK7xC4bGaEfxb9-Sqf0_Ia-TVCIDhv_23-1cs
    route:
    - weight: 100
      destination:
        host: cm-acme-http-solver-5txgr
        port:
          number: 8089

The copied parts are:

metadata.name
spec.rules[0].host -> spec.hosts[0]
spec.rules[0].http.paths[0].path -> spec.http[0].match[0].uri.exact
spec.rules[0].http.paths[0].backend.serviceName -> spec.http[0].route[0].destination.host
spec.rules[0].http.paths[0].backend.servicePort -> spec.http[0].route[0].destination.port.number

spec.gateways[0] needs to be configurable as I mentioned above.

After doing this, cert-manager gets the request routed correctly and is able to issue the certificate.

Let me know if I should post this somewhere else, or if there is anything else you need to get cert-manager to work with Istio

[sneko]

Hi @rfevang, I ended to the same solution but that's still a manual process to replay every 2 months to have good variables values routing Let's Encrypt.

[rfevang]

Yes, it's not the best long term process, which is why I tried writing up the details like @munnerz was requesting. Some way of having cert-manager generate the VirtualService would be all that's required, either through knowledge of Istio, or a way of specifying a template for it to fill in.

[spekr]

I really do not see how this could be working. For the traffic to be sent to the acme-challenge-pods, the pods need to have an istio-sidecar. Currently the acme-challenge-pods have the annotation istio-sidecar-injection set to false hardcoded... For me it is a mystery how you guys have got this to work.
@rfevang if you got it to work with a custom made virtualservice, it should also work without it. The only prerequisite is that you have the k8s-autogenerated-gateway. (Read the article I referred to in a previous post, if you'd like to know more.)
If anyone can tell me, how the virtualservice can be working without the sidecar (istio-proxy), I'd be really interested to know.

[rfevang]

The article you linked to uses an undocumented and not supported way of accessing the cluster using Ingress resources. I wouldn't bet on that staying around for too long, Istio switched away from using Ingress for good reasons. The code I posted above routes the request using Istio's new traffic management system. AFAIK the destination service doesn't need to have a side car (maybe it does if mTLS is enabled?). The gateway is running on a pod with a sidecar (istio-ingressgateway above), which has traffic entering it from one of the cluster's external IPs. The VirtualService above then tells the gateway how to route the request to the acme challenge pod.

If this doesn't work with mTLS or other special Istio setups, you might have to add a ServiceEntry for the challenge service. As that wasn't required in my fairly standard setup I didn't look into it (I also don't know if this is required at all).

I don't know what you mean by "it should be able to work without a VirtualService". Without a VirtualService the gateway pod has no way of knowing where to route the request, and the gateway pod is what the certificate URL resolves to.

[Kansuler]

@spekr @sneko
I got this exact problem today and I found a temporary solution if you got a cluster with Istio mTLS enabled. I created a namespace where i set mTLS mode to PERMISSIVE. In that namespace i put the certificate, because then the certmanager will start the pod in that namespace and traffic will work. I will use this namespace only for this purpose.

I hope this helps until the sidecar.istio.io/inject: "false" becomes configurable.

apiVersion: v1
kind: Namespace
metadata:
  name: certificate
  labels:
    istio-injection: enabled
---
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: default
  namespace: certificate
spec:
  peers:
    - mtls:
        mode: PERMISSIVE
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: disable-mtls
  namespace: certificate
spec:
  host: "*.certificate.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ingress-cert
  namespace: certificate
spec:
  secretName: ingress-cert
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: "---"
  dnsNames:
    - "---"
  acme:
    config:
      - http01:
          ingressClass: istio
        domains:
          - "---"

[lentzi90]

@Kansuler do I understand correctly that this workaround requires a separate gateway (and deployment) in the same namespace that the certificate is in? Please correct me if I'm wrong but I think that

• the certificate must be in the same namespace as the secret it references
• the secret must be in the same name as the gateway that will use it
• the gateway must be in the same namespace as the deployment it selects

I'm not sure how to make this work. Do you expose the two gateway deployments through the same service?

Edit: I guess that if the certificate namespace is really istio-system it would work.

[spekr]

@lentzi90 this works because you put that namespace in permissive mode. Which basically means you'll be disabling mtls for that specific namespace. The reason it doesn't normally work is due to the mtls. That's why this workaround will probably succeed.
We have considered this option for our own cluster, but didn't feel quite comfortable to disable the mtls... even in one separate namespace. We've 'fixed' the issue by manually uploading our ssl certificates, but this takes time and is more error prone than leaving it to the cert-manager. It's a trade-off which you'll have to decide for yourself if it's worth it.

[lentzi90]

@spekr I think I understand the part about mTLS, what I don't understand is if this is possible to do in a separate namespace (e.g. certificate) instead of in istio-system.

If I use a separate namespace like @Kansuler suggested, then the ingress-gateway would also have to be in that namespace or it wouldn't be able to use the secret, right? So either I have to put all certificates in istio-system or deploy one ingress-gateway per namespace. (Or do some manual copying of secrets between namespaces.)

Edit:

• The Gateway is required to be in the same namespace as the pods it selects according to this
• The credentialName is a bit more vague as it doesn't mention any namespace. I think this means it has to be in the same namespace.

[spekr]

@lentzi90 you are quite right: the gateway should indeed be in the same namespace as the secrets. Likewise for the loadbalancer deployment.
However, you wouldn't need a gateway for each namespace. In our cluster we have three namespaces and the virtualservices for each namespace point at the one gateway we have in the istio-system namespace.

Here is an example of a virtualservice we use (I've left out some lines which are irrelevant for this example.

Name:         some-app
Namespace:    development
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Spec:
  Gateways:
    istio-autogenerated-k8s-ingress.istio-system.svc.cluster.local
  Hosts:
    development.ndcmediagroep.nl
  Http:
    Match:
      Uri:
        Prefix:  /
    Route:
      Destination:
        Host:  some-app.development.svc.cluster.local
        Port:
          Number:  8080

Hope this helps.

Edit:

This link you are referring to, has to do with the gateway. The pod it is selecting is actually the loadbalancer and this should indeed be in the same namespace. You're own pods however can be in different namespaces, as described above.

[lentzi90]

I see. Thanks for the clarification @spekr. It still bothers me a bit that all the certificates has to be in the same namespace but I guess I have to live with that. The way I would want it to work is that the certificates (and secrets) would be in the same namespace as the application and virtualservice. I thought this was what @Kansuler had achieved but maybe I misinterpreted.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[retest-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle rotten
/remove-lifecycle stale

[tony84727]

@Kansuler @spekr

I hope this helps until the sidecar.istio.io/inject: "false" becomes configurable.

I fear it has to do with the istio-sidecar being absent in the acme-challenge-solver pods. (We have mtls enabled.)

I run into this today. The solver pod doesn't spawn with the sidecar.
And I found this PR #1749.

It seems that we can override the annotation by podTemplate. (Doc: https://cert-manager.io/docs/reference/api-docs/#acme.cert-manager.io/v1alpha2.ACMEChallengeSolverHTTP01Ingress)

Here's my ClusterIssuer manifest:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: <your-email>
    privateKeySecretRef:
      name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: istio
          podTemplate:
            metadata:
              annotations:
                sidecar.istio.io/inject: "true"

[retest-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

[jetstack-bot]

@retest-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to jetstack.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[XanderAtBackboneSystems]

Nooooooooooooo... :(

[haf]

/remove-lifecycle rotten

[haf]

/reopen

[jetstack-bot]

@haf: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[haf]

Related:

• https://github.com/jetstack/cert-manager/pull/1749
• https://github.com/istio/istio.io/issues/868#issuecomment-462761222
• https://github.com/istio/istio/issues/20680#issuecomment-588124135
• https://github.com/jetstack/cert-manager/issues/1535
• https://github.com/istio/istio/issues/6486
• https://github.com/istio/istio/issues/19296#issuecomment-578841838
• https://istio.io/docs/tasks/traffic-management/ingress/ingress-certmgr/

[sneko]

/reopen

[jetstack-bot]

@sneko: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sneko]

1 year later... every 3 months I'm still required to create VirtualServices so the solvers requests go to the right services.

That's really annoying, really.

It doesn't seem since then someone has been complaining about VirtualServices... how did you manage Istio routing Let's Encrypt requests onto the solvers?

Thank you,

EDIT: that's funny because I never succeeded to make k8sIngress from Istio working, so I had to every 3 months customize some VirtualServices to make calls going to solvers and then remove those VirtualServices.

After loosing my hair a new time, I got it to work... I removed my own gateway that was catching those domains, and just recreated it. I think that when I enabled k8sIngress a few months ago, the istio-autogenerated-k8s-ingress generated gateway may not get the higher priority compared to my own gateway. I cannot guarantee that's the explication, but a lot of time I saw for routing earliest rules taking advantages over others (so maybe that's the same for gateways ^^).

_I leave this issue open since other people requested for_

[alexivkin]

Hit the same issue and I can confirm, it comes to which gateway takes priority - the autogenerated one, which is hardcoded in istio to handle native K8s Ingress resources, or any other you might have in your cluster. These are unable to forward requests to the native Ingress resources created by cert-manager's HTTP01 issuers, and so the cert issuer never succeeds.

What seem to have solved it for me is defining host FQDNs on the custom Istio gateways, instead of the *, which avoids conflics with istio-autogenerated-k8s-ingress

The proper solution though is to add support for VirtualServices to cert-manager. Please?

[BradErz]

Currently running into the exact same issue to have cert-manager working with istio ingress gateways.

I guess the easiest way right now would be to use dns01 instead of http01 challange right untill VirtualServices are understood by cert-manager.

[sneko]

@BradErz from what I remember I chose HTTP01 over DNS01 because the latter requires API from Google Cloud DNS (I'm with Google Domains) and it would cost me $$$ just for CertManager being able to renew domains certificates :/

[alexivkin]

The bigger issue for us with DNS01 is the fact that you have to keep DNS management API keys in a K8s secret all the time. Imagine what one can do if they get a hold of them.

[cpick]

We also use DNS01 as a workaround, but this has two drawbacks:

1. (As @alexivkin said) we need to give authorization for cert-manager to be able to edit DNS records (sometimes in other GCP projects).
2. Using DNS01 means that we cannot use wildcard A records (wildcard DNS records only work if there are no records of any type with the more-specific name. When cert-manager creates the TXT record for the DNS01 challenge, that's on the specific FQDN which breaks/cancels the wildcard A records for that sub domain.

This is just to say that DNS01 is a usable workaround in our case, but with decent drawbacks that we'd like to eliminate.

[munnerz]

In order to tidy up our issue tracker as we have quite a few open issues around Istio, I've retitled this to make it a feature gate for us to better support HTTP01 via VirtualService resources.

https://github.com/jetstack/cert-manager/issues/2526 is an issue to track us documenting how to get Istio working today using cert-manager with the current Ingress based solution - if anyone has this working already and can contribute there, that'd be great 😄

/kind feature
/remove-kind bug
/area acme/http01
/area api

[khernyo]

We are using cert-manager with Istio and have a workaround for routing the HTTP01 challenge via a VirtualService, but we are willing to develop a proper solution in cert-manager.

Would you be interested in such a PR? If so, do you have any specifics in mind about an ideal implementation or should I just go ahead and submit a PR?

[kriston13]

Hi.
I'd dearly love an answer to this too.
We're using a bunch of custom gateways for different namespaces & subdomains, but can't get http01 challenges solved.

[wutz]

@kriston13

We use the internal priority rules of istio-ingressgateway in our cluster.

1. Remove port 80 definition of specific domain name in all custom Gateway
2. Add port 80 definition matching all domain names

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: general-gateway
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        protocol: HTTP
        name: http
      hosts:
        - "*"
      tls:
        httpsRedirect: true
   ```
3. Add a ClusterIssuer or Issuer definition
```yaml
apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
  name: letsencrypt-istio
  namespace: istio-system
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [email protected]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-istio
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: istio       

Simply understand how this works

1. When creating a Certificate for the first time, cert-manager will create an Ingress of a specific domain name, and the priority of the specific domain name is higher than matching all
2. Once the cert-Manager completes the certificate request,Ingress will be removed so that all matching rules are in effect
3. When cert-manager wants to renew the certificate, it will create an Ingress of a specific domain name, and then how to do the above steps

But this scheme has a flaw that you cannot define a specific domain name on port 80 yourself, so the Ingress created by cert-manager will not take effect.

[ph1387]

@wutz
I had the same issues as described above. Your solution works like a charm!

[RenaudDemarneffe]

Hi @wutz , could you please explain how to "Remove port 80 definition of specific domain name in all custom Gateway".
I presume it's into the IstioOperator configuration but I don't find the configuration parameter.

Thanks for your help.
Renaud

[wutz]

@RenaudDemarneffe It's not in IstioOperator to configure, just remove port 80 definition from every Gateway

[RenaudDemarneffe]

@wutz , ok, I understand. Thank you for your help