Istio: Istio + Cert-Manager + Let’s Encrypt
Istio (Envoy) + Cert-Manager + Let’s Encrypt for TLS guide does not work with Istio 0.8.0 any more ...
The problem is that Istio 0.8.0 no longer support combining Kubernetes Ingress specs with Istio routing rules. So we can't create an ingress for /.well-known/acme-challenge/.* path and map it in the gateway.
I've tried to create mTLS setup for cert-manager and add Gateway/VirtualService. The problem here is that cert-manager create a cm-acme-http-solver-xxx pod in the istio-system namespace, which does not have the sidecard inject label...
Is there a solution for Istio 0.8.0 + CertManager 0.3.0? Some alternative solution would be also great...
yacut
All 73 comments
@yacut did not go through the detail of this blog, one question, so with cert-manager, we will not need Citadel for cert management in istio mTLS?
Can you try edit the pod by adding follow annotation and restart pod?
annotations:
sidecar.istio.io/inject: "true"
gyliu513
on 22 Jun 2018
@gyliu513 unfortunately this will not help. Cert Manager creates for each certificate issue the cm-acme-http-solver-xxx temporary resources (pod, service and ingress) without sidecar.istio.io/inject: "true" annotation and the istio-system namespace has no istio-injection label:
$ kubectl get pod --all-namespaces
cert-system cert-manager-864896c6cd-4bszs 2/2 Running 0 21m
default details-v1-7b97668445-9d5dw 2/2 Running 0 23h
default productpage-v1-7bbdd59459-8k8tj 2/2 Running 0 23h
default ratings-v1-76dc7f6b9-n7p57 2/2 Running 0 23h
default reviews-v1-64545d97b4-756gv 2/2 Running 0 23h
default reviews-v2-8cb9489c6-ztx54 2/2 Running 0 23h
default reviews-v3-6bc884b456-225p8 2/2 Running 0 23h
istio-system cm-acme-http-solver-fvmj6 1/1 Running 0 17m
istio-system grafana-6bcb55748b-m8d8v 1/1 Running 0 1d
istio-system istio-citadel-85d65bfb4-n2v65 1/1 Running 0 13h
istio-system istio-cleanup-old-ca-wffm2 0/1 Completed 0 1d
istio-system istio-egressgateway-68ff555b46-2hhdl 1/1 Running 1 1d
istio-system istio-ingressgateway-c99f56956-m86f8 1/1 Running 1 1d
istio-system istio-mixer-post-install-l8vdc 0/1 Completed 0 1d
istio-system istio-pilot-566fd44cb9-rw5x7 2/2 Running 0 13h
istio-system istio-policy-6694bbcf7f-6wvsw 2/2 Running 0 13h
istio-system istio-sidecar-injector-645fddc6db-c6zds 1/1 Running 0 12h
istio-system istio-statsd-prom-bridge-66d59f964d-gxz4d 1/1 Running 0 1d
istio-system istio-telemetry-6b846c68d9-lvdrb 2/2 Running 0 13h
istio-system istio-tracing-cbcd767c7-7jvn7 1/1 Running 0 1d
istio-system prometheus-65844fff75-7zqmc 1/1 Running 0 1d
istio-system servicegraph-6598c77486-sfkjw 1/1 Running 0 1d
$ kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-system cm-acme-http-solver ClusterIP 10.19.240.63 <none> 8089/TCP 23m
default details ClusterIP 10.19.245.218 <none> 9080/TCP 23h
default kubernetes ClusterIP 10.19.240.1 <none> 443/TCP 1d
default productpage ClusterIP 10.19.253.12 <none> 9080/TCP 23h
default ratings ClusterIP 10.19.249.113 <none> 9080/TCP 23h
default reviews ClusterIP 10.19.249.64 <none> 9080/TCP 23h
istio-system cm-acme-http-solver-w9gkj NodePort 10.19.247.45 <none> 8089:31715/TCP 19m
istio-system grafana ClusterIP 10.19.248.95 <none> 3000/TCP 1d
istio-system istio-citadel ClusterIP 10.19.253.144 <none> 8060/TCP,9093/TCP 1d
istio-system istio-egressgateway ClusterIP 10.19.250.146 <none> 80/TCP,443/TCP 1d
istio-system istio-ingressgateway LoadBalancer 10.19.249.46 xx.xxx.xxx.xxx 80:31380/TCP,443:31390/TCP,31400:31400/TCP 1d
istio-system istio-pilot ClusterIP 10.19.247.46 <none> 15003/TCP,15005/TCP,15007/TCP,15010/TCP,15011/TCP,8080/TCP,9093/TCP 1d
istio-system istio-policy ClusterIP 10.19.246.58 <none> 9091/TCP,15004/TCP,9093/TCP 1d
istio-system istio-sidecar-injector ClusterIP 10.19.252.199 <none> 443/TCP 1d
istio-system istio-statsd-prom-bridge ClusterIP 10.19.246.124 <none> 9102/TCP,9125/UDP 1d
istio-system istio-telemetry ClusterIP 10.19.242.213 <none> 9091/TCP,15004/TCP,9093/TCP,42422/TCP 1d
istio-system prometheus ClusterIP 10.19.246.249 <none> 9090/TCP 1d
istio-system servicegraph ClusterIP 10.19.243.109 <none> 8088/TCP 1d
istio-system tracing LoadBalancer 10.19.241.11 xx.xxx.xxx.xxx 80:32742/TCP 1d
istio-system zipkin ClusterIP 10.19.243.28 <none> 9411/TCP 1d
$ kubectl get ingress --all-namespaces
NAMESPACE NAME HOSTS ADDRESS PORTS AGE
istio-system cm-acme-http-solver-45jdg bookinfo.example.com 80 22m
$ kubectl --namespace istio-system describe ingress cm-acme-http-solver-45jdg
Name: cm-acme-http-solver-45jdg
Namespace: istio-system
Address:
Default backend: default-http-backend:80 (10.16.2.13:8080)
Rules:
Host Path Backends
---- ---- --------
bookinfo.example.com
/.well-known/acme-challenge/xxx cm-acme-http-solver-w9gkj:8089 (<none>)
Annotations:
kubernetes.io/ingress.class: none
Events: <none>
I think a solution could be a custom label for the cm-acme-http-solver pod, but if the https://github.com/jetstack/cert-manager/pull/622 PR will be merged, then mTLS setup for the cert manager is not an option any more.
The question here is: How can I include an ingress to the istio gateway?
Please see https://github.com/istio/istio/pull/6509 example. I hope this helps to understand the issue.
yacut
on 22 Jun 2018
@yacut Does this help by adding the injection label to istio-system and re-depoy?
kubectl label namespace istio-system istio-injection=enabled
Also can you help my question: so with cert-manager, we will not need Citadel for cert management in istio mTLS?
gyliu513
on 22 Jun 2018
@yacut We've been using a DNS challenge provider with cert-manager on 0.8.
markns
on 22 Jun 2018
@gyliu513 This will not work, because we can't select a cm-acme-http-solver pod with dynamic labels to expose it.
I do not think that the CertManager can replace the Citadel. The main goal of the CertManager is to generate a trusted certificate (with LetsEncrypt) for the ingress resource. If the Citadel could generate browser trusted certificates, then there would be no reason to use the CertManager.
@markns Thanks for advice, but we do not use any of these dns providers. The
HTTP01 Challenge is only option for us now.
yacut
on 22 Jun 2018
Hi there.
I'm the one who wrote the blog post cited here.
As of Istio 0.8.0 I recommend not using the HTTP challenge and go with the DNS01 one. I'm about to write a blog post about it.
There's still some issues here and I don't see any simple way to resolve them :
you can define only one Gateway per port
This means that if you have 1.site.com and 2.site.com both listening on the port 443, you have to create one Gateway with two hosts, something like :
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
name: http-gateway
namespace: test
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- 1.site.com
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/1.key
serverCertificate: /etc/istio/ingressgateway-certs/1.crt
- hosts:
- 2.site.com
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/2.key
serverCertificate: /etc/istio/ingressgateway-certs/2.crt
each hosts of a Gateway needs it's own certificate file inside THE SAME secret
As you can see above, each hosts have it's own certificate inside the same secret. This secret is named istio-ingressgateway-certs and is mounted to /etc/istio/ingressgateway-certs.
So you need to update this single secret everytime something change, like adding a new domain name.
The other solution here is to use the default tls secret, which will create only 2 files inside the sercret, tls.crt and tls.key.
Doing it this way require that :
- you create a multi-domain SSL certificate
- you define your
Gatewayso that everyhostsuse the same certificate files, something like :
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
name: https-gateway
namespace: test
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- 1.site.com
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
- hosts:
- 2.site.com
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
In fact, doing it this way is the default behaviour of the latest Cert-Manager. Put all the FQDN (domain names) inside a single Certificate definition.
Ex :
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
name: https-ingress-certs
namespace: istio-system
spec:
acme:
config:
- dns01:
provider: aws-dns-prod
domains:
- 1.site.com
- 2.site.com
commonName: 1.site.com
dnsNames:
- 1.site.com
- 2.site.com
issuerRef:
kind: ClusterIssuer
name: letsencrypt-prod
secretName: istio-ingressgateway-certs <-------------------- important
Then you will reach the next problem :
the Ingress Gateway does not reload the SSL from the secret when it is updated
If you've done all the above, Cert-Manager will create/refresh you certificate inside a secret used by the Ingress Gateway... a 10 days later the SSL certificate is refreshed... and it's never updated !
You'll have to kill the pod of the Ingress Gateway (and break your service) so it can start with the latest version of the certificate.
Same issue if you add a new domain name inside the certificate.
So, While I think I have a (ugly) solution to use Cert-Manager with Istio 0.8.0 and the new API, What is the plan so the IngressGateway can monitor the certificate and reload on change ?
Maybe (not tester) we could send a signal to the Ingress Gateway Envoy so it will hot reload the config and certificates (as the mounted Secret is updated as soon as it is changed from the API) ?
Maybe it's the Pilot's job to achieve that ?
Any thought from the Istio Devs ?
As a side note, I made a modification to Cert-Manager so it can put each certificate inside different files in the same secret. It's a dirty hack, but it's working. But as the Ingress Gateway does not reload on secret change, we still reach the same issue.
prune998
on 26 Jun 2018
Could the hot reload issue be solved with vault integration? The cert manager supports vault, so the istio ingress gateway could also make it.
yacut
on 27 Jun 2018
Subscribing to this
Please also see https://github.com/jetstack/cert-manager/issues/733 for our longer term plans - some of the requirements around formatting of the TLS key secret seem to be an issue with Istio and not cert-manager, however I'm keen to hear the Istio teams views and input so we can adapt our own proposal to fit Istio well 😄
I recently merged https://github.com/jetstack/cert-manager/pull/622 which may also help with solving HTTP01 challenges. I've not had a chance to give it a go yet, but will update here when I do 😄
munnerz
on 17 Jul 2018
@munnerz It will only work with Istio <=0.7. I've tested certmanager 0.4.0 and istio 1.0.0/0.8.0 tomorrow, the problem is not solved, because with the _Istio 0.8 release_ we have the Ingress Gateway instead of k8s Ingress and the challenge resolver pod is not in Istio Mesh.
With a custom label (or a label with certificate host) for the challenge resolver pod it would be possible to create a k8s svc for challenge resolvers and a Service Entry and then include it to Istio Mesh. I recently created an issue for it: https://github.com/jetstack/cert-manager/issues/672
I think a 'heavy' controller would also solve this problem.
yacut
on 17 Jul 2018
To split the issues: I've got http01 challenge working with 1.0 (pending some fixes to Ingress code and config in istio). Not ideal - long term we should still try to switch to Gateway, but at least we can focus on the real problems.
Someone mentioned a way to place multiple certificates inside the same secret, does anyone have details ? Secrets are refreshed (with some delay), the restart is only needed for mounting of new secrets.
Long term: the plan is to use the 'secret discovery service' instead of mounted secrets, and the SDS component will take care of integration with Vault or watching k8s secrets. Backup plan is to watch the secrets from a small helper.
I think the main problem remains the behavior of envoy if a Gateway defines a secret, but the secret is not found. Right now the entire listener is invalidated.
costinm
on 17 Jul 2018
@prune998 in your configuration, please make sure each 'port' has a different name. Spent some time debugging this, in 1.0 the port name is used to name the RDS route. We'll need to remove this limitation, but that's the current behavior.
costinm
on 17 Jul 2018
@costinm, you're right, there are many issued here :
http01 challenge
This one is tricky as CertManager will create the needed pods/service/(...) without knowing if Istio is in use. There is an issued opened for that, but the http01 challenge seems not the right way in this situation = dns01 challenge is a better fit
multiple certificates inside the same secret
this is done natively by CertManager 0.4.0 by using the doc above. see https://medium.com/@prune998/istio-0-8-0-envoy-cert-manager-lets-encrypt-for-tls-d26bee634541
reload of Envoy Ingress Gateway when the SSL Secret is updated
it's a work in progress and will have to wait for the Envoy SDS to be implemented (it seems).
One good solution to me would be that Pilot monitor the Secrets API and trigger a reload/hot-restart when it is updated
I'll have a look at the ports names as you mentionned and will update my blog post to reflect that. Thanks
prune998
on 17 Jul 2018
http01 challenge is now working with istio ( or will work when I submit #7144 ).
dns01 - I don't think it's viable for most users, putting DNS credentials in a cluster is too risky.
costinm
on 17 Jul 2018
@prune998 - I'm looking for more details on:
"As a side note, I made a modification to Cert-Manager so it can put each certificate inside different files in the same secret. It's a dirty hack, but it's working. But as the Ingress Gateway does not reload on secret change, we still reach the same issue."
since it can avoid restarting ingress gateway.
costinm
on 18 Jul 2018
You will still have to restart the ingress gateway...
If you add a new file in the secret, the Ingress Gateway will be able to use it when you create the Gateway/VirtualService pair.
If you update the certificate (renew) you will still need to restart the Ingress Gateway as a whole, closing any connections to the clients.
As I said, it's not a solution for this issue.
It would still be good way to have the Ingress Gateway setup so that each Gateway use it's own certificate instead of a stack of all certificates from a single file/secret.
Am I clear in my explanations ? :)
prune998
on 18 Jul 2018
The files in the Secret are automatically updated - with some delay. And renewal is certainly not a concern, it's every few months.
We already support each Gateay having its own certificate - using SNI, that's not a problem.
What I still don't know - and is not at all clear - is how to get certmanager to put a new cert in an existing Secret.
costinm
on 18 Jul 2018
If successful, the resulting key and certificate will be stored in a secret named acme-crt-secret with keys of tls.key and tls.crt respectively
What we need is to store them in acme-crt-secret - but with custom names (foo.key and foo.crt)
costinm
on 18 Jul 2018
@costinm this is exactly what my patch was doing.
note that, by default, the Istio Ingress Gateway is only mounting the secret istio-ingressgateway-certs in /etc/istio/ingressgateway-certs.
By default this secret is a TLS secret, which implies a tls.crt and tls.key, which is not what you want if you need to use one file per certificate.
When Cert-Manager is told to put all the certificates in the same Secret, all the certs are appended into the same tls.crt file.
Check my code at https://github.com/prune998/cert-manager to see how I changed cert-manager to put each cert in a new file in one signe secret.
prune998
on 19 Jul 2018
So basically your fork supports 'secretKeyName' and 'secretCrtName'. And I assume you tested with multiple domains, and multiple pending requests - and it's doing edit properly. That would work - but not sure if using your fork is the right option here, and it doesn't seem to be merged upstream.
I guess one option would be to fork certmanager under istio (we did it in the past with other projects where we needed to cherry-pick changes), and have the fork under istio hub - but it's far too late for 1.0
costinm
on 19 Jul 2018
Well @costinm I did not made a PR as I found this solution was not helping much, and I wanted to take time and check with cert-manager's devs what would be the best solution.
My code was just a proof of concept, and yes, it's working as far as I have tested it. But it's far from a proposal...
Either Cert-Manager have to be changed to accomodate Istio, or the other way around... I'm not the right person to take this decision though :)
prune998
on 19 Jul 2018
Well, it would help to create a PR for that - it will certainly help avoid ingress restarts and seems a generic feature. From Istio side we will likely add some Secret monitoring - but that'll take some time.
We need to consider dropping certmanager and just adding minimal code (maybe to citadel) - lego is a simple library and a http challenge is relatively easy to implement. But of course it would be better if certmanager can solve the problem and we don't have to spend the time on that.
costinm
on 20 Jul 2018
I would rather have Lego (or anything else) added to Istio... Traefik is doing this with great success...
Citadel is new to me. I'll check the code and see if I can be of any help on that.
prune998
on 23 Jul 2018
Citadel is the cert manager used by istio (for service to service TLS cert provisioning). I was thinking to map .well-known/acme to citadel, and just use Lego library with http validation - as simple and clean as possible.
The reason to use citadel is that they're in process of moving from using Secret objects to envoy SDS (secret discovery service), and using a node agent - it will avoid a lot of problems.
costinm
on 25 Jul 2018
FWIW, we have plans to add a pluggable interface for solving HTTP01 challenges, and one of the first targets for this interface will be Istio (i.e. Gateway) and Contour (IngressRoute).
I'd imagine it'll be 2-3mths until this support is ready and merged into a release tag of cert-manager, but this will then allow users to solve HTTP01 challenges with cert-manager too.
wrt. some of the nuances around the key names with Istio - I'd hope that any changes to Istio/Pilot that switch it to use SDS, would also include some mechanism for allowing those secrets to be stored in a more 'traditional' manner (i.e. a secret per Gateway, with fixed/static keys in each Secret).
Please reach out if you've got any questions on cert-manager. I recently had a call with @davecheney to plan out a strategy for support IngressRoute natively, and would be happy to have a similar conversation with the Istio core developers on this to ensure our plans align well 😄
munnerz
on 28 Jul 2018
I would rather have Lego (or anything else) added to Istio... Traefik is doing this with great success...
This might make things easier if you find cert-manager difficult to integrate, however we do intend to have full support for Istio (for both HTTP01 and DNS01), and I will also add that maintaining integrations with many DNS providers as well as dealing with the nuances of managing the lifecycle of the resources required for an HTTP01 challenge, as well as handling Let's Encrypt rate limits is a painful task (one that takes up a lot of my own time!)
munnerz
on 28 Jul 2018
If we can get deterministic label/service for the HTTP01 resolver, and the patch to store multiple certs in a secret - I think can get cert-manager to work well enough with Istio.
costinm
on 29 Jul 2018
Deterministic label is easy enough to add.
and the patch to store multiple certs in a secret
This, not so much. This will involve changing our API surface as well as
cause issues with owner references etc.
I wonder what the justification for a design based on a single secret is in
Istio, and would question whether long term this is a viable solution.
If someone is serving 1000 domains through Istio, the only way to carve up
who can view/modify those certificates is to run many ingress gateways.
Long term it seems like Istio will change this requirement - and so I am
reluctant to modify our own API to facilitate it in the interim.
Have you got any details on what secret management/discovery will look like
in Istio in future?
On Sun, 29 Jul 2018 at 16:55, Costin Manolache notifications@github.com
wrote:
If we can get deterministic label/service for the HTTP01 resolver, and the
patch to store multiple certs in a secret - I think can get cert-manager to
work well enough with Istio.—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/6486#issuecomment-408687013, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMbP-iyQtvLlJ0VdbgQX2UDCFNJgL3-ks5uLdsJgaJpZM4UyopZ
.
munnerz
on 29 Jul 2018
@munnerz where could we find details on what secret management/discovery will look like
in Istio in future ?
Who may have any insight ?
prune998
on 30 Jul 2018
Could you guys please confirm whether this point is true or false:
the Ingress Gateway does not reload the SSL from the secret when it is updated
Forget the Cert-Manager, what if I manually add a new certificate inside istio-ingressgateway-certs, will I need to restart Ingress Gateway and cause downtime? What if I update an existing certificate inside that k8s secret, again will I need to restart the Gateway or not?
maximbaz
on 2 Aug 2018
@maximbaz, if i'm not mistaken :
- when you update a secret, the change is reflected inside the pod... at come point. you have no clue when the update happen
- if you create a new gateway after the secret is updated in the pod, the new certificate will be used
- if you change the secret of an existing gateway, it is not picked-up, you have to restart the IngressGateway (or delete/re-create it)
prune998
on 2 Aug 2018
Thanks for the answer 🙂 Could you clarify the difference between the first and the last items, do you mean that when I update the existing certificate inside the k8s secret, it will be discovered automatically at some point, but when I add a new certificate into the existing k8s secret, it will never be picked up?
maximbaz
on 2 Aug 2018
Is there any roadmap document for istio on any plans to address multiple rotating secrets for SNI?
farcaller
on 4 Aug 2018
@prune998 Thanks a lot for writing that blog post, that's very helpful!
However, you can indeed define multiple ingress Gateways for the same port number, if the port name (label) differs.
E.g.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: app-a
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https-app-a
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
hosts:
- "app-a.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: app-b
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https-app-b
protocol: HTTPS
tls:
mode: SIMPLE
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
hosts:
- "app-b.example.com"
Using a single cert here, similar to your example (could also be wildcard).
Istio seems to generate config rules with the port definition taken as names. @costinm I think that's what you're referring to, right?
See `istioctl proxy-config route -n istio-system istio-ingressgateway-xxxxxxxxx-xxxxx -o json
[
{
"name": "https.443.https-app-a",
"virtualHosts": [
...
If the port is always named after the protocol only, that collides.
Hope that helps :)
sdaschner
on 22 Aug 2018
You're right @sdaschner. Few things changed from 0.8.0 to 1.0.0 and my blog post needs to be updated !
prune998
on 22 Aug 2018
ldemailly
on 4 Sep 2018
Hi @ldemailly,
I already saw the PR you mentioned but I can't make HTTP01 validation working with Istio.
Is it still possible? And if yes, do you have some documentation please?
Before I was using the old way you described by creating Service/Ingresses manually but with the new cert-manager version I don't know anymore 😢
Thanks,
sneko
on 4 Sep 2018
did you try the steps above ? you may want to ask for help at the next environments or networking community meeting. There is also https://github.com/istio/istio.github.io/issues/868 ongoing.
ldemailly
on 4 Sep 2018
@ldemailly I had followed steps in the article since I'm still running Istio 0.7 with Ingresses. The major problem is that I was starting from files I made a few months ago when Cert-manager was not generating by itself Kubernetes entities compatible with Istio...
After upgrading the Cert-manager I forgot to move my certificate from the "istio-system" to another one where "acme solver" pods could be watched by the Istio injection and get Envoy sidecars to make Ingress traffic up to them.
Sorry for the inconvenience 🔨
Thank you,
sneko
on 8 Sep 2018
This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 7 Dec 2018
unstale (still a problem)
farcaller
on 7 Dec 2018
Im not going to be using istio until they sort out some basics, like using letsencrypt. I might be the only 1 who feels like that, but ya know, just sayin.
nigel-dewar
on 1 Jan 2019
This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.
stale[bot]
on 1 Apr 2019
Activity!
Bessonov
on 1 Apr 2019
Folks who caring about this feature, can you comment on the the new feature as part of 1.1 release https://istio.io/docs/examples/advanced-gateways/ingress-certmgr/?
I just quickly scanning all the threads, seems like the gateway restart and multi secret problem are now satisfied. We'd like to know some feedback whether gluing with secrets as of now is good enough, or you want more customization and flexibility by implementing SDS yourself?
I believe we can closing this issue now and file separate bug if more customization is needed. Let me know if I'm wrong.
/cc @JimmyCYJ @lei-tang @wattli @myidpt
incfly
on 21 May 2019
@incfly I'm not sure SDS allow full Cert-Manager integration, but I'm upgrading a cluster to latest Istio and testing the SDS. I'll let you know.
Thanks
prune998
on 21 May 2019
@incfly I just tested and I can confirm it's working, following the documentation at https://istio.io/docs/examples/advanced-gateways/ingress-certmgr/
The only thing I can't explain, is the usage of an Ingress resource, while I thought we were supposed to only use gateways and virtualServices... any info on that ?
FYI, in final, the only thing to change is your gateway, where you replace the paths to your CRT/Key by sds :
tls:
credentialName: cert-authd.my-domain.com <--- this is the secret name where Cert-Manager put the SSL CRT/Key pair
mode: SIMPLE
privateKey: sds
serverCertificate: sds
Is it possible to use different certs for each ingress resource (for example when managing multiple domains in the cluster)? I haven't found any note in the documentation so far.
muhlba91
on 22 May 2019
@muhlba91 you can use one SSL certificate by domain name listed in your gateway. like :
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: gw-hello-world-com
namespace: alerting
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- helloworld.com
port:
name: https-443-helloworld.com
number: 443
protocol: HTTPS
tls:
credentialName: cert-helloworld.com
mode: SIMPLE
privateKey: sds
serverCertificate: sds
- helloworld.ORG
port:
name: https-443-helloworld.org
number: 443
protocol: HTTPS
tls:
credentialName: cert-helloworld.org
mode: SIMPLE
privateKey: sds
serverCertificate: sds
The Ingress resource you are referring to is another way to define a VirtualService, which is how you bind your Gateway to your Service, and does not involve SSL.
prune998
on 22 May 2019
I'm trying to use SDS for secret delivery. Configuration with file mount works, but configuration with SDS - does not. My certificate/secret is created using Cert-Manager. I see that there is a different secret type used in documentation.
For file based approach documentation says to use secret type "tls" https://istio.io/docs/tasks/traffic-management/secure-ingress/mount/ :
kubectl create -n istio-system secret tls istio-ingressgateway-certs --key httpbin.example.com/3_application/private/httpbin.example.com.key.pem --cert httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
For SDS approach documentation sais to use secret type "generic" https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/ :
kubectl create -n istio-system secret generic httpbin-credential \
--from-file=key=httpbin.example.com/3_application/private/httpbin.example.com.key.pem \
--from-file=cert=httpbin.example.com/3_application/certs/httpbin.example.com.cert.pem
Cert-Manager creates secret with type "tls", so I think this might be a reason why it does not work for me.
When configured with SDS key delvery - connection to gateway times out, I cannot see any errors in ingress gateway proxy logs.
jurna
on 24 May 2019
That doesn't sound right; I was definitely able to load cert-manager certs into the istio ingress via sds. What's your configuration block like?
farcaller
on 24 May 2019
Here is configuration which gives timeout from gateway:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
annotations:
name: prfx-gateway
namespace: domain-demo
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- demo.domain.com
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- hosts:
- demo.domain.com
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: istio-ingressgateway-certs
mode: SIMPLE
privateKey: sds
serverCertificate: sds
Here is configuration which works:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
annotations:
name: prfx-gateway
namespace: domain-demo
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- demo.domain.com
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- hosts:
- demo.domain.com
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
@jurna when using SDS you don't need (and maybe must not) use istio-ingressgateway-certs.
Put your SSL in it's own secret by changing it in your Certificate definition (like cert-demo-domain-com).
Which version of Istio are you using ? Did you enable the SDS functionality in the gateway ?
All my SDS secrets are kubernetes.io/tls (as generated by Cert-Manager) and it's working fine so far...
prune998
on 24 May 2019
Ok, now I've issued another certificate with certificate name cert-domain.com and secret name secret-domain.com ant it's working fine. I don't know was the problem the name of the certificate/secret (istio-ingressgateway-certs) or that both secret and certificate had the same name. Thank you for your help.
It is also missing in documentation how to deal with namespaces. Do we allways have to create certificates in the same namespace as ingress gateway? If it can be created in different namespace, how to specify namespace for credentialName?
jurna
on 24 May 2019
when using SDS you don't need (and maybe must not) use istio-ingressgateway-certs.
sds explicitly filters out prefixes of "istio" and "prometheus" :-) of course it's not documented. It will also skip secrets that have a field named "token" (commonly the service accounts).
If it can be created in different namespace, how to specify namespace for credentialName?
Sds will try to fetch the secret from the same namespace the ingress deployment is in. There's no code currently to support other namespaces.
farcaller
on 24 May 2019
Has this issue been resolved? @yacut
myidpt
on 2 Aug 2019
@myidpt use Istio 1.2.x + SDS
prune998
on 3 Aug 2019
@prune998 Do you have any functional example? I have not been able to make it work...
cymarechal
on 11 Aug 2019
check for https://medium.com/@prune998/istio-1-1-7-lets-encrypt-working-9100cea9f503 for some hints.... Maybe I should create a step by step doc ?
prune998
on 12 Aug 2019
@prune998 Thank you for the article! Actually I already followed it without noticing it was yours. I guess my error comes from my Istio configuration. I tried tweaking the "demo-auth" configuration profile but I don't really understand the difference between "global.sds" (and its link with "global.controlPlaneSecurityEnabled") and "gateways.istio-ingressgateway.sds"...
cymarechal
on 12 Aug 2019
Not sure what your question is :)
Enabling SDS on the Gateway (to gather SSL certificates) is one thing. Enabling mTLS and using SDS for it is another thing, which I'm not using and can't tell about :)
But there are 2 separate subjects. Istio Gateway (Envoy proxy) needs a special config to use SDS, which is triggered by the option set gateways.istio-ingressgateway.sds.enabled=true when installing from the Helm chart.
prune998
on 12 Aug 2019
I was looking for explanations about these settings, which you just did. It already helps me a lot, thank you! :)
cymarechal
on 12 Aug 2019
I've been dealing with this the whole week (using http01). I'll relate a few issues that are pointing to the same subject. This seems to be the biggest discussion on the subject, so I'll pin it here:
https://github.com/istio/istio/issues/5421
https://github.com/istio/istio/issues/7075
https://github.com/istio/istio/issues/13113
https://github.com/istio/istio/issues/15277
https://github.com/istio/istio/issues/15148
Possible solutions
https://github.com/jetstack/cert-manager/issues/1097 (_not tested_)
https://discuss.istio.io/t/using-gateway-virtualservice-http01-sds/2534/5
https://discuss.istio.io/t/using-gateway-virtualservice-http01-sds/2534/8
I haven't solved this; I'm still getting a 503 in the certificate challenge trying to reach the pod.
gustavovalverde
on 13 Sep 2019
cc: @munnerz @howardjohn @prune998 as you might have updated information / guidelines on the subject
gustavovalverde
on 13 Sep 2019
@gustavovalverde I'm not using the HTTP01 provider, the workflow is too complicated. DNS01 is working great though.
Sorry I can't help right now.
prune998
on 13 Sep 2019
I finally got this working (with a bunch of hours of investigation and trial & error).
I have a very _particular scenario_ were we have access to our domain DNS (which is easy using dns01), but we also have to issue certificates for external domains owned by some of our clients, and as we can't use dns01 our only option is http01
Using istio 1.2.2 and cert-manager 0.10, we "solved" it with this approach:
Preparing cert-manager
cert-manager issuer
This had to comply with both approaches so we decided to use selectors; one using http01 for external domains that we didn't control, and the other one for our domains.
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: istio-system
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- selector:
dnsNames:
- "service.client1.com"
- "service.client2.net"
- "service.client3.org"
http01:
ingress:
class: istio
- selector:
dnsNames:
- "mydomain.com"
- "*.mydomain.com"
dns01:
clouddns:
project: my-project
serviceAccountSecretRef:
name: cert-manager-credentials
key: gcp-dns.json
cert-manager certificate(s)
One for our domain:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: mydomain-cert
namespace: istio-system
spec:
secretName: mydomain-tls-cert
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
commonName: mydomain.com
dnsNames:
- sub1.mydomain
- sub2.mydomain
- sub3.mydomain
A new one for each external domain:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: client1-cert
namespace: istio-system
spec:
secretName: client1-tls-cert
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
commonName: service.client1.com
dnsNames:
- service.client1.com
Preparing Istio
istio Gateway
We need to use a "special" gateway for cert-manager to find the pod with the generated challenge. The easiest way to make it work is adding the _istio-autogenerated-k8s-ingress_ allowing all traffic to port 80 (this could be from the helm install parameters or by manually adding it)
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-autogenerated-k8s-ingress
namespace: istio-system
labels:
app: ingressgateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
protocol: HTTP2
name: http
hosts:
- "*"
---
istio Destination Rule
The gateway alone won't work if mTLS is active, so we need to disable mTLS for Let's Encrypt, which uses port 8089, otherwise we will get a 503 error on the certificate challenge.
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: cert-htt01-challenge
namespace: istio-system
spec:
host: "*.istio-system.svc.cluster.local"
trafficPolicy:
tls:
# keeping the same mTLS mode as `default` DestinationRule in istio-system
mode: DISABLE
portLevelSettings:
- port:
# CertManager generate services to perform the challenge on port 8089
# it looks so far no other services in istio-system use this port
number: 8089
tls:
mode: DISABLE
Preparing your custom gateway
istio custom-gateway
For each certificate that gets created, a new HTTPS port has to enabled with the respective certificate secret, as cert-manager does not allow to have different certificates on the same secret.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: custom-gateway # Individual name (this is the one to be used in the VirtualServer)
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP2
tls:
httpsRedirect: true
hosts:
- "*"
- port:
number: 443
name: https-client1
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: "client1-tls-cert"
hosts:
- "service.client1.com"
- port:
number: 443
name: https-default
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: "mydomain-tls-cert"
hosts:
- "*.mydomain.com"
Summary
This will allow you to have http01 and dns01 working side by side; or just one of them if need it (for dns01 you won't need all that hassle with gateways and DRs)
There's mainly 2 problems with this approach:
- You have to delete the _istio-autogenerated-k8s-ingress_ gateway for your service to work correctly after your certificate is issued. There's a weird behavior when this gateway is side by side working with the custom one, causing redirection between http and https
- Each new certificate that you create will also require manual intervention on the custom gateway, as cert-manager does not support multiple certificates on the same secret.
References:
https://medium.com/@gregoire.waymel/istio-cert-manager-lets-encrypt-demystified-c1cbed011d67
https://discuss.istio.io/t/using-gateway-virtualservice-http01-sds/2534/8
gustavovalverde
on 13 Sep 2019
This works for Istio 1.2.x and letsencrypt that comes bundled with it at least. I could not get it to work with Istio 1.3.0; I have yet to figure out why.
Instead of Istio returning 503s for .wellknown... URIs, I get 404s, which indicates some kind of routing issue that I was unable to determine. I wonder if ACME solver pod templates might be the solution?
https://github.com/jetstack/cert-manager/issues/1097
Then again, I could be totally wrong. Perhaps I can get time to test it out this weekend.
gswallow
on 17 Sep 2019
@gswallow I also failed to make it works with 1.3.0. It seems like the traditional K8S ingresses are not converting to gateway and virtual host automatically with 1.3.0. Not sure is that a bug of 1.3.0 or is this something people expected?
fangpenlin
on 18 Sep 2019
@fangpenlin can you look at https://github.com/istio/istio/issues/17148 and see if setting those values in the mesh config (istio configmap) fixes the problem? We may have erroneously changed the default value breaking inrgess
howardjohn
on 18 Sep 2019
@howardjohn I will give it a try and let you know
fangpenlin
on 19 Sep 2019
@howardjohn yep, I can confirm ingress works after adding those values into config map mesh field
fangpenlin
on 19 Sep 2019
Thanks @fangpenlin ! I have a fix that will be in 1.3.1 next weekish. Its basically just setting those values, so that should be a good proper fix until then: https://github.com/istio/istio/pull/17190
howardjohn
on 19 Sep 2019
🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2019-09-19. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.
_Created by the issue and PR lifecycle manager_.
istio-policy-bot
on 3 Jan 2020
@howardjohn @gustavovalverde @prune998
Need one help here with your expertise.
I am unable to receive a working certificate thru letsencrypt prod/staging API:
The current setup includes below approach
- K8s cluster setup with 1.0.2 Istio Dex Yaml, mentioned in kubeflow.org documentation, which comes with istio 1.3.1 and cert-manager by default.
- As my major task is to complete this with kubeflow enabled with istio-ingressgateway along with HTTPS Lets-Encrypt ACME SSL Certificate generation.
Kindly let me know a possible solution.
K8s API version 1.14
Istio -1.3.1
Cert-Manager: the same which comes with default manifest.
Anujay-Saraf
on 22 Jul 2020
Related issues
Stono
·
92Comments
hamon-e
·
59Comments
fhoy
·
139Comments
lhotrifork
·
78Comments
linsun
·
58Comments
Most helpful comment
Im not going to be using istio until they sort out some basics, like using letsencrypt. I might be the only 1 who feels like that, but ya know, just sayin.