Jx: boot: Support external-dns with AWS/EKS and TLS

[ccojocar]

Summary

external-dns has support for Route53
https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/aws.md

To enable TLS automation over lets encrypt, we should:

• setup the external-dns in boot https://github.com/jenkins-x/jenkins-x-boot-config/blob/501f77211f21bbdfc181aebc82a58b6259c2e498/systems/external-dns/values.tmpl.yaml#L7 to use Route53 https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/aws.md
• setup the cert-manager's issuers in boot https://github.com/jenkins-x/jenkins-x-boot-config/blob/501f77211f21bbdfc181aebc82a58b6259c2e498/systems/acme/templates/cert-manager-prod-issuer.yaml#L25 to use the DNS challenge https://docs.cert-manager.io/en/latest/tasks/issuers/setup-acme/dns01/route53.html

Steps to reproduce the behavior

Expected behavior

Actual behavior

Jx version

The output of jx version is:

COPY OUTPUT HERE

Jenkins type

• [x] Serverless Jenkins X Pipelines (Tekton + Prow)
• [ ] Classic Jenkins

Kubernetes cluster

Operating system / Environment

[ccojocar]

cc @dgozalo @daveconde

[hugoduncan]

We just set this up on our jenkins-x instance. Changes made were:

• changed external-dns values to use aws provider, with kube2iam pod annotations
• changed cm values to add --issuer-ambient-credentials extraArg, and kube2iam podAnnotations
• changed acme templates to use route53 solver instead of clouddns
• add IAM roles for the above

Also had to manually delete the existing cert-manager deploy, orders and challenges.

[Larzack]

When do you think this issue can be fixed ?

[tdcox]

LetsEncrypt policy is to block old versions of cert-manager 3 months after the release of the latest so the 0.9.0 release may not remain viable for much longer.

[dgozalo]

We will be moving to 0.11 on boot clusters very soon for both EKS and GKE.

We unfortunately had to lock it to 0.9.1 for jx install based clusters because of a dependency problem, but it will be deprecated by February anyway.

[polothy]

Food for thought - you could avoid using cert-manager altogether in AWS/EKS by attaching a wildcard certificate to the NLB which should be possible in k8s 1.15 or later. The certificate is issued from AWS Certificate Manager which is so far, very easy for us to use and it handles all the auto-renew stuff. To attach a cert, you just add an annotation on your Ingress/Service that is associated to the NLB: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "certificate ARN"

Then you can use external-dns to register any hosts you may need. We have been using it and it works really well so far.

[dgozalo]

@polothy that's a great idea indeed.

Sadly, EKS doesn't support k8s 1.15 yet, but we will be looking into doing that as soon as they release it.

[polothy]

@dgozalo did you mean to close this?

EKS support for 1.15 is in the works and I think kops just released support for it.

[dgozalo]

Yes, sorry for not giving context on it but version 0.11 of cert-manager and external-dns have been enabled in Jenkins X boot through https://github.com/jenkins-x/jenkins-x-versions/pull/710 and https://github.com/jenkins-x/jenkins-x-boot-config/pull/119.

There should be documentation and a demo coming up soon.

About EKS 1.15, we'll open another issue to support AWS Certificate Manager when it's supported :)

[pguimaraes]

@hugoduncan Since you were able to get 'jx boot' working with external-dns with AWS/EKS and TLS, would you be willing to provide more detailed documentation on what you did?

I am no longer able to run 'jx upgrade ingress' to create TLS certificates for applications running in my existing EKS cluster where jx was previously installed with 'jx install'. And when I try to use 'jx boot' to install jx in my new EKS cluster, I can't get TLS to work. This is a big problem for me so any help would be greatly appreciated!

[dgozalo]

@pguimaraes we will be releasing documentation for EKS very soon and we cover how to configure External DNS.

[alex-sainer]

@dgozalo any updates for the dokumentation-stuff? trying to get JX working on eks with route53 but i've no idea how to get it working... -.-

or maybe @hugoduncan can share more information about his workaround?

[carlosrodlop]

@dgozalo any updates for the dokumentation-stuff? trying to get JX working on eks with route53 but i've no idea how to get it working... -.-

or maybe @hugoduncan can share more information about his workaround?

https://jenkins-x.io/docs/install-setup/installing/create-cluster/eks/#externaldns