Aws-load-balancer-controller: [Feature Request] Import cert from tls.secretName into ACM

[tcolgate]

Hi,
It looks like ACM now permits importing certificates via the API, I'm pretty sure this has been missing previously.
Would you be interested in a PR to sync secrets to ACM? I've started working on it under the assumption that you would (didn't see any existing PRs).
My current plan is:

• if tls.secretName is set, try and pull a cert, key and certchain from the secret, following format of the secrets used by CertificateManager.
• if all are good, import that cert, if that works, add an annotation to the secret to indicate the ARN
• keep a cache of hashes of some of the cert info (isuer, issued at and issued by), and compare those with the related info from ACM, use those to indicate if an update is needed.

[tcolgate]

I've started digging around the code a little. It's a bit more heavy going than expected. I'm going to have a poke at it, but it's not going to be quite as quick as expected (may not be able to give it high priority as we can, for now, manage with manually managed ACM certs)

[M00nF1sh]

@tcolgate
Actually i had an PR for this but closed it.
The main reason is ACM only allows u to import like 10 certs per year(total imports instead of active imported cert), which basically makes it a useless feature.

seems they changed it to 1000 now....

[gmauleon]

Nice!
FYI, related issue on the certmanager side https://github.com/jetstack/cert-manager/issues/2302

[alejandrox1]

adding some labels to make this more discoverable to others -the idea sounds good
also, here are the docs I could find regarding the limit of 1000 https://docs.aws.amazon.com/acm/latest/userguide/acm-limits.html

/kind feature
/priority important-longterm

[tcolgate]

A lot of the AWS limits can be raised with service requests, though I've not looked at cert limits I must admit.

[tcolgate]

@M00nF1sh that looks very promising.

[Morriz]

@M00nF1sh can you maybe pick up that closed PR? Would be awesome to just mention a list of secrets in an ingress annotation so that they get synced with ACM, and their ARNs added to the alb.ingress.kubernetes.io/certificate-arn list.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[metral]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[Morriz]

/remove-lifecycle rotten
/remove-lifecycle stale

[vainkop]

Any updates on using cert-manager let's encrypt certs + alb + eks?

[marcellodesales]

@M00nF1sh any work on this? Looking for the same here... Any progress with lestencrypt + cert-manager

[M00nF1sh]

@marcellodesales
I'll put this into our backlog, we should be able to work on this post reinvent.

[vainkop]

Sharing my not very elegant but working hack for Cert-Manager Let's Encrypt certs + ALB + EKS using a combination of kubectl & aws cli:

1) Use the following cluster issuer to get the certificate into the k8s secret:
ClusterIssuer.yml:

kind: ClusterIssuer
apiVersion: cert-manager.io/v1
metadata:
  name: prod-clusterissuer
spec:
  acme:
    privateKeySecretRef:
      name: prod-clusterissuer
    server: 'https://acme-v02.api.letsencrypt.org/directory'
    solvers:
      - http01:
          ingress:
            class: alb
            ingressTemplate:
              metadata:
                annotations:
                  alb.ingress.kubernetes.io/scheme: "internet-facing"
                  alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
                  alb.ingress.kubernetes.io/group.name: "group1"

2) Deploy the app using Helm:
app-helm-values.yml (only relevant part):

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: "alb"
    alb.ingress.kubernetes.io/scheme: "internet-facing"
    alb.ingress.kubernetes.io/listen-ports : '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/group.name: "group1"
    kubernetes.io/tls-acme: "true"
    alb.ingress.kubernetes.io/certificate-arn: ""
  hosts:
    - host: $HOST
      paths:
        path: /
  tls:
    - secretName: $HOST-tls
      hosts:
        - $HOST

3) Wait for the certificate to be created in the K8s secret by cert-manager & then import it into the ACM using aws cli:

kubectl get secret $HOST-tls -o yaml | yq r - data.[tls.crt] | base64 -d | sed -e '/-----END CERTIFICATE-----/q' > $CI_PROJECT_DIR/Certificate.pem

kubectl get secret $HOST-tls -o yaml | yq r - data.[tls.crt] | base64 -d > $CI_PROJECT_DIR/CertificateChain.pem

kubectl get secret $HOST-tls -o yaml | yq r - data.[tls.key] | base64 -d > $CI_PROJECT_DIR/PrivateKey.pem

aws acm import-certificate \
  --certificate fileb://Certificate.pem \
  --certificate-chain fileb://CertificateChain.pem \
  --private-key fileb://PrivateKey.pem

export CERTIFICATE_ARN=$(aws acm list-certificates --query CertificateSummaryList[].[CertificateArn,DomainName] --output text | grep "$HOST" | cut -f1 | head -n 1)

kubectl annotate --overwrite ingress app alb.ingress.kubernetes.io/certificate-arn=$CERTIFICATE_ARN

In my case head -n 1 always shows the certificate which is in use by ALB & to verify it I'm using the following query expecting a non empty output:
aws acm describe-certificate --certificate-arn $(aws acm list-certificates --query CertificateSummaryList[].[CertificateArn,DomainName] --output text | grep "$HOST" | cut -f1 | head -n 1) --query Certificate.InUseBy --output text

You can use it without head -n 1 & loop over the output ARNs to find which certificate ARN is in use & use exactly it (in case the behavior of the aws cli list-certificates output changes.

[marcellodesales]

@vainkop That's a great start... thank you! As a next step... Independently of the ClusterIssuer type used (DNS0, or HTTP01), we could have a container listening to events on the creation of the certs...

Questions

• Did that work leaving the certificate-arn as empty? I hadn't check using that...
• Didn't you need to refer to the cluster issuer? prod-clusterissuer?

Next steps maybe?

• Require the user to create an IAM Role with permissions to the ACM APIs
• Create an AWS CLI container that listens to the events of the Cert creation
  
  
  
  • It should have the OIDC role created for the ServiceAccount for the container
  
  
• Implement the CRD that listens to the events of certs rotation
  
  
  
  • so that we can renew the cert...
  
  

I wish I had time to tacle that, but I hope to see the work from @M00nF1sh if he's closer to get done...

[vainkop]

Questions

• Did that work leaving the certificate-arn as empty? I hadn't check using that...
  Yes it works or you can create the ingress without this annotation & set the port to f.e. 80 only just to get the traffic into the acme solver for the certificate to be created in the K8s secret & then import the certificate into ACM & annotate the ingress with the certificate arn & add the 443 port (that's how I did it)
  kubectl annotate --overwrite ingress app alb.ingress.kubernetes.io/listen-ports='[{"HTTP":80}, {"HTTPS":443}]'

• Didn't you need to refer to the cluster issuer? prod-clusterissuer?

No I don't need to refer to the cluster issuer as I've used the following options to install the cert-manager:

helm upgrade -i cert-manager jetstack/cert-manager \
  --set ingressShim.defaultIssuerName=prod-clusterissuer \
  --set ingressShim.defaultIssuerKind=ClusterIssuer \
  --set ingressShim.defaultIssuerGroup=cert-manager.io \
...

Next steps maybe?

• Require the user to create an IAM Role with permissions to the ACM APIs

Currently It works like that for me as I'm using the CI with IAM credentials enough to perform the necessary operations.

• Create an AWS CLI container that listens to the events of the Cert creation

Here's what I use to wait for the certificate (I know it's a little ugly, but again: it's working)
while ! kubectl get secret $HOST-tls; do echo "The certificate is not ready yet. Sleeping 10 seconds..." && sleep 10; done

It's all automated in one pipeline.

• Implement the CRD that listens to the events of certs rotation
  
  
  
  • so that we can renew the cert...
  
  

I can see how it can be automated easily, f.e. by creating a k8s job which checks the certificate expiration & then triggers the pipeline job to reimport the certificate like I've written above.

I really hope someone implements it in the cert-manager itself without all this bashism! :)

[vainkop]

& while that works I'm hitting the aws api & acm limits :(
An error occurred (LimitExceededException) when calling the ImportCertificate operation (reached max retries: 2): You have imported the maximum number of 20 certificates in the last year.

[Morriz]

I asked them for an increase and was granted the maximum per year of 5000 ;)

Also, I have created a similar solution that I abandoned because of that same limitation. I just decided to avoid ALBs until something would change.
You can find that part in something much bigger that we open sourced last week under the radar. Please don't expect all documentation to match the developer experience just yet.
Please look at redkubes.github.io/otomi/ and from there check out the parts. You should find it in the tasks repo.

Would love to get feedback!

[vainkop]

I asked them for an increase and was granted the maximum per year of 5000 ;)

Also, I have created a similar solution that I abandoned because of that same limitation. I just decided to avoid ALBs until something would change.
You can find that part in something much bigger that we open sourced last week under the radar. Please don't expect all documentation to match the developer experience just yet.
Please look at redkubes.github.io/otomi/ and from there check out the parts. You should find it in the tasks repo.

Would love to get feedback!

Yes, it's not a problem to get an increase.

The other limitation I'm concerned with is the absense of "normal" rewrites on the ALB side. The solution using "actions" doesn't look clean to me. Issue: https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/835

The link on the page is broken?
https://redkubes.github.io/otomi/docs/about-otomi
https://redkubes.com/otomi-tasks/

Holy kubes! You seem to have found a page that no longer exists...
We are very sorry for this inconvenience.

I've found the relevant code in the repo (https://github.com/redkubes/otomi-tasks/blob/master/src/tasks/otomi/certs-aws.ts)
& it looks like you're basically doing the same thing as me but in typescript & async-like :)

I'm actually also thinking about abandoning ALBs because of all that.

Is using NLB + nginx/haproxy/... ingress controller in daemonset/... the most common practice on AWS/EKS or there's something better & more "AWS/EKS native"? Of course the main idea for using ALB was integration with other AWS services like WAF & etc

[Morriz]

The link on the page is broken?

Thanks for the feedback, just fixed it.

I'm actually also thinking about abandoning ALBs because of all that.

Is using NLB + nginx/haproxy/... ingress controller in daemonset/... the most common practice on AWS/EKS or there's something better & more "AWS/EKS native"? Of course the main idea for using ALB was integration with other AWS services like WAF & etc

We automated all of it, but it feels wrong, like you said. If somebody wants to contribute/maintain our so called "Task" that would be awesome. We don't give it prio now (also because NL is much more Azure oriented, unfortunately).