Application-gateway-kubernetes-ingress: Ability to add a custom root Certificate Authority cert to the Application Gateway

[alex-skinner]

I am working on deploying our application into a newly created AKS cluster using the Application Gateway for ingress which all seems to be working correctly. However, I am trying to use a set of custom certificates (a CA and a signed SSL cert) that I have generated and initially, I was getting issues as the health probes were all failing with the message

The root certificate of the server certificate used by the backend does not match the trusted root certificate added to the application gateway. Ensure that you add the correct root certificate to whitelist the backend

After doing some research, I realised that I could add my custom CA into the HTTP Settings through the portal:
AppGateway > HTTP settings > [Name of HTTP setting to change] > Uncheck Use Well Known CA Certificate > Upload CER file & set Override host name to my own custom host name.
Microsoft docs

or via PowerShell:
Microsoft docs

This did resolve my issue and my health probes appeared as 'healthy' after changing these settings.

However, they get overridden periodically, meaning I have to keep changing it back which (even though it's only a Dev environment) is less than ideal.

In a prod environment, we will be using a cert signed by a one of the known trusted root CA's but we will have several development/testing environments that we will be standing up and tearing down periodically so using custom CA/ssl cert would be easier.

My thinking was to have a couple of additional annotation's:

1. RootCA
   appgw.ingress.kubernetes.io/RootCA: <SOME_KUBERNETES_SECRET>
   Where the K8s secret is the CA CER file.

2. Override host name
   `appgw.ingress.kubernetes.io/OverrideHostName: ''

I would have a go at implementing this myself but I honestly wouldn't know where to begin. Also, apologies if I've missed something in the docs on how this could be achieved but I have had read through them and couldn't see anything. I am definitely not a certificate expert (anything but) so there may be a better way of going about this. Any advice would be great appreciated!

If it helps, here is the Azure CLI command to upload a CA cert to an app gateway
Docs

Update HTTP Setting to use root CA
Docs
There doesn't seem to be a --root-ca-cert parameter for this but there is an --auth-certs one so maybe this can be used in it's place. I believe authorization certs have been deprecated but not 100% sure.

Overriding host name
Docs
It would seem that the --host-name parameter is what we're looking for.

Thank you for all your hard work guys! :-)

[alex-skinner]

Having done a little more research, I came across the following Azure PowerShell cmdlet

Set-AzApplicationGatewayBackendHttpSetting

which has a parameter that looks like it would do the job
[-TrustedRootCertificate <PSApplicationGatewayTrustedRootCertificate[]>]

I've had a play with it and it seems to be a little hit and miss... and the cmdlet is also being deprecated in an upcoming release of Azure Powershell. Again, not sure if it helps but thought I'd pop the info here too.

docs

[akshaysngupta]

@alex-skinner These are great suggestions! I will add this to our backlog but can't comment on the ETA yet.

[alex-skinner]

No problem! Thank you again.

[nerijusk]

Seeing the backend hostname annotation implemented (https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/annotations.md#backend-hostname), this issue is the last missing part for a decent AGIC operation in environments where end to end TLS is a strict requirement. It would really help those running such environments to provide their own CA certificate for AppGw to use when communicating with backends over TLS. Many organisations have their own PKI and have internal CA they trust. Then encrypting communication all the way from client to the servicing pod can be achieved by using certificate signed by a well known CA for endpoint exposed on AppGw and then using certificates signed by internal intermediary CA for pods in AKS.

Ideally, the annotation would enable to specify a secret in AKS for AGIC to take the CA cert from and push into AppGw HTTP pool configuration.

Having integration with Azure Keyvault would be a bonus too.

There's a related issue to this: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/284

[nerijusk]

In addition to my previous comment, it is possible to upload and specify a custom CA certificate on application gateway http settings when the back-ends are not AKS pods. This should be treated as a feature parity request.

[akshaysngupta]

@nerijusk as trusted root certificate is not a secret, it doesn't belong in k8s secret.
Do you think putting the base64 encoded cert into the ingress resource would make any sense?
Also, this feature would need to support atleast 2 root certificates to allow for root certificate rotation on pod.
Another option could be a CRD.

[nerijusk]

I have a solution for automated certificate provisioning for pods by cert-manager. The certificate resource is created by initContainer on pod start. It waits for the corresponding secret to be created by cert-manager in Kubernetes before passing control to the main container. So the part of pod individual certificates is covered. The missing part is the issuing CA certificate pushed into http settings on application gateway.

You're right about the trusted root certificate. It's not a secret, but it is in one for the cert-manager together with its key. See https://cert-manager.io/docs/configuration/ca/.
Anyway, having it as base64 value in ingress resource should be sufficient. As long as ingress controller is able to set it in http settings in application gateway.

[bpinter]

Hi. This solution would be a must for our company as well. Like an Application Gateway which can drive traffic to the AKS, plus it can handle additional reverse proxies (outside AKS, path-based rule) with self-signed certificate endpoints.

[sandy-wang]

The lack of ability to set custom ca for http settings by agic is really annoying. Everytime when agic updates, the ca for all http settings go back to wellknown ca, which results error 502 for each single site the application gateway serves.
We would like to see this become an annotation to k8s ingress object, like other annotations, and we only need to define it in ingress rule, then agic would pick it up and change the http settings when configuring application gateway.

[akshaysngupta]

This is now supported through appgw.ingress.kubernetes.io/appgw-trusted-root-certificate annotation.

[sandy-wang]

@akshaysngupta could you advise which commit(s) or PR has been involved in this ticket? I cannot see the information here. Thank you!

[akshaysngupta]

You can try this feature out using version 1.2.0-rc1.
Here are instructions: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/annotations.md#appgw-trusted-root-certificate

[sandy-wang]

You can try this feature out using version 1.2.0-rc1.
Here are instructions: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/annotations.md#appgw-trusted-root-certificate

Thanks for the quick reply, I was looking through recent commits.