Ingress-nginx: Error generating CA certificate chain

[kinolaev]

BUG REPORT:
--enable-ssl-chain-completion don't respect ingress annotation kubernetes.io/ingress.class

NGINX Ingress controller version:
0.19.0

Kubernetes version (use kubectl version):
v1.11.2

Environment:

• Cloud provider or hardware configuration: vSphere
• OS (e.g. from /etc/os-release): rancherOS
• Kernel (e.g. uname -a): Linux pws2 4.14.67-rancher2
• Install tools: Rancher 2.0.8
• Others:

What happened:
I have multiple ingress controllers configured with different --ingress-class and ingress resource with kubernetes.io/ingress.class annotation. But all controllers trying to check certificate chain for certificate referenced in that ingress resource.
I found this message in logs of each controller:

backend_ssl.go:161] Error generating CA certificate chain for Secret "mynamespace/mysecret": Invalid certificate.

where secret mynamespace/mysecret referenced only in one ingress with specified kubernetes.io/ingress.class annotation.

What you expected to happen:
Error message must appears only in controller's log specified with kubernetes.io/ingress.class annotation.

[mitchellmaler]

I just noticed this same issue. Our Vault which provides the CA Chain has a internal CA cert for its hostname. How can I add this to the ingress-nginx container so it validates the CA Chain correctly?

[SeriousM]

I have the same issue with a public certificate where the chain consists of

• my wildcard certificate
• GlobalSign Organization Validation CA
• GlobalSign Root CA

Could it be that the ingress have to know the GlobalSign Root CA to verify the certificate?

The client doesn't have an issue with the certificate as they know the root ca via windows updates.

[Zyzle]

@SeriousM Did you find a work-around for this? I'm seeing the same issue using a certificate chain exactly as you describe above.

[SeriousM]

Our solution was to create a certificate that contains the wildcard cert and the validation ca certificate. This version is valid for the browser side and valid for many ssl certificate checker services. The downside is that the ingress service is still complaining a out the invalid chain.

[Zyzle]

Yeah this is what we're seeing too, after making a cert chain the client side is happy enough but the ingress controller complains about invalid chain

[mitchellmaler]

I think the only workaroind which I have done in the past wirh other containers is to either create a custom ingress image where you just bake in your CA cert or do it with an init container somehow. That is until they alow you to pass the nginx controller a custom bundle that it can use.

[SeriousM]

Does you know how to install root CA in an image?
Can anyone tell me the function that checks for the certificate chain to
reproduce the problem under test conditions?

[mitchellmaler]

To do it during a docker build you can do something like this

FROM quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0

USER root

COPY certs.pem /usr/local/share/ca-certificates/certs.crt
RUN update-ca-certificates

USER www-data

[mitchellmaler]

I haven't tested it yet though to see if that fixes the error.

[mitchellmaler]

but the long term fix would be to allow mounting a ca bundle and passing a flag to the ingress-controller to use it.

[SeriousM]

It would be enough to install the global root CAs, any way to do that?

[dberuben]

+1 same issue for me, the TLS works fine

[neutrongenious]

+1 same issue

[jseparovic]

+1 same here. works fine, just getting the error

[prbajrac]

Do we have an ETA on this ?

[anton-johansson]

I have the same issue where we use a custom CA for our internal services.

Using the suggestion by @mitchellmaler, I had to tweak it a little bit:

FROM quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0

USER root

COPY custom-ca.crt /usr/share/ca-certificates/extra/
RUN echo "extra/custom-ca.crt" >> /etc/ca-certificates.conf && update-ca-certificates

USER www-data

This installs the CA properly, however I'm still getting the error messages. Also, it's really tedious to have to maintain our own image for this, it would be awesome if this could be supported natively somehow (if I could actually get rid of the error messages).

[amargherio]

@anton-johansson what errors were you seeing?

[anton-johansson]

Same as the OP:

Error generating CA certificate chain for Secret "kube-dashboard/star.svc.example.local": Invalid certificate.
Error generating CA certificate chain for Secret "default/star.svc.example.local": Invalid certificate.

star.svc.example.local is a self-signed certificate with our own custom CA. We use it for all internal services. The certificate is stored as secrets, one secret for each Kubernetes namespace that requires the certificate.

I'm going to take a look at the CLI argument --enable-ssl-chain-completion=false, but I'm not sure what downsides that have.

EDIT: The CLI argument got rid of the error messages. But again, I'm not sure what downsides it has. But it will have to do for now.

[amargherio]

We had a similar experience turning it off, but I also didn't pay attention to or dig in to find the side effects of leaving it disabled.

We had to re-enable chain completion due to some additional requirements and used your Dockerfile snippet above and haven't seen any errors in our logs (other than ingress rules with typo'd secret names or the like).

I'll take a look at the Dockerfile in question and see if there was anything different we did that would resolve the errors.

[Bujail]

Anybody got any idea about the Side effects for --enable-ssl-chain-completion=false ?

[viniciusps2]

Here we face the issue of having a certificate without the certificate authority running on a Kubernetes. We did following steps:

1) Concatenate both certificate and the certificate authority, in a temp file:

-----BEGIN CERTIFICATE-----
...(here is your certificate)...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...(here is your authority certificate)...
-----END CERTIFICATE-----

2) Encode to base64 in terminal:

cat some_temp_file | base64 -w 0

3) Put the resulting base64 in your Ingress deployment configuration at property "data.tls.crt":

apiVersion: v1
data:
  tls.crt: ... <-- here
  tls.key: ...
kind: Secret
metadata:
  name: sslcerts
type: Opaque

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[simo-eskalera]

Same issue here. SSLLabs.com reports the following error:

This server's certificate chain is incomplete. Grade capped to B.

I tried using the certificate chain file that was generated from letsencrypt certbot, but creating a secret doesn't work since the certificate chain doesn't match the private key.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[frittentheke]

/remove-lifecycle rotten

[ppodbielski]

+1 same issue for me

[YazzyYaz]

+1 same issue for me

Is there a fix for this?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SeriousM]

/remove-lifecycle rotten

This issue is still present and still occurs!

[SeriousM]

/reopen

[k8s-ci-robot]

@SeriousM: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SeriousM]

To everyone who still struggles with the CA problem:

You can create a secret with the certificate, key and the ca which builds the ca chain for you:
https://github.com/kubernetes/ingress-nginx/blob/76077ac0271e13da20b5bb04733e4d1b3043665f/docs/examples/auth/client-certs/README.md#creating-certificate-secrets

1. You can create a secret containing CA certificate along with the Server Certificate, that can be used for both TLS and Client Auth.
   kubectl create secret generic ca-secret --from-file=tls.crt=server.crt --from-file=tls.key=server.key --from-file=ca.crt=ca.crt

There is no need to annotate the ingress because the secret already has the full information in it.