Cert-manager: Problem With Certification with Nginx

[altreze]

Describe the bug:
nginx-ingress is not able to load the certification generated by cert-manger.

Events:
  Type     Reason                   Age                 From                      Message
  ----     ------                   ----                ----                      -------
  Warning  AddedOrUpdatedWithError  17m (x3 over 130m)  nginx-ingress-controller  Configuration for default/nginx-ing was added or updated, but not applied: Error reloading NGINX for default/nginx-ing: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] cannot load certificate \"/etc/nginx/secrets/default-MY-DOMAIN-tls\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)\n"
finished with error: exit status 1
  Normal   AddedOrUpdated    11m  nginx-ingress-controller  Configuration for default/nginx-ing was added or updated
  Warning  UpdatedWithError  11m  nginx-ingress-controller  Configuration for default/nginx-ing was updated but was not applied: Error when updating config from ConfigMap: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] cannot load certificate \"/etc/nginx/secrets/default-MY-DOMAIN-tls\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)\n"
finished with error: exit status 1

The ingress configuration is :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.org/location-snippets: include /etc/nginx/custom-snippets/cors.conf;
    nginx.org/server-snippets: gzip on;
    nginx.org/websocket-services: altreze-xwindows-ws
  name: xwin-ing
  namespace: default
spec:
  rules:
  - host: MY-DNS
    http:
      paths:
      - backend:
          serviceName: altreze-xwindows
          servicePort: 5901
        path: /
      - backend:
          serviceName: altreze-xwindows-ws
          servicePort: 5900
        path: /websockify
  tls:
  - hosts:
    - MY-DNS
    secretName: MY-DNS-tls

Expected behaviour:
A concise description of what you expected to happen.

Steps to reproduce the bug:

• A working nginincnginx-ingress deployment
• A working cert-manger

Environment details::

• Kubernetes version (v1.17.0) baremetal;
• cert-manager version (0.14.0):
• ngininc nginx/nginx-ingress:1.6.3;

/kind bug

[RazaGR]

there is already an identical open issue https://github.com/jetstack/cert-manager/issues/2707

[altreze]

Then I will close this issue #2712 and keep tracking the problem on #2707.

[altreze]

The solution for #2707 does not solve this issue, I am reopening this one.

[ajcamilo]

any news about this? a workaround?

[durmisi]

any news about this? a workaround?

[munnerz]

What does the actual Secret resource containing your certificate/private key actually contain? Can you share a copy of the tls.crt entry for us to inspect? It looks like either the nginxinc ingress controller does not accept the certificate format we are using (which I've never seen before 🙄) or perhaps your certificate is not being issued in the first place.

/triage support
/remove-kind bug
/triage needs-information

[domfaber]

I have the same issue running kubernetes with AWS EKS and using cert manager to create the cert for nginx ingress. Would be great if I could get any help. Here is the error:

INX when updating Secret: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
stderr: "nginx: [emerg] cannot load certificate \"/etc/nginx/secrets/default-letsencrypt2-production\": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)\n"

[waelghaith]

There is any update for this issue ?!

[Dr4s1l]

Any update on this issue? Using HAPproxy and ClusterIssuer here, also following #2842 and #2715 None of suggested workaround works

[arvtiwar]

any update guys?

[arvtiwar]

@munnerz appreciate an update. Thanks

[arvtiwar]

Please let em know if you need any info

[adamthomason]

Also experiencing this same issue

[waelghaith]

any update guys?

Look at cert-manager logs, I think you will see logs have this link https://letsencrypt.org/docs/rate-limits/

[Dr4s1l]

I solved my issue by deleting a certificate.yaml helm charts which was required in older version, add the http01-edit-in-place annotation and this solve my issue. I followed the fact that the cm-acme-isuer doesn't get the ingress class name.

[2point0]

Have a look at https://github.com/jetstack/cert-manager/issues/2111 regarding featureGates. Setting that flag worked for me.

[meyskens]

Could you let us know if --feature-gates=ExperimentalCertificateControllers=true fixed the issue?
If not getting a copy of the secret (see https://github.com/jetstack/cert-manager/issues/2712#issuecomment-619086601) would be nice for us to take a look (of course please redact most of the private key)

[munnerz]

Based on https://github.com/jetstack/cert-manager/issues/2707#issuecomment-600589963, I believe this is caused by the tls.crt entry being empty for a brief period.

As @meyskens notes above, enabling the ExperimentalCertificateControllers will mean that cert-manager no longer creates empty tls.crt entries at any point, which shall hopefully resolve this (and based on @2point0's comment above, I think that's the case 😄)

I'm going to close this now as I think it's resolved, but if you are still running into the issue after upgrading & enabling this feature gate (and possibly deleting your existing Secret resource if it still has any empty tls.crt), please re-open 😄

[riupie]

let me know if this has been resolved. I have the same problem.

[robertgates55]

Likewise. Can we reopen?

[Imccccc]

Experiencing the same issue. Why does nginx finding a different secert name?

[robertgates55]

In our case, it transpired that there _was_ a tls secret without a tls.crt - but not for the ingress we were investigating. It got there because we were trying to fetch certs for an invalid domain.

Turns out that nginxinc’s nginx-ingress breaks all ingresses if _one_ of the ingresses has a dodgy certs secret... unideal.

[tamphuc0503]

I think this issue relates to the way nginx reload when add solvers into conf.d

E0715 15:50:10.102060 1 controller.go:559] Error updating endpoints for [aprilsea/cm-acme-http-solver-chkk8]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""

My ingress and issuer:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: admin1
namespace: aprilsea
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: "admin1"
spec:
tls:

• hosts:
  
  
  
  • admin1.test.com
    
    secretName: admin1-cert
    
    rules:
  
  
• host: admin1.test.com
  http:
  paths:
  
  
  
  • path: /
    
    backend:
    
    serviceName: admin
  
  

servicePort: 80

apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
name: admin1
namespace: aprilsea
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: admin1
solvers:
- http01:
ingress:
ingressTemplate:
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.org/redirect-to-https: "false"

Right after that, i exec 1 of nginx pods to inspect and found that:

1. this command => $ /usr/sbin/nginx -s reload
   returns:
   nginx: [emerg] cannot load certificate "/etc/nginx/secrets/aprilsea-admin1-cert": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)

2. inspect /etc/nginx/secrets/aprilsea-admin1-cert: only contains RSA key.

3. inspect cm-acme-solver returns:
   Configuration for aprilsea/cm-acme-http-solver-chkk8 was added or updated, but not applied: Error reloading NGINX for aprilsea/cm-acme-http-solver-chkk8: nginx reload failed:

Is this because cm-acme-solver write RSA secret first, create configuration and reload nginx. Because only key RSA exists so that unable to reload nginx, so configuration for both admin1.test.com configuration and solver configuration will not be applied. so, we cannot call http://admin1.test.com/.well-known/xxxx to write cert.

Can we have a patch for this?

[tamphuc0503]

I think this issue relates to the way nginx reload when add solvers into conf.d

E0715 15:50:10.102060 1 controller.go:559] Error updating endpoints for [aprilsea/cm-acme-http-solver-chkk8]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""

My ingress and issuer:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: admin1
namespace: aprilsea
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: "admin1"
spec:
tls:

• hosts:

  • admin1.test.com
    secretName: admin1-cert
    rules:

• host: admin1.test.com
  http:
  paths:

  • path: /
    backend:
    serviceName: admin
    servicePort: 80

apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
name: admin1
namespace: aprilsea
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: admin1
solvers:

• http01:
  ingress:
  ingressTemplate:
  metadata:
  annotations:
  ingress.kubernetes.io/ssl-redirect: "false"
  nginx.org/redirect-to-https: "false"

Right after that, i exec 1 of nginx pods to inspect and found that:

1. this command => $ /usr/sbin/nginx -s reload
   returns:
   nginx: [emerg] cannot load certificate "/etc/nginx/secrets/aprilsea-admin1-cert": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
2. inspect /etc/nginx/secrets/aprilsea-admin1-cert: only contains RSA key.
3. inspect cm-acme-solver returns:
   Configuration for aprilsea/cm-acme-http-solver-chkk8 was added or updated, but not applied: Error reloading NGINX for aprilsea/cm-acme-http-solver-chkk8: nginx reload failed:

Is this because cm-acme-solver write RSA secret first, create configuration and reload nginx. Because only key RSA exists so that unable to reload nginx, so configuration for both admin1.test.com configuration and solver configuration will not be applied. so, we cannot call http://admin1.test.com/.well-known/xxxx to write cert.

Can we have a patch for this?

I have fixed by preventing add more ing into current nginx config.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: admin1
namespace: aprilsea
annotations:
kubernetes.io/ingress.class: "nginx"
acme.cert-manager.io/http01-edit-in-place: "true"
cert-manager.io/issuer: "admin1"

Thank you so much. this is so great.

[gusnaughton]

I think this issue relates to the way nginx reload when add solvers into conf.d
E0715 15:50:10.102060 1 controller.go:559] Error updating endpoints for [aprilsea/cm-acme-http-solver-chkk8]: Error reloading NGINX when updating endpoints: nginx reload failed: Command /usr/sbin/nginx -s reload stdout: ""
My ingress and issuer:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: admin1
namespace: aprilsea
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: "admin1"
spec:
tls:

• hosts:

  • admin1.test.com
    secretName: admin1-cert
    rules:

• host: admin1.test.com
  http:
  paths:

  • path: /
    backend:
    serviceName: admin
    servicePort: 80

apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
name: admin1
namespace: aprilsea
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: [email protected]
privateKeySecretRef:
name: admin1
solvers:

• http01:
  ingress:
  ingressTemplate:
  metadata:
  annotations:
  ingress.kubernetes.io/ssl-redirect: "false"
  nginx.org/redirect-to-https: "false"

Right after that, i exec 1 of nginx pods to inspect and found that:

1. this command => $ /usr/sbin/nginx -s reload
   returns:
   nginx: [emerg] cannot load certificate "/etc/nginx/secrets/aprilsea-admin1-cert": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
2. inspect /etc/nginx/secrets/aprilsea-admin1-cert: only contains RSA key.
3. inspect cm-acme-solver returns:
   Configuration for aprilsea/cm-acme-http-solver-chkk8 was added or updated, but not applied: Error reloading NGINX for aprilsea/cm-acme-http-solver-chkk8: nginx reload failed:

Is this because cm-acme-solver write RSA secret first, create configuration and reload nginx. Because only key RSA exists so that unable to reload nginx, so configuration for both admin1.test.com configuration and solver configuration will not be applied. so, we cannot call http://admin1.test.com/.well-known/xxxx to write cert.
Can we have a patch for this?

I have fixed by preventing add more ing into current nginx config.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: admin1
namespace: aprilsea
annotations:
kubernetes.io/ingress.class: "nginx"
acme.cert-manager.io/http01-edit-in-place: "true"
cert-manager.io/issuer: "admin1"

Thank you so much. this is so great.

Hi @tamphuc0503. This looks promising, but I'm not quite following what you mean by this. Are you suggesting adding another ingress controller arbitrarily to the affected namespace? Thanks for helping clarify!!