ingress-nginx using default fake SSL certificate even if different certificate specified in ingress yaml

[lsambolino]

NGINX Ingress controller version: unable to verify

Kubernetes version (use kubectl version): 1.13

Environment:

• Cloud provider or hardware configuration: On premise, underlying hypervisor is VMware
• OS (e.g. from /etc/os-release): Debian
• Kernel (e.g. uname -a): Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 GNU/Linux-
• Install tools: N/A
• Others: N/A

What happened: Nginx istances not using the tls specified in the ingresses but sticking to the default certificate in local /etc/kubernetes/ssl folder

What you expected to happen:
Nginx istances should use certificates specified by ingresses instead of the default ones.

Local nginx did not receive the configuration specified in the ingress text.

How to reproduce it:
Implement an ingress like the following one:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "1m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "3"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "on"
    nginx.ingress.kubernetes.io/ssl-redirect: "False"


    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"


    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ( $request_method !~ ^(GET|OPTIONS|HEAD)$ ) {
          return 405;
      }
      set $vpn "yes";
      if ($not_vpn_ip) {
          set $vpn "no";
      }

      if ($ingress_name != "") {
        set $service "assets_apps";
        set $subenv "our_apps";
        set $service_id "apps_mycompany-assets_apps";
        set $real_ingress "true";
      }
      more_set_headers "X-companyname-Uuid: $tid";
      more_clear_headers 'x-companyappname-*';
      more_clear_headers 'x-powered-by';
      more_clear_headers 'x-our-orchestrator';
      more_clear_input_headers 'x-ourwebsite-cache-id';
      more_set_headers "Our-Service-Name: $service";
      more_set_headers "Our-Service-Subenv: $subenv";
      more_set_headers "Our-Service-Id: $service_id";
      more_set_headers "Our-Service-Ingress: $ingress_name";
      more_set_headers "strict-transport-security: $strict_header";
      proxy_set_header X-CompanyHeader-UUID $tid;

      set $cors "false";
      set $cors_done "false";

  name: mywebsite-apps
  namespace: apps--ourcompany
spec:
  tls:
  - hosts:
    - mywebsite
    secretName: mywebsite-certs
  rules:
  - host: mywebsite
    http:
      paths:

      - path: /Apps(.*)
        backend:
          serviceName: apps--ourcompany-assets--apps-default-service
          servicePort: 8090

Verify if the NGINX Configuration appears as this:

server {
        server_name apps.ourwebsite.it ;

        listen 80 proxy_protocol;

        set $proxy_upstream_name "-";
        set $pass_access_scheme $scheme;
        set $pass_server_port $server_port;
        set $best_http_host $http_host;
        set $pass_port $pass_server_port;

        listen 443 proxy_protocol  ssl http2;

        # PEM sha: 5c5cb985aced79d49bef4011061c96ef8882fbf4
        ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;

        ssl_certificate_by_lua_block {
            certificate.call()
        }

        location / {

            set $namespace      "ourstore";
            set $ingress_name   "apps.ourwebsite.it-ourstore";
            set $service_name   "ourstore-assets--ourstore-default-service";
            set $service_port   "8090";
            set $location_path  "/";

            rewrite_by_lua_block {
                lua_ingress.rewrite({
                    force_ssl_redirect = true,
                    use_port_in_redirects = false,
                })
                balancer.rewrite()
                plugins.run()
            }

            header_filter_by_lua_block {

                plugins.run()
            }
            body_filter_by_lua_block {

            }

            log_by_lua_block {

                balancer.log()

                monitor.call()

                plugins.run()
            }

            if ($scheme = https) {
                more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains";
            }

            port_in_redirect off;

            set $proxy_upstream_name    "ourstore-ourstore-assets--ourstore-default-service-8090";
            set $proxy_host             $proxy_upstream_name;

            client_max_body_size                    1m;

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;

            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Request-ID           $req_id;
            proxy_set_header X-Real-IP              $the_real_ip;

            proxy_set_header X-Forwarded-For        $the_real_ip;

            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;

            proxy_set_header X-Original-URI         $request_uri;

            proxy_set_header X-Scheme               $pass_access_scheme;

            # Pass the original X-Forwarded-For
            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers to proxied server

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      120s;
            proxy_read_timeout                      120s;

            proxy_buffering                         off;
            proxy_buffer_size                       4k;
            proxy_buffers                           4 4k;
            proxy_request_buffering                 on;

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            # In case of errors try the next upstream server before returning an error
            proxy_next_upstream                     error;
            proxy_next_upstream_tries               3;

            if ( $request_method !~ ^(GET|POST|OPTIONS|DELETE|PUT|HEAD)$ ) {
                return 405;
            }
            set $vpn "yes";
            if ($not_vpn_ip) {
                set $vpn "no";
            }

            if ($ingress_name != "") {
                set $service "assets_ourstore";
                set $subenv "ourstore";
                set $service_id "ourstore-assets_ourstore";
                set $real_ingress "true";
            }
            more_set_headers "X-Our-UUID: $tid";
            more_clear_headers 'x-ourapp-*';
            more_clear_headers 'x-powered-by';
            more_clear_headers 'x-our-orchestrator';
            more_clear_input_headers 'x-our-cache-id';
            more_set_headers "X-Our-Service-Name: $service";
            more_set_headers "X-Our-Service-Subenv: $subenv";
            more_set_headers "X-Our-Service-Id: $service_id";
            more_set_headers "X-Our-Service-Ingress: $ingress_name";
            more_set_headers "strict-transport-security: $strict_header";
            proxy_set_header X-Our-UUID $tid;

            set $cors "false";
            set $cors_done "false";

            proxy_pass http://upstream_balancer;

            proxy_redirect                          off;

        }

    }

More specifically I don't expect nginx to use the local default SSL certificate as specified in the following line:

ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem;

[janosi]

@lsambolino ssl certs are handled dynamically by this part of the nginx.conf file

ssl_certificate_by_lua_block {
            certificate.call()
        }

I.e. the certificates are not listed in the nginx.conf file explicitly. The ingress controller loads the certs and keys into a shared memory area when it processes the Ingress definitions and this lua script referenced here gets the cert from that cache when a request is processed by nginx.

If according your your experience TLS does not work with your config please check the nginx logs. It writes logs if it has problems with the secret defined in tls.secretName: mywebsite-certs

Possible reasons are:

• the secret is not in the same namespace with the ingress definition
• the cert or key is not stored properly in the secret (the relevant key names in the secret should be tls.crt and tls.key respectively)
• the validation of the cert reveals that it does not have CN or SAN for the domain/host defined in the Ingress.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[chrissound]

Also running into this error, strangely this only occurs with one particular namespace with a name of staging.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mac2000]

But even if certificate is in place and valid (e.g. browser and curl says everything is ok) default certificate is still being used if there is no SNI in request, e.g.:

openssl s_client -connect contoso.com:443

will answer with default certificates provided in config

[jgato]

Something similar happens to me. The situation is: I have several services:
service.domain.com
service2.domain.com
using https and a self signed certificate. To resolve these domains, I have added some entries to my /etc/hosts. In that way everything is working ok, I am forwarded to the proper service, receiving a different self-sigende cert each.
But, now that I have real DNS, I am no longer using my /etc/hosts, becuase these domains are really existing ones. So now the ingress controller is failing:

for service.domain.com, the service is properly accesible but with an HTTPS FAKE default cert provided by the nginx controller

for service2.domain.com the service is not accessible, I am always recieving 503 error and these Fake default cert.

If I add again these domains to my /etc/hosts, everything is working ok again.

[mac2000]

4674 and this one is kind of related not even sure which one is correct

[jgato]

my problem was solved, my fault. It was a problem with the DNS server :(

[nelsonyaccuzzi]

@jgato can you tell us what was the problem?

[jgato]

@nelsonyaccuzzi very funny situation was my case.
By mistake, I asked to add the DNS entries to my CT Team. I gave them wrong details, so the DNS entries were pointing wrongly. But!!!!! just a matter of coincidende were pointing to wrong IPs that also had an nginx not working. NGINX is the proxy I am also using. This is why I had these 503 error from NGINx!!! These were no my servers.-
is not funny? ;)