Ingress-nginx: HTTPS consistangly gives Fake Kubernetes Certificate

[pkelleratwork]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

NGINX Ingress controller version:
helm stable/nginx-ingress

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:
staging

• Cloud provider or hardware configuration:
  Azure

• OS (e.g. from /etc/os-release):
  CentOS7

• Kernel (e.g. uname -a):
  Darwin Kernel Version 17.3.0: Thu Nov 9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64 x86_64

• Install tools:
  helm
  kubectl

• Others:
  have front end ingress controller pointed to pod containg nginx and multi app containers

What happened:
HTTPS presents 404 backend
HTTP works

What you expected to happen:
HTTPS should work

How to reproduce it (as minimally and precisely as possible):
$ helm install stable/nginx-ingress
$ kubectl get pods -n=kube-system

NAME                                                             READY     STATUS    RESTARTS   AGE
olfactory-prawn-nginx-ingress-controller-d44cd7595-mbklx         1/1       Running   0          28m
olfactory-prawn-nginx-ingress-default-backend-6d45c95c66-j8pgj   1/1       Running   0          28m

$kubectl get services -n=kube-system

NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
olfactory-prawn-nginx-ingress-controller        LoadBalancer   10.0.2.170     52.165.230.62   80:31713/TCP,443:32578/TCP   29m
olfactory-prawn-nginx-ingress-default-backend   ClusterIP      10.0.236.187   <none>          80/TCP                       29m

nginx-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: subdomain.domain.com
      http:
        paths:
          - backend:
              serviceName: ingress-nginx
              servicePort: 80
            path: /
  tls:
    - hosts:
      - subdomain.domain.com
    secretName: tls-secret



md5-cf505a7acd133a4885d8d9c0c7db1352



$ curl -v https://52.165.230.62
* Rebuilt URL to: https://52.165.230.62/
*   Trying 52.165.230.62...
* TCP_NODELAY set
* Connected to 52.165.230.62 (52.165.230.62) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate

SSL continues to default to Kubernetes fake certificate.

thanks.

Anything else we need to know:

[aledbf]

@pkelleratwork you cannot use the IP address. You need to use something like

curl -k -I --resolve subdomain.domain.com:52.165.230.62 https:/subdomain.domain.com/ 

or you will be directed to the default server (and that exposes the default SSL certificate)

[pkelleratwork]

 curl -k -I --resolve poppy.domain.com:52.165.230.62 https:/poppy.domain.com/
HTTP/2 404
server: nginx/1.13.5
date: Thu, 25 Jan 2018 19:54:42 GMT
content-type: text/plain; charset=utf-8
content-length: 21
strict-transport-security: max-age=15724800; includeSubDomains;

[pkelleratwork]

from url - https://poppy.domain.com/login

get - default backend 404, kubernetes fake cert....

[Stono]

I have some funky behaviour similar to this. When using -k on curl, I get the fake certificate:

❯ curl -kvL https://my-site.com >/dev/null
* Rebuilt URL to: https://my-site.com/
...
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: Kubernetes Ingress Controller Fake Certificate

And when I don't use -k, I get the actual cert:

❯ curl -vL https://my-site.com >/dev/null
* Rebuilt URL to: https://my-site.com/
....
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: *.autotrader.co.uk

Both requests return the correct content.

[pkelleratwork]

heres this if it helps....

kubectl describe ingress nginx-ingress -n=kube-system
Name:             nginx-ingress
Namespace:        kube-system
Address:
Default backend:  default-http-backend:80 (<none>)
TLS:
  tls-secret terminates poppy.domain.com
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  poppy.domain.com
                          /   nginx:80 (<none>)
Annotations:
Events:
  Type    Reason  Age   From                Message
  ----    ------  ----  ----                -------
  Normal  CREATE  19s   ingress-controller  Ingress kube-system/nginx-ingress

[pkelleratwork]

$ kubectl describe service olfactory-prawn-nginx-ingress-controller -n=kube-system
Name:                     olfactory-prawn-nginx-ingress-controller
Namespace:                kube-system
Labels:                   app=nginx-ingress
                          chart=nginx-ingress-0.8.26
                          component=controller
                          heritage=Tiller
                          release=olfactory-prawn
Annotations:              <none>
Selector:                 app=nginx-ingress,component=controller,release=olfactory-prawn
Type:                     LoadBalancer
IP:                       10.0.2.170
LoadBalancer Ingress:     52.165.230.62
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31713/TCP
Endpoints:                10.244.1.90:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  32578/TCP
Endpoints:                10.244.1.90:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

is there any other info i can provide for you?

[pkelleratwork]

closing. im an idiot. the cert i'm using was missing '---END CERTIFICATE----'

[dtandersen]

My problem was the ingress and secret weren't in the same namespace.

[evanrich]

I'm having this same problem. Trying to use gitlab +gitlab registry in K8s. Have cert loaded, if viewed from a web browser, real cert is provided (Comodo issued cert), but if I try openssl s_client, I get fake cert...even for sites that are set with
nginx.ingress.kubernetes.io/secure-backends: "true"

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nextcloud
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  tls:
  - hosts:
    - cloud.mysite.com
    secretName: star.mysite.com
  rules:
  - host: cloud.mysite.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nextcloud
          servicePort: 34443

openssl s_client -showcerts -connect cloud.mysite.com:443

results in:

CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
-----BEGIN CERTIFICATE-----
yaddayaddayadda
-----END CERTIFICATE-----
---
Server certificate
subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
---

but if I go to it in browser:

If I use -servername, it works
openssl s_client -showcerts -connect cloud.evanrichardsonphotography.com:443 -servername cloud.evanrichardsonphotography.com

however this doesn't fix my problem since gitlab-runners are throwing this same error

[aledbf]

@evanrich you are missing the -servername (SNI) parameter in the command

openssl s_client -showcerts -servername cloud.mysite.com -connect cloud.mysite.com:443

[evanrich]

@evanrich you are missing the -servername (SNI) parameter in the command

openssl s_client -showcerts -servername cloud.mysite.com -connect cloud.mysite.com:443

thanks I updated my answer with that. -servername works, but my main issue is running gitlab runners. Gilab runners try to talk to https://gitlab.mysite.com... which is resulting in the same issue. AFAIK, there isn't a way to specify something like "-servername" to gitlab runners...

[aledbf]

@evanrich please check your secret. From the test https://www.ssllabs.com/ssltest/analyze.html?d=cloud.evanrichardsonphotography.com&latest you need to add the intermediate certificate to avoid issues.

[evanrich]

@aledbf thanks let me check that... Im not sure thats the issue but here's what gitlab's runner complains about when it tries to register:

status=couldn't execute POST against https://gitlab.evanrichardsonphotography.com/api/v4/runners: Post https://gitlab.evanrichardsonphotography.com/api/v4/runners: x509: cert
ificate signed by unknown authority

SSLlabs seems to show comodo though.. let me try the intermediate cert.

edit: damn you're right. Intermediate was missing, was only the base crt. going to try with the intermediate + root added. thanks!

[evanrich]

@evanrich please check your secret. From the test https://www.ssllabs.com/ssltest/analyze.html?d=cloud.evanrichardsonphotography.com&latest you need to add the intermediate certificate to avoid issues.

thanks ended up NOT having the intermediate in there...just replaced my secret and now ssllabs is at least happy:

will test gitlab -runners now.

edit: Runners work! thanks @aledbf !!

[tunnyblock]

@aledbf thanks let me check that... Im not sure thats the issue but here's what gitlab's runner complains about when it tries to register:

status=couldn't execute POST against https://gitlab.evanrichardsonphotography.com/api/v4/runners: Post https://gitlab.evanrichardsonphotography.com/api/v4/runners: x509: cert
ificate signed by unknown authority

SSLlabs seems to show comodo though.. let me try the intermediate cert.

edit: damn you're right. Intermediate was missing, was only the base crt. going to try with the intermediate + root added. thanks!

@evanrich Would I have a ask about how did you edit the crt file with intermediate . We have met the same problem.

[nickjanssen]

I also had this problem today - turns out my secret key in the Ingress config didn't actually exist. Still, nothing complained when launching the ingress, and doing kubectl describe ing mysite didn't show any errors (definitely room for improvement there).

[GaryCarneiro]

I had this issue
Open SSL gave certificates properly.
openssl s_client -connect myloadbalancer.com -servername actual_domain.com

But curl ran into issues
curl -vH 'Host: actual_domain.com' myloadbalancer.com/my_url

This was resolved by making a DNS entry. ( I made a CNAME of myloadbalancer.com to actual_domain.com )

After this curl also was running fine without Host header too.

curl -v myloadbalancer.com/my_url

[moltar]

Having the same issue, where curl with Host: ... header is not having effect on the request and a "Kubernetes Ingress Controller Fake Certificate" cert is always returned.

Setting the host in /etc/hosts and then curling does work and produces an expected "Fake LE Intermediate X1" cert.

[zhangguanzhang]

I also had this problem......

[praveenperera]

I am having this problem as well

[gdpaulmil]

same, specified my own TLS secret, but its still serving acme

[praveenperera]

I found out that the fake certificate is being used until the real one is received from LetsEncypt. My issue was I had hit the rate limit. Found out by looking at the logs:

kubectl get certificate --namespace=main
kubectl describe certificate myapp-main-com --namespace=main
kubectl describe order myapp-main-com-4054404929 -n main

Also check the certmanager logs

kubectl logs cert-manager-7c8d697cf8-lhgmr -n cert-manager

[gdpaulmil]

guess its related to
https://success.docker.com/article/how-to-configure-a-default-tls-certificate-for-the-kubernetes-nginx-ingress-controller

[anoop-p]

@aledbf thanks let me check that... Im not sure thats the issue but here's what gitlab's runner complains about when it tries to register:

status=couldn't execute POST against https://gitlab.evanrichardsonphotography.com/api/v4/runners: Post https://gitlab.evanrichardsonphotography.com/api/v4/runners: x509: cert
ificate signed by unknown authority

SSLlabs seems to show comodo though.. let me try the intermediate cert.

edit: damn you're right. Intermediate was missing, was only the base crt. going to try with the intermediate + root added. thanks!

@evanrich/ @aledbf Can you explain how did you edit the crt file with intermediate . We have met the same problem.
I have added the intermediate certificate with certificate file and try to add using tls as well as generic secret both didn't work.

Generic

kubectl create secret generic example-cert --from-file=tls.crt=fullchain.crt --from-file=tls.key=privkey.key

tsl

kubectl create secret tls example-cert --key api-privkey.key --cert api-fullychain.crt

[anoop-p]

https://github.com/kubernetes/ingress-nginx/issues/416#issuecomment-493056157

@aledbf thanks let me check that... Im not sure thats the issue but here's what gitlab's runner complains about when it tries to register:

status=couldn't execute POST against https://gitlab.evanrichardsonphotography.com/api/v4/runners: Post https://gitlab.evanrichardsonphotography.com/api/v4/runners: x509: cert
ificate signed by unknown authority

SSLlabs seems to show comodo though.. let me try the intermediate cert.
edit: damn you're right. Intermediate was missing, was only the base crt. going to try with the intermediate + root added. thanks!

@evanrich/ @aledbf Can you explain how did you edit the crt file with intermediate . We have met the same problem.
I have added the intermediate certificate with certificate file and try to add using tls as well as generic secret both didn't work.

Generic

kubectl create secret generic example-cert --from-file=tls.crt=fullchain.crt --from-file=tls.key=privkey.key

tsl

kubectl create secret tls example-cert --key api-privkey.key --cert api-fullychain.crt

Please see https://github.com/kubernetes/ingress-nginx/issues/416#issuecomment-493056157

[SpicySyntax]

I am having this issue as well

[ritheshgm]

I am using the client certificate and added the intermediate certificate as well. when I opened it in the browser it's working as expected, But when I'm trying to get the certificate by running below command it's not working it is showing a fake certificate.

openssl s_client -connect my.site.com:443 -showcerts

[arslanhashmi]

@ritheshgm were you able to solve this issue? I am also facing it.

[jsmedmar]

For people using cer-manager>0.7.2, this comment saved me so much time: https://github.com/jetstack/cert-manager/issues/1650#issuecomment-518953464. Specially on how to configure the ClusterIssuer and Certificate.

[AndiDog]

I am using the client certificate and added the intermediate certificate as well. when I opened it in the browser it's working as expected, But when I'm trying to get the certificate by running below command it's not working it is showing a fake certificate.

openssl s_client -connect my.site.com:443 -showcerts

Use openssl s_client -connect ...:443 -servername ... to enable the SNI extension.

[lkh-smile]

我也遇到同样的问题还是无法解决

[JoshuaSmeda]

I also had this issue.

I fixed it by making sure my certificate common name is the same as the domain I was accessing (i.e testing.internal)

[rmazzine]

I solved by checking the log status of the nginx ingress controller pod:

Get the pod name of ingress-nginx-controller

kubectl get pods -n ingress-nginx

Then get the log searching by any message related to certificates

kubectl logs pod/ingress-nginx-controller-<HASH_OF_YOUR_POD> -n ingress-nginx | grep cert

There I saw, in my case, the Subject Alternative Name was incorect.