Minikube: ingress addon: "default backend - 404" when HTTPS is used.

Created on 15 Jul 2017 · 27Comments · Source: kubernetes/minikube

BUG REPORT

Minikube version: v0.20.0

Environment:

OS: Windows 10 Pro (Anniversary Edition)
VM Driver: hyperv
ISO version: minikube-v0.20.0.iso

What happened:

$ kubectl run hello-world --image=tutum/hello-world:latest --port=80
deployment "hello-world" created

$ kubectl expose deployment hello-world --type=NodePort
service "hello-world" exposed

$ curl $(minikube service hello-world --url)
<html>
<head>
        <title>Hello world!</title>
...

$ minikube addons enable ingress
ingress was successfully enabled

$ curl $(minikube ip)
default backend - 404

So far so good. Created a file minikube-ingress.yaml as so:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minikube-ingress
  annotations:
spec:
  rules:
  - host: hello.world
    http:
      paths:
      - path: /*
        backend:
          serviceName: hello-world
          servicePort: 80

Added my minikube ip to my hosts file: 192.168.0.25 hello.world

$ kubectl apply -f minikube-ingress.yaml
ingress "minikube-ingress" created

$ curl http://hello.world
<html>
<head>
        <title>Hello world!</title>
...

$ curl -k https://hello.world
default backend - 404

Ok, so I should be able to hit the hello-world service with https. Supposedly the default configuration of the ingress is SSL Termination, with some auto-generated self-signed certs.
I did try hitting this in my browser as well, where I was asked to accept the cert first.
But no matter what, it still goes to the 'default backend' instead of my hello-world service.

Also tried the following configuration:

apiVersion: v1
kind: Secret
metadata:
  name: minikube-ingress-secret
  namespace: default
type: Opaque
data:
  tls.crt: LS0t...
  tls.key: LS0t...

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: minikube-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - secretName: minikube-ingress-secret
  backend:
    serviceName: hello-world
    servicePort: 80

The above ingress definition completely ignores whatever I put in "backend" and just sends everything to the "default backend 404".
I also tried adding the rule from the first config into this, and several other permutations, all with the same results.
I also tried deleting all the resources and creating them again, with no luck.

I've tried deleting all my dns and ingress pods, to see if when they came back it would work. That did not help.

What you expected to happen:
I would expect that I could curl https://hello.world and get back my hello world html results with 200, instead of going to default-backend-404.

I would also expect that I could overwrite what the default backend is by specifying the backend block in the above spec, but it is getting compltely ignored.

And I would also expect that the ssl-redirect would work, yet it seems to also be ignored (http continues to work just fine).

addoingress chyperv good first issue help wanted kinbug lifecyclrotten prioritbacklog 2019q2

Source

veqryn

👍10

Most helpful comment

it seems adding the following made it work for me:

spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret

paulczar on 8 Nov 2017

👍4 🎉1

All 27 comments

I'm seeing the same thing. Interestingly, if I kubectl exec -it <ingress-controller-pod> bash and then curl from there, it works. So it seems something is mangling the request before it arrives at the ingress controller?

carlpett on 6 Nov 2017

I can confirm I'm also seeing this. not only does it 404, but it appears to ignore my TLS certs from secret.

paulczar on 8 Nov 2017

it seems adding the following made it work for me:

spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret

paulczar on 8 Nov 2017

👍4 🎉1

I'm wondering if minikube addons enable ingress creates some sort of hidden ingress somewhere? Because if I do kubectl port-forward to the ingress and point to localhost, it also works as intended...

carlpett on 8 Nov 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 7 Feb 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot on 9 Mar 2018

/remove-lifecycle rotten

elgalu on 15 Mar 2018

Does minikube ingress support https? No matter what I do, I cannot get it to work (I can get http to work). Is there a tutorial anywhere for minikube ingress that shows how to get https to work?

snorthov on 22 Apr 2018

👍3

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 21 Jul 2018

I am facing the same issue. my ingress is https://gist.github.com/darkedges/80def8628fa3faa5bb13f0c5d00ed36c

I know it is working as when I hit the http:// address it redirects tohttps://, but I get the backend not found response.

If I kubectl exec -it -n kube-system nginx-ingress-controller-67956bf89d-5c9zx bash I can see my certificates are there

root@nginx-ingress-controller-67956bf89d-5c9zx:/ingress-controller/ssl# ls -lrt
total 20
-rw------- 1 root root 2933 Jul 21 19:48 default-fake-certificate.pem
-rw-r--r-- 1 root root 1659 Jul 21 19:48 default-darkedges.com-full-chain.pem
-rw------- 1 root root 3364 Jul 21 21:18 default-darkedges.com.pem
-rw-r--r-- 1 root root 1659 Jul 21 21:19 default-darkedges-com-tls-full-chain.pem
-rw------- 1 root root 3364 Jul 21 21:58 default-darkedges-com-tls.pem

but in /etc/nginx.conf I can see all my host entries on port 80, but nothing on 443.

Any ideas on what I am doing wrong?

Edit:

Found this via kubectl logs -n kube-system nginx-ingress-controller-67956bf89d-5c9zx
W0721 22:34:37.322152 6 controller.go:1027] Validating certificate against DNS names. This will be deprecated in a future version. W0721 22:34:37.322162 6 controller.go:1032] ssl certificate default/darkedges-com-tls does not contain a Common Name or Subject Alternative Name for host as.tpp.forgerockdev.darkedges.com. Reason: x509: certificate is valid for *.darkedges.com, darkedges.com, not as.tpp.forgerockdev.darkedges.com W0721 22:34:37.322388 6 controller.go:1026] unexpected error validating SSL certificate default/darkedges-com-tls for host as.bank.forgerockdev.darkedges.com. Reason: x509: certificate is valid for *.darkedges.com, darkedges.com, not as.bank.forgerockdev.darkedges.com

So mine is an issue with my certificates not matching.

darkedges on 22 Jul 2018

I am having same issue

jainishshah17 on 17 Aug 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 16 Sep 2018

Still happening. Please don't close. Can someone provide an example where this is working?

snorthov on 17 Sep 2018

A working example would be great indeed. Dropping a line to follow this...

ametad on 10 Oct 2018

This is working (well... almost):

it seems adding the following made it work for me:
spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret

Little change:

spec:
  tls:
  - hosts:
    - hello.world
  - secretName: minikube-ingress-secret

(Notice the last line)

ametad on 10 Oct 2018

👍1

Sweet, the - secretName fixed it for me too

sccoe on 28 Dec 2018

The workarounds seem reasonable. Help wanted to get this into the addon itself!

tstromberg on 24 Jan 2019

Hilarious! After moving on to other things and months passing, I decided to try out the work around today. I was just about to report back here that is was working and tstromberg commented.

There is a Software God (or Gods) and a Single Global Cache!

snorthov on 24 Jan 2019

Sweet, the - secretName fixed it for me too
Mee too! After an evening of struggling with 404. Thanks, ametad!

wavilov on 6 Mar 2019

This is working (well... almost):
it seems adding the following made it work for me:
spec:
  tls:
  - hosts:
    - hello.world
  secretName: minikube-ingress-secret
Little change:
spec:
  tls:
  - hosts:
    - hello.world
  - secretName: minikube-ingress-secret
(Notice the last line)

New info! Ingress config change will force ingress to use fake certificate. Site will work but with big browser ssl alarm.

In my case true reason has been in certificate, that was stored in another kubernetes' namespace from ingress. After I put certificate in same namespace, HTTPS worked with default ingress config.

Idea taken from this commentary
kubernetes/ingress-nginx#1984 (comment)

wavilov on 6 Mar 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 20 Aug 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 19 Sep 2019

/assign rajalokan

rajalokan on 24 Sep 2019

If you are using SSL on your host OS with an ingress for a host which only your host OS knows about then you need to generate a certificate for your host if you don't want to get the browser warning. You can do this with omgwtfssl

Then you need to add the the certificate bundle by appending the domain certificate to the CA certificate. Something like this:
cert.pem

bundle.pem

You can manually create a secret or there is an option with omgwtfssl to create a kubernetes secret when generating the certificate. It should look like this:

apiVersion: v1
kind: Secret
metadata:
  name: traefik-local-default-cert
  labels:
type: kubernetes.io/tls
data:
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBejUybUFKemF0dHRqN1dnRjZKK1o4U0R5Qy9iSnpYaXZQNjRwbE5HSDBUZjMvY2hQCk5ydnN1aGRuNEVYRkpNU2w4M29MOFpocnl3N25UbXR1RlhTSjVQL2RldVV2d2Q5KzJTdkZuUzY2OFFmZTVjTTcKOFVLbWZiY1FWUVluZTZCcjhZS1hacTNmZWZ2Y3dZZ2x6eDVncXNIeFNPNzk3T01hQVBLb0ViYmZZZGhrYVl1NQo1bTlxWlB6dzdjS1IzUjg5bVF0TElmSzJwV0cvZjZySkJENDZ6bUJJQm8yT29pNGVVQXloOXRwMFdnWjlRRUt5CnhQZU9lL2c2Rk5CNlA1NStmUVZiVVVySk1FZHBGY0pYVjIwSzZJNkVtWStrUnpoc2VzKzh4VUpWQkFPSVhPOFEKcDdEUHZSaThvcGNaVi9tV0wxVkovMFJ5aDlDazlqTDVZMVBhdlFJREFRQUJBb0lCQVFDbXN2RVArdUdPQXRmWQpUamE0VWpYYnExVk5qK2oyeHRrSHE0S05rRE9hSkhORlhPbGxqRHlHS3Jib0YzMDBpWVNwMnl2dDN6Rmx5R0NtCjBaR0JQcUpkc2FlV3ZxQzY3UVcvditxYXkyT2tsbW85VjZZd2tCQjRUbDJadHloS2xrTTFsbUtVT2pWK2oyRHcKTDY0VUFGTWpxRGRoRlN2allYR0dvMCtaY1prb3lKRnZ2MGo3QU9yWXBUd2RiWmMySG0wTEpFNmxOTE9MVUVQKwprc0l1aDNHSWdENXdiaXBidldMUTlicDd0RURNZ01ic2ZaOFJHOUs0cnNHSUszcGpuKzdMek8yeWQ0cXhoV0wwCk5lb1M4VGs1eGRiN3VpS2RjTHF3NTUvWGxVSVZDcUVCVThnWHoranorT0ltMXgwQ3MxU1pNMUZJQ25xbUNCVzkKNjd2b3kwRUJBb0dCQU8xSEU1bzlhTmlGOEkwaHN1elh0MGVoMEFPSVMzWTN3S0lqc1pMMTJ6V0Q5ZDhYZjR1ZgpDN3BYeTNVOTYvSGtQTlRmeGxZZUhVVzByTy92N09nZWNITmNJZ2tBaTd1TCsxV20wRzEwb0kyaVlySmlPc25lCkdvOTN3M3RZTGZwTXBIbU9JalN6Qi9RMnNnTnpuSzJLeENoTDgwWGJMWjE4YVRlWEpzVDM1Q2I5QW9HQkFOLy8KYWQ3Z2Rtdld5WXFWRlZ1Y0tlNTV6UTRQUXpIa1FNYmlsRElwcmwrKzJHd0QwZ1NVU0Q0QWRFeHduL0tQWTZsUQpoV20xNzBTTVlic1F3Zk43Y0MrSGowK0pIRnVnaUFLWHdNZmo0ZG5FWDQwNUh3Wi94V3dZK1FrSHZ4UHFNVG1EClJ3Q05VcFZJOTVVVTlFTkVlMUZzN2trbUlQZXJ4UUxVQjFOMTVTN0JBb0dBYTBzL3B2aHI1N1V4WUhCRG0ydTAKT2hVR25nSk10VWxpdHk3U3huU2NTbWZBajcvdGNmSmFlRW5vKzd3amJEOWI3cHE5OGdVdUVXNHc0VktwMnhDZwpvdG1mbmxoNzF6UG1WQVhackJVWDROUDBNMXZOMzZpcWRBT2hCcTZLSSt2eVYvRVlzS0hnVCtOSHZkM1NsSHJ5ClZwSjVrRFliTzJLZHBZQ1pER3BYNWZVQ2dZQVROeC9uNUsyRDlycjNUbTYwbXlMbHVFRWs4WjQ4QzI2YklSakIKYk4zdWpMcVlHWVBNck1POXZlQThKblV5STZiVEFYdFFaREVRV2xLUDc1SVR4d1VLcnNCV2ZGYVliWjc5U2ZOWgpzbXpQZDQ4UzJGR1hCeUx5Y1BuOEVsUUw2MDBwdDk5Qlp0NHN2RU93NEVsTDgvWDRPRFlBeEdxVGxJS2tDWGhQCmFvK3NnUUtCZ1FEc3dDT2xTN3pSWGdYUkdHRGNpWGRMbEFQQ2dIQklaNUFWVXh6ekhDMGI3WUIxQ0FHbnAxNXkKbk9CcmhMdzhGUnloSHROVEJrSGM5SjQ0REVKL3Y1N1VrbWRocU9PZ0NqbUM2dXZTTXRSdnVEWEhLV0t4Q2JsTwpORmtkVTl0K0x4MFlQeDF3YUtnWW1EZFVsL2ErN2tjNHhUZjFIV1F0LzZPQVJGTHliKzZnUFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGVENDQWYyZ0F3SUJBZ0lKQU5Dc2gvdCtwT1ZKTUEwR0NTcUdTSWIzRFFFQkN3VUFNQmd4RmpBVUJnTlYKQkFNTURYUnlZV1ZtYVdzdWJHOWpZV3d3SGhjTk1Ua3dOREl4TWpFd01UQTRXaGNOTXpnd05qSXdNakV3TVRBNApXakFZTVJZd0ZBWURWUVFEREExMGNtRmxabWxyTG14dlkyRnNNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DCkFROEFNSUlCQ2dLQ0FRRUF6NTJtQUp6YXR0dGo3V2dGNkorWjhTRHlDL2JKelhpdlA2NHBsTkdIMFRmMy9jaFAKTnJ2c3VoZG40RVhGSk1TbDgzb0w4WmhyeXc3blRtdHVGWFNKNVAvZGV1VXZ3ZDkrMlN2Rm5TNjY4UWZlNWNNNwo4VUttZmJjUVZRWW5lNkJyOFlLWFpxM2ZlZnZjd1lnbHp4NWdxc0h4U083OTdPTWFBUEtvRWJiZllkaGthWXU1CjVtOXFaUHp3N2NLUjNSODltUXRMSWZLMnBXRy9mNnJKQkQ0NnptQklCbzJPb2k0ZVVBeWg5dHAwV2daOVFFS3kKeFBlT2UvZzZGTkI2UDU1K2ZRVmJVVXJKTUVkcEZjSlhWMjBLNkk2RW1ZK2tSemhzZXMrOHhVSlZCQU9JWE84UQpwN0RQdlJpOG9wY1pWL21XTDFWSi8wUnloOUNrOWpMNVkxUGF2UUlEQVFBQm8ySXdZREFKQmdOVkhSTUVBakFBCk1Bc0dBMVVkRHdRRUF3SUY0REFkQmdOVkhTVUVGakFVQmdnckJnRUZCUWNEQWdZSUt3WUJCUVVIQXdFd0p3WUQKVlIwUkJDQXdIb0lOZEhKaFpXWnBheTVzYjJOaGJJSU5kSEpoWldacGF5NXNiMk5oYkRBTkJna3Foa2lHOXcwQgpBUXNGQUFPQ0FRRUFHNjA4ei9SaE1DeWxJVHdjTVpaUkxHSDRxRndOSHF0UzRyMkMrVzVSaDVTd3c1R0Q2aWZ6ClZJUC96Q0hodmNLWTRveERmQTZnNzBzSVJQT2FKcURaR2FhbGhoWjMxeC9mQmkvd1FMUUROdVQ4elNtckk4SUgKSWtTWUpYcHhHTE9DaEt1NG9nSHNseVdVb2hXZVhuSytMVWNkVm45Z2VxbXU4SzhZLzVhZFJvZERmNVlMZUdRegp6Nmxmc1BEU3pZcHZqQmVlQ1ltUU01dVpLS2diYlNkZHZlcDUrZUpWY2NTZCt4c1liZFJXZDU5Tmc4V3dxcVF0CjgxdzVneVJDSzFERXg3ODhFTzM1aGJRSkFvN2JiKzlmSWNJODFsNUlRMjdobzNPVU1aWlRZQ2w1U1Z1ZkdBckEKdHV0K2lOTm9Ca05HZjRrTndob1lOS0p6M2ZUU2xBVGMwdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

The tls.key is the base64 encoded private key
The tls.cert is the base64 encoded bundled certificate

In your ingress specify the certificate name:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-traefik-dashboard
  annotations:
    #optional
    ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
    - host: traefik.local
      http:
        paths:
          - backend:
              serviceName: my-traefik-dashboard
              servicePort: 443
  tls:
    - secretName: traefik-local-default-cert

On mac OS you can trust the bundled cert like so

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain bundle.pem

On Linux it is like this

sudo cp bundle.pem /usr/local/share/ca-certificates/traefik.local.bundle.crt
sudo cp cert.pem /usr/local/share/ca-certificates/traefik.local.cert.pem
sudo update-ca-certificates --fresh
certutil -d sql:$HOME/.pki/nssdb -A -t "P,," -n /usr/local/share/ca-certificates/traefik.local.cert.pem -i /usr/local/share/ca-certificates/traefik.local.cert.pem

Not sure how to do this on windows.

I think there needs to be a PR that explains all this in the documentation since minikube is run locally and people will not expect to get SSL error warnings and this is really the only right way to resolve both the matching and the certificate error warnings in web browsers.

woodcockjosh on 1 Oct 2019

It would be really nice if minikube had a service that ran on the local machine and installed SSL certificates for https ingress resources. But alas no such capability exists. Maybe an addon could do this?

woodcockjosh on 1 Oct 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 31 Oct 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.