Ingress-nginx: External Authentication Nginx Example --> 500 Server Internal Error

Created on 25 Sep 2017  路  10Comments  路  Source: kubernetes/ingress-nginx

Hi, I am petty new to kubernetes and ingress, I am trying to follow the https://github.com/kubernetes/ingress/tree/master/examples/external-auth/nginx example to setup an cluster with an external oauth2.0 provider.

I am running a minikube v0.20.0 locally on my MAC
ingress controller: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13

I setup my cluster as exactly identical as the instruction, except for two changes

  • use a echo server as a the backend, instead of a dashboard
  • moved resources to the default namespace, instead of a mix of kube-system and default namespace

There are two ingress resources setup

$ kubectl describe ing
Name:           external-auth-oauth2
Namespace:      default
Address:        192.168.99.102
Default backend:    default-http-backend:80 (<none>)
Rules:
  Host      Path    Backends
  ----      ----    --------
  foo.bar.com   
            /   http-svc:80 (<none>)
Annotations:
  auth-signin:  https://$host/oauth2/sign_in
  auth-url: https://$host/oauth2/auth
Events:     <none>

Name:           oauth2-proxy
Namespace:      default
Address:        192.168.99.102
Default backend:    default-http-backend:80 (<none>)
TLS:
  tls-secret terminates foo.bar.com
Rules:
  Host      Path    Backends
  ----      ----    --------
  foo.bar.com   
            /oauth2     oauth2-proxy:4180 (<none>)
Annotations:
Events: <none>

The curling "/", is expected to redirect me to /oauth2/auth for authentication, but it actully returns 500 server internal error

$ curl https://192.168.99.102 -H 'host: foo.bar.com' -v -k
* Rebuilt URL to: https://192.168.99.102/
*   Trying 192.168.99.102...
* TCP_NODELAY set
* Connected to 192.168.99.102 (192.168.99.102) port 443 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: Kubernetes Ingress Controller Fake Certificate
> GET / HTTP/1.1
> host: foo.bar.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Server: nginx/1.13.5
< Date: Mon, 25 Sep 2017 17:40:04 GMT
< Content-Type: text/html
< Content-Length: 193
< Connection: close
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
< 
<html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx/1.13.5</center>
</body>
</html>
* Closing connection 0

The /oauth2 works fine:

$ curl https://192.168.99.102/oauth2 -H 'host: foo.bar.com' -v -k
*   Trying 192.168.99.102...
* TCP_NODELAY set
* Connected to 192.168.99.102 (192.168.99.102) port 443 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: Kubernetes Ingress Controller Fake Certificate
> GET /oauth2 HTTP/1.1
> host: foo.bar.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 403 Forbidden
< Server: nginx/1.13.5
< Date: Mon, 25 Sep 2017 17:38:59 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: _oauth2_proxy=; Path=/; Domain=foo.bar.com; Expires=Mon, 25 Sep 2017 16:38:59 GMT; HttpOnly; Secure
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
< 
[body]

I think my configuration is correct, as when I remove auth-url: https://$host/oauth2/auth from the annotation list, the first request will go to the backend echo server directly without redirection.

So I am wondering if there's any thing wrong with the annotation.

I am also confusing about how the $host variable works, appreciate if anyone can provide me a link to its documentation.

picture seanhuxy

Most helpful comment

Problem solved:
for annotation:
the url ingress.kubernetes.io/auth-url should be internally (inside k8s cluster) resolvable, in my case, the in oauth2-proxy service url: 

ingress.kubernetes.io/auth-url: http://oauth2-proxy.default.svc.cluster.local:4180/oauth2/auth

The ingress.kubernetes.io/auth-signin should be externally resolvable, i.e. end-user should be able to get access to it from browser
in my case 

ingress.kubernetes.io/auth-signin: http://foo.bar.com/oauth2/sign_in
picture seanhuxy on 27 Sep 2017
👍11 🎉2 ❤1

All 10 comments

@seanhuxy please check the ingress pod logs

aledbf picture aledbf on 25 Sep 2017

@aledbf Sure, the log seems to say it cannot find foo.bar.com

2017/09/25 18:03:00 [error] 602#602: *188 foo.bar.com could not be resolved (3: Host not found), client: 192.168.99.1, server: foo.bar.com, request: "GET / HTTP/1.1", subrequest: "/_external-auth-Lw", host: "foo.bar.com"
2017/09/25 18:03:00 [error] 602#602: *188 auth request unexpected status: 502 while sending to client, client: 192.168.99.1, server: foo.bar.com, request: "GET / HTTP/1.1", host: "foo.bar.com"
192.168.99.1 - [192.168.99.1] - - [25/Sep/2017:18:03:00 +0000] "GET / HTTP/1.1" 502 0 "-" "curl/7.54.0" 0 0.000 [internal] - - - -
192.168.99.1 - [192.168.99.1] - - [25/Sep/2017:18:03:00 +0000] "GET / HTTP/1.1" 500 193 "-" "curl/7.54.0" 75 0.000 [internal] - - - -
seanhuxy picture seanhuxy on 25 Sep 2017

@seanhuxy well, that's the issue. Please validate foo.bar.com can be resolved from the pod

aledbf picture aledbf on 25 Sep 2017

Closing. Please reopen if

aledbf picture aledbf on 26 Sep 2017

Hi, @aledbf
Yeah, but doesn't it suppose to be solvable from ingress controller? Where should I configure the ingress controller to make the host resolvable.
Thank you.

seanhuxy picture seanhuxy on 27 Sep 2017

@seanhuxy what are you trying to do? You need a valid hostname. If you just use foo.bar.com in the host that will never work

aledbf picture aledbf on 27 Sep 2017

Hi @aledbf
I am sorry to take you so long, but I am assuming ingress controller knows how to forward the request to "foo.bar.com" to the right services.

This is my ingress resource yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/auth-url: http://foo.bar.com/oauth2/auth
  name: external-auth-oauth2
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /
---
apiVersion: extensions/v1beta1
kind: Ingress

metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  name: oauth2-proxy
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 4180
        path: /oauth2
  tls:
  - hosts:
    - foo.bar.com
    secretName: tls-secret

As you can see, if I have ingress.kubernetes.io/auth-url: http://foo.bar.com/oauth2/auth should ingress controller suppose to know it should forward the request to the service oauth2-proxy?

I also tried to write ingress.kubernetes.io/auth-url: http://oauth2-proxy:4180/oauth2/auth
As oauth2-proxy is resolvable within my ingress controller

root@nginx-ingress-controller-3192290068-j1c69:/# nslookup oauth2-proxy
Server:     10.0.0.10
Address:    10.0.0.10#53

Name:   oauth2-proxy.default.svc.cluster.local
Address: 10.0.0.216

I am confusing on what is the right url for ingress.kubernetes.io/auth-url in order to redirect unauthenticated request to foo.bar.com to the oauth2 proxy foo.bar.com/oauth2/auth

Thank you in advance.

seanhuxy picture seanhuxy on 27 Sep 2017

Problem solved:
for annotation:
the url ingress.kubernetes.io/auth-url should be internally (inside k8s cluster) resolvable, in my case, the in oauth2-proxy service url: 

ingress.kubernetes.io/auth-url: http://oauth2-proxy.default.svc.cluster.local:4180/oauth2/auth

The ingress.kubernetes.io/auth-signin should be externally resolvable, i.e. end-user should be able to get access to it from browser
in my case 

ingress.kubernetes.io/auth-signin: http://foo.bar.com/oauth2/sign_in
seanhuxy picture seanhuxy on 27 Sep 2017
👍11 🎉2 ❤1

@seanhuxy A different workaround appears to be to set disable-ipv6-dns: "true" for the nginx ingress controller. With that configuration setting in place, my ingresses could use the following values for auth-url and auth-signin. 

nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
petergardfjall picture petergardfjall on 5 Sep 2018
👎2 👍2 🚀1

@seanhuxy A different workaround appears to be to set disable-ipv6-dns: "true"

@petergardfjall can you explain why is that?

aug70 picture aug70 on 29 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bashofmann picture bashofmann  路  3Comments
smeruelo picture smeruelo  路  3Comments
oilbeater picture oilbeater  路  3Comments
geek876 picture geek876  路  3Comments
cabrinoob picture cabrinoob  路  3Comments