Ingress-nginx: External authentication causes 403

[nixgadget]

Nginx Ingress: 0.17.1
Baremetal K8: 1.11.1

Following this setup https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/README/ I have tried to setup my external authentication with oauth2_proxy as a proxy and google being the provider however I keep getting 403 before even reaching the proxy. Im not sure whats causing the issue at the moment.

The Ingress of the application that I want to protect with oauth2,

  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/auth-url: "https://dev-oauth2.test.domain.com/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://dev-oauth2.test.domain.com/start"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"
    nginx.ingress.kubernetes.io/secure-backends: "false"
spec:
  tls:
  - hosts:
    - dev-app.test.domain.com
    secretName: app-tls
{{- end }}
  rules:
    - host: dev-app.test.domain.com
      http:
        paths:
          - path: /
            backend:
              serviceName: app
              servicePort: 20001

And this is the Ingress of the oauth2_proxy,

  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"
    nginx.ingress.kubernetes.io/secure-backends: "false"
spec:
  tls:
  - hosts:
    - dev-oauth2.test.domain.com
    secretName: oauth2-proxy-tls
{{- end }}
  rules:
    - host: dev-oauth2.test.domain.com
      http:
        paths:
          - path: /oauth2
            backend:
              serviceName: oauth2-proxy
              servicePort: 4180

In the logs all I see is,

<public ip> - [<public ip>] - - [23/Jul/2018:07:50:51 +0000] "GET / HTTP/1.1" 403 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0" 0 0.039 [control-app-20001] 104.31.93.33:443 0 0.020 403 82b9d691c3f52e0851d736a75e48fd26
<public ip> - [<public ip>] - - [23/Jul/2018:07:50:52 +0000] "GET /favicon.ico HTTP/1.1" 403 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0" 0 0.013 [control-app-20001] 104.31.92.33:443 0 0.010 403 3d64fe37fe548e6d45d6795920a90a89

This has deviated from the example because I want to have a single point for oauth2 proxying so that I can protect multiple applications. i.e. with a single google client secret. I can authenticate fine if I navigate to https://dev-oauth2.test.domain.com/oauth2/start.

In saying all this if I try to curl as below then it seems be working as expected with a 302.

curl -kiv https://<public ip> -H "Host:dev-app.test.domain.com"                                                                                                                                                                       
* Rebuilt URL to: https://<public ip>/
*   Trying <public ip>...
* TCP_NODELAY set
* Connected to <public ip> (<public ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Jul 23 03:38:22 2018 GMT
*  expire date: Jul 23 03:38:22 2019 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f9a27805800)
> GET / HTTP/2
> Host: dev-app.test.domain.com
> User-Agent: curl/7.54.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 302
HTTP/2 302
< server: nginx/1.13.12
server: nginx/1.13.12
< date: Mon, 23 Jul 2018 07:55:33 GMT
date: Mon, 23 Jul 2018 07:55:33 GMT
< content-type: text/html
content-type: text/html
< content-length: 162
content-length: 162
< location: https://dev-oauth2.test.domain.com/oauth2/start?rd=https://dev-app.test.domain.com/
location: https://dev-oauth2.test.domain.com/oauth2/start?rd=https://dev-app.test.domain.com/
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains
< set-cookie: __cfduid=d757a178ed3a954654e9f339722fa46bb1532332533; expires=Tue, 23-Jul-19 07:55:33 GMT; path=/; domain=.steadypay.co; HttpOnly
set-cookie: __cfduid=d757a178ed3a954654e9f339722fa46bb1532332533; expires=Tue, 23-Jul-19 07:55:33 GMT; path=/; domain=.steadypay.co; HttpOnly

<
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx/1.13.12</center>
</body>
</html>
* Connection #0 to host <public ip> left intact

Nginx log shows,

<public ip> - [<public ip>] - - [23/Jul/2018:07:55:33 +0000] "GET /oauth2/auth HTTP/1.1" 401 21 "-" "curl/7.54.0" 458 0.051 [control-op-oauth2-proxy-4180] 192.168.0.82:4180 21 0.060 401 45d771ba7d68cc52cc9708f9af40be46
127.0.0.1 - [127.0.0.1] - - [23/Jul/2018:07:55:33 +0000] "GET / HTTP/2.0" 401 0 "-" "curl/7.54.0" 0 0.098 [control-app-20001] 104.31.93.33:443 0 0.100 401 22d2f12d6d8aee78a680869dcd2cb9b8

Any help would be appreciated to figure this out.

[nixgadget]

If I set the auth-url as nginx.ingress.kubernetes.io/auth-url: "http://192.168.0.82:4180/oauth2/auth" instead of the public address then i can get a 302 successfully on the browser.

This tells me that something isnt quite right with the SSL validation/termination.
Any advice is welcome.

[Stono]

That's fine. The call to /auth is internal to your cluster so should be http://service.namespace.svc.cluster.local:4180

[nixgadget]

Thanks @Stono.
As my cluster is baremetal and I use CNI and Nginx hostNetwork I couldn't resolve the internal service FQDN at first but adding dnsPolicy: ClusterFirstWithHostNet fixed it. So I no longer have to use the internal ip address. And yes you are right. You have to use the internal service FQDN for nginx.ingress.kubernetes.io/auth-url.

Following this, I encountered another issue with oauth2_proxy which doesnt allow a proper redirection due to the issues explained in oauth2_proxy/pull/464 and oauth2_proxy/pull/461.

I will post back my working solution here for anyone with a similar setup.

[nixgadget]

As promised heres my working example.

• First and foremost I created a patched version of oauth2_proxy with the patches oauth2_proxy/pull/464 and oauth2_proxy/pull/621 as given in steadypay/oauth2_proxy/
• Then I deployed the patched version with the following options,

  email-domain: "*"
  upstream: "file:///dev/null"
  http-address: "0.0.0.0:4180"
  cookie-refresh: "1h"
  cookie-expire: "168h0m"
  cookie-secure: "true"
  cookie-domain: ".test.domain.com"
  ssl-insecure-skip-verify: "true"
  whitelist-domain: ".test.domain.com"
  pass-basic-auth: "false"
  pass-access-token: "true"
  pass-user-headers: "true"
  set-authorization-header: "true"
  pass-authorization-header: "true"

• I then created the app ingress with the following annotations as discussed above,

nginx.ingress.kubernetes.io/auth-url: "http://op-oauth2-proxy.control.svc.cluster.local:4180/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://dev-oauth2.test.domain.com/oauth2/start"

Also if you want to enforce SSL,

kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

This has worked so far with Kubernetes Dashboard and few of my other applications
For Kubernetes Dashboard I had to additional add as it is already SSL enforced,

nginx.ingress.kubernetes.io/secure-backends: "true"