Kubeapps: Azure login not working since change to oauth2_proxy

[kristinnardal2]

Description:

It seems to us that OIDC may not have been working correctly together with Azure since the change to oauth2_proxy.
This change was made in chart version 3.0.0 of bitnami/kubeapps, which also changed the Kubeapps version from 1.5.1 to 1.6.0.

We've been trying to get OIDC working with Kubeapps and Azure Active Directory and we just couldn't get it to work. We were able to do this last year so it was weird that it didn't work this time.

In our investigation we noticed that there had been a change of auth proxy, so we decided to try using an old version to see if we could get that one to work. We used chart version 2.1.7 of bitnami/kubeapps and the exact same clientId, clientSecret and discoveryURL as for the new chart version. Our values.yaml files look something like this:

• When we used chart version 2.1.7 of bitnami/kubeapps:
  yaml authProxy: enabled: true clientID: <Azure AD server ID> clientSecret: <Azure AD server secret> discoveryURL: https://sts.windows.net/<Azure AD tenant ID>/
• When we used chart version 3.4.3 of bitnami/kubeapps:
  yaml authProxy: clientID: <Azure AD server ID> clientSecret: <Azure AD server secret> cookieSecret: <Random cookie secret> provider: azure enabled: true additionalFlags: - -azure-tenant=<Azure AD tenant ID> - -cookie-secure=true

We also had to make a slight change in the Azure portal: It seems that the redirect URL was changed between the two versions from /oauth/callback to /oauth2/callback.

Starting Kubeapps using the 2.1.7 chart version works and allows us to sign in with Azure AD, while Kubeapps using the 3.4.3 chart version does not.

Our setup

We are using Azure and have a Kubernetes cluster with Kubernetes 1.15.10 installed. We use Helm 3 to install the chart in our cluster. We have a reverse proxy routing traffic to the Kubeapps instance; this is an Nginx reverse proxy listening to Kubernetes ingress resources.

Problem description

After installing Kubeapps using the values described above with chart version 3.4.3, we go to the Kubeapps web page.
We're asked to log in; when clicking the login button, we're redirected to a Microsoft Azure login page, and upon completion of the process we're sent back to the same main page we started on (the Kubeapps login page).

Closer inspection of the log from the container named auth-proxy (running oauth2_proxy) and living in a pod controlled by the deployment kubeapps reveals that oauth2_proxy successfully completes the authentication process:

...[AuthSuccess] Authenticated via OAuth2: Session ...

Further information on the user is shown and seems to be correctly retreived from Azure.

However, the request from the browser to /api/kube returns a 401 - Unauthorized with this response:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

This prompts Kubeapps to take us back to the login page again.

The request to /api/kube is however completed successfully when using chart version 2.1.7 (with keycloak).

Version of Helm, Kubeapps and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

• Output of helm list <kubeapps-release-name>:

NAME            NAMESPACE       REVISION        UPDATED                                         STATUS          CHART           APP VERSION
kubeapps        kubeapps        2               2020-04-03 11:35:31.301887345 +0200 CEST        deployed        kubeapps-3.4.3  v1.8.2 

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-12T13:43:46Z", GoVersion:"go1.13.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"150f36044fe31bee3891b5a3fae69c17237e022c", GitTreeState:"clean", BuildDate:"2020-02-21T10:02:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[absoludity]

Thanks for all the details. I've not had a chance to set this up to try it, but please check if this helps to find out more. So from your info the authz is working but then the AKS API server is rejecting the credentials which are being included in the request?

TLDR: If you haven't already, check for the known issue where Azure's session store (on oauth2 proxy) is too large for nginx (at the end), then (more likely) try using the the oidc provider, rather than the specific azure provider for oauth2-proxy. Read on for more details.

A number of things I can think of but I've not yet tried them. First, afaict, AKS can be accessed in two ways:

• via OIDC using a specific AD server created in your account and the cluster configured to use that. Based on the slack history where you mention that you setup using this doc (Integrate Azure Active Directory with AKS), I assume this is what you are wanting to use,
• via Azure's older OAuth2 (non-v2).

Similarly, oauth2-proxy enables both generic oidc support (where you can define the issuer URL), as well as custom OAuth2 Azure support, which seems to use the older issuer URL (not v2 urls).

I'm not 100% (as I've not tested it), but from the docs you referenced, it looks like you may be intending the newer v2 oidc support (and have setup your cluster for that), but are logging in using oauth2-proxy's azure provider which uses the older (nonv2) oauth2, so although it still returns both an access_token and an id_token, I'm not sure that it's the id_token expected by the cluster (ie. signed correctly by the specific provider you created in your azure account). If you want to check before going down this road, the k8s api server logs will should show you the authz issue with the request it receives - if it says "invalid signature" or similar, we can be more confident before trying).

To try it, just use the oidc provider instead of azure. It should just work as long as you set the --oidc-issuer-url to something like https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration as per the oidc client docs linked from your above Integrate AAD with AKS doc (the sts.windows.net/{tenant} one which you apparently used previously is the non v2 one, it seems).

Other details

When updating the auth proxy from keycloak, one of the things I noticed was that the dashboard client was at the time actually receiving and storing the credential (the OIDC token base64-encoded in local storage) from the auth provider, and then sending that with subsequent requests in the Authorization header (you should be able to see with with 2.1.7). It isn't great security-wise for the frontend client to be seeing and storing the credentials, and so not something I wanted to replicate. When updating to auth-proxy (for other reasons), I used it's (sane) default which ensures the dashboard client never even sees the actual credential (unless you change the defaults), rather it sets an encrypted cookie on the dashboard client which is then sent with every request, which auth-proxy decrypts to extract and update the request with the relevant credentials.

authproxy encrypts and includes both the OIDC (JWT) id_token and the OAuth2 access_token, sending the id_token as the Authorization bearer token. This is fine for vanilla k8s clusters which trust a signed OIDC JWT id_token, but I found that Google's k8s service was expecting the access_token in the Authorization header explicitly, which is why I added the proxypassAccessTokenAsBearer, but I don't expect this to be relevant for AKS.

If none of the above helps, I'll try to put some time aside to do the same setup and debug (and update the docs with the results to make it clear for others). Thanks.

[kristinnardal2]

Thank you for the detailed response.

We have gotten it to work using oidc provider now. We did have to increase the proxy_buffer_size for our nginx reverse proxy as you suggested which seemed like the main blocker that we were facing. Without this increase trying to login to kubeapps resulted in a 502: Bad Gateway response code. It's probably worth pointing this out in the documentation as I suspect many people will have kubeapps behind an nginx reverse proxy.

There are many ways of increasing proxy_buffer_size but the way we did it was with an annotation on the ingress for kubeapps. For us it was enough to increase the proxy_buffer_size to 8k Our configuration is now working as intended and looks like this:

useHelm3: true

authProxy:
  enabled: true
  clientID: <Azure AD server ID>
  clientSecret: <Azure AD server secret>
  cookieSecret: <Random cookie secret>
  provider: oidc
  additionalFlags:
    - -oidc-issuer-url=https://sts.windows.net/<Azure AD tenant ID>/

ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "8k"
    # Additional annotations
  # Additional ingress settings

frontend:
  service:
    type: LoadBalancer

As I said earlier I would suggest we add something about this to the OIDC documentation so that people will be able to avoid this hassle in the future. I have created a pull request that adds a note to the documentation.

[absoludity]

We have gotten it to work using oidc provider now. We did have to increase the proxy_buffer_size for our nginx reverse proxy as you suggested which seemed like the main blocker that we were facing.

Thanks for the info. I'm particularly keen to understand whether you needed both changes, or whether the proxy_buffer_size enabled you to login without changing the provider?

It also surprised me that you needed to update the nginx ingress proxy buffer size, but not the kubeapps frontend nginx proxy buffer size - was the latter not necessary?

Glad it helped!

[kristinnardal2]

We did have to change the provider to oidc.
To make sure I tested changing the values back to use provider azure and set the flag azure-tenant instead of oidc-issuer-url and it did not work.

When I look at what the browser sees it is getting a 401 response from the call to /api/kube which is the same behaviour we had before increasing proxy_buffer_size.

I did not have to change the proxy_buffer_size for the kubeapps frontend nginx. The only change I made was the new annotation to the ingress part of the configuration.