Kubeapps: [FEATURE REQUEST] Option to manually select kube-api version

[lindhe]

Background:

We have an AKS cluster with Kubernetes version 1.17, and when running Kubeapps in that cluster we see that it sends API calls with version 1.17, which makes sense (I assume Kubeapps tries to match the server API version, please fill me in on this).

Unfortunately, as we have explained to Microsoft (here, here and in an email to their support), Microsoft broke the allowed version skew policy with the introduction of their AKS-Managed AAD feature. With that feature enabled, even if the server side is running 1.17, RBAC is broken for kubectl <1.18 (and I guess the entire client side of the API). So in this case, client side API calls with version 1.17 won't work to API server 1.18 (at least not with RBAC).

AKS-Managed AAD is an optional feature that is not enabled by default, but is available in AKS (and non-reversible once enabled) from 1.17 and onward.

Requested feature:

I think this is clearly not Kubeapps' fault. It should be possible to infer the required API version from the server version, but it's not. Microsoft is to be blamed here. However, I think there's a work-around that might not be too much work to implement. Please pitch in and say if it seems feasible, @absoludity .

If I was able to set the desired API version in Values, I could forcibly deploy a version that uses API 1.18 even if the cluster is 1.17.

I suspect that this feature could be useful in many different cases, for debugging etc. And also, but now I'm really just guessing, perhaps it could make sense when deploying Kubeapps in a multi-cluster context where the clusters are at different API versions...

Steps to reproduce the issue:

1. Create an AKS cluster with AKS-Managed AAD disabled.
2. Install Kubeapps and see that it works with RBAC as expected.
3. Enable AKS-Managed AAD.
4. See you cannot login to Kubeapps anymore (see response here)

Describe the results you received:

I get a 401 because of some new way that AKS uses tokens.

Describe the results you expected:

Being able to authenticate towards the kube-api server.

[absoludity]

We have an AKS cluster with Kubernetes version 1.17, and when running Kubeapps in that cluster we see that it sends API calls with version 1.17, which makes sense (I assume Kubeapps tries to match the server API version, please fill me in on this).

We're just using version 0.18.0 of client-go, which is compatible with 1.17 as per the compatibility-matrix, but I'm surprised that it reports itself as the server version, if that's the case.

We'd need to look more closely at client-go to verify what's happening and whether the client version is configurable.

[andresmgot]

So what API version are we talking about? What's the version change that causes the breakage? As far as I can tell, RBAC version is still at rbac.authorization.k8s.io/v1 in Kubernetes 1.18 so I am not sure what's the breaking change?

If I was able to set the desired API version in Values, I could forcibly deploy a version that uses API 1.18 even if the cluster is 1.17.

I don't think that's possible, or at least is not really something that can be maintained, in every call to the k8s API, we would need to choose between one endpoint or the other (unless I am missing something).

[absoludity]

I assumed he meant that the k8s api server is responding differently not because different versions of group apis are being used, but because a different client header is sent with the request.

[lindhe]

Hi. Sorry for dropping out on this, have been busy with other things lately. When inspecting the logs from the nginx container in the kubeapps pods, I saw a bunch of lines saying 10.240.0.6 - - [05/Oct/2020:11:16:33 +0000] "GET / HTTP/1.1" 200 1155 "-" "kube-probe/1.17" so I assumed that meant the API calls were Kubernetes version 1.17. But I might very well have misunderstood the situation.

We'd need to look more closely at client-go to verify what's happening and whether the client version is configurable.

Yeah, that sounds reasonable in that case.

So what API version are we talking about?

I assumed he meant that the k8s api server is responding differently not because different versions of group apis are being used, but because a different client header is sent with the request.

That's more or less my understanding, that it's client version 1.17. But I don't even think the server responds differently depending on client version. I think it's worse than that. I think the server responds _the same way_ no matter what the client version is. And it happens to be incompatible with client version 1.17. Again, that breaks the allowed version skew described in the compatibility matrix. It shouldn't break, but it does. And that's Microsoft's fault.

BTW: We will upgrade our cluster to 1.18 and see if that resolves the issue. Will get back to you when I have more info on that.

[lindhe]

We just tried it on a 1.18 AKS cluster with Managed AAD enabled, and it still didn't work. I double-checked, and now the log did indeed say "kube-probe/1.18" instead of "kube-probe/1.17".

This means that my feature request would not resolve our issue. Closing this ticket for the time being. I think we will have to get in touch with Microsoft Support again, because there has to be a feature that's either missing or is very different from how we used to configure things before Managed AAD. I'll try to remember posting an update here to document how to actually configure Kubeapps for AKS with Managed ADD, if we figure it out.