Zero-to-jupyterhub-k8s: Azure AKS now enabling RBAC by default

[tjcrone]

It looks like Azure AKS is now creating clusters with RBAC enabled by default, because ostensibly they now support it. This is a good thing, because RBAC is a good thing, but now the z2jh guide does not work on Azure as written because when rbac:enabled=false, the jupyterhub install fails with the brutal "timed out waiting for the condition" error. When rbac:enabled=true, it throws:

Error: clusterroles.rbac.authorization.k8s.io "hook-image-awaiter-jupyter-1-1521992989" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["daemonsets"], APIGroups:["apps"], Verbs:["get"]}] user=&{system:serviceaccount:kube-system:tiller 0eff592d-3040-11e8-988a-0a58ac1f0754 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]

For testing purposes, has anyone figured out how to create an AKS cluster with RBAC disabled? Or, has anyone figured out the steps required to get jupyterhub to install with it enabled? Any help on this would be greatly appreciated!

Related: https://github.com/pangeo-data/pangeo/issues/167

[tjcrone]

cc @kaipak

[yuvipanda]

Thank you for reporting this in detail!

I think AKS clusters come with helm pre-installed. Can you create a new
cluster and try not doing the 'helm init' step in the z2jh guide, but
doing everything else (with rbac turned on)?

On Sun, Mar 25, 2018 at 12:33 PM, Tim Crone [e-mail]
wrote:

cc @kaipak https://github.com/kaipak

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/602#issuecomment-375996807,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23ia9VnPXnuaVcbM6TnXbq6YdHTXaks5th_EkgaJpZM4S6MbD
.

--
Yuvi Panda T
http://yuvi.in/blog

[tjcrone]

When skipping that step I get: Error: could not find tiller. And both before and after this step there is no cluster-admin account:

tjc@europa:~/research/pangeo/gce$ kubectl get serviceaccounts --all-namespaces
NAMESPACE      NAME                                      SECRETS   AGE
azure-system   default                                   1         12m
default        default                                   1         14m
jupyter        default                                   1         1m
jupyter        hook-image-awaiter-jupyter-1-1522009291   1         1m
jupyter        hook-image-awaiter-jupyter-1-1522009390   1         11s
kube-public    default                                   1         14m
kube-system    default                                   1         14m
kube-system    heapster                                  1         14m
kube-system    kube-dns                                  1         14m
kube-system    tiller                                    1         5m
tjc@europa:~/research/pangeo/gce$ kubectl get clusterrolebinding --all-namespaces
NAMESPACE   NAME      AGE
            tiller    5m

[yuvipanda]

hmm, are you sure AKS now has RBAC enabled? if there's no
clusterrolebindings at all, that feels doubtful to me. There's also no
public announcement or anything about RBAC being enabled.

When doing 'helm init', what if you just did helm init - no service account
specification?

On Sun, Mar 25, 2018 at 1:25 PM, Tim Crone [e-mail] wrote:

When skipping that step I get: Error: could not find tiller. And both
before and after this step there is no cluster-admin account:

tjc@europa:~/research/pangeo/gce$ kubectl get serviceaccounts --all-namespaces
NAMESPACE NAME SECRETS AGE
azure-system default 1 12m
default default 1 14m
jupyter default 1 1m
jupyter hook-image-awaiter-jupyter-1-1522009291 1 1m
jupyter hook-image-awaiter-jupyter-1-1522009390 1 11s
kube-public default 1 14m
kube-system default 1 14m
kube-system heapster 1 14m
kube-system kube-dns 1 14m
kube-system tiller 1 5m
tjc@europa:~/research/pangeo/gce$ kubectl get clusterrolebinding --all-namespaces
NAMESPACE NAME AGE
tiller 5m

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/602#issuecomment-376000583,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23g3Id2nG5whHzyKEHLpU52tI1S4rks5th_0rgaJpZM4S6MbD
.

--
Yuvi Panda T
http://yuvi.in/blog

[tjcrone]

Thanks for helping with this @yuvipanda. It's definitely possible that RBAC is not enabled. The reason I thought it might be is because I get the same "timed out waiting for the condition" error on GCE when rbac:enabled=false now that RBAC is enabled by default for clusters on that provider. See https://github.com/pangeo-data/pangeo/issues/167. I suppose there could be other reasons for getting this same error when rbac:enabled=false? Any thoughts on what it could be?

[yuvipanda]

The only other reason I can think of is if you did not skip step 2 in
https://zero-to-jupyterhub.readthedocs.io/en/latest/setup-helm.html#initialization.
AKS requires you skip that step...

On Sun, Mar 25, 2018 at 2:35 PM, Tim Crone [e-mail] wrote:

Thanks for helping with this @yuvipanda https://github.com/yuvipanda.
It's definitely possible that RBAC is not enabled. The reason I thought it
might be is because I get the same "timed out waiting for the condition"
error on GCE when rbac:enabled=false now that RBAC is enabled by default
for clusters on that provider. See pangeo-data/pangeo#167
https://github.com/pangeo-data/pangeo/issues/167. I suppose there could
be other reasons for getting this same error when rbac:enabled=false? Any
thoughts on what it could be?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/602#issuecomment-376005499,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23pXH74zHBqnsTLWqLFWS0zUvinmsks5tiA2tgaJpZM4S6MbD
.

--
Yuvi Panda T
http://yuvi.in/blog

[tjcrone]

Okay, I was able to get jupyterhub installed on AKS with rbac:enabled=false by doing a full helm reset. Not sure why that worked but it did! Also made absolutely sure to skip step 2. Thanks everyone.

[choldgraf]

@tjcrone thanks for reporting your experiences here! One of the trickiest things with keeping the docs up to date is that the k8s offerings on the cloud providers (esp MS+Amazon) are a bit of a moving target, so we appreciate it!

[fuglede]

Also ran into the same issue on the default AKS setup. Including

rbac:
  enabled: false

in config.yaml took care of the issue.

[manics]

@fuglede how did you install your cluster? I've just updated the docs: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/949#issuecomment-425914370, RBAC should work

[fuglede]

@manics Through the web interface, which probably explains it, as RBAC is disabled by default there:

[ryanlovett]

According to this FAQ, clusters created with the azure-cli have RBAC enabled by default. I created my cluster with az however I ran into the same issue as @tjcrone.

Not sure what I've different than @manics. I'm on a 1.11.x cluster.

[ryanlovett]

I decided to let aks use the default k8s version which is 1.9.11 and everything has worked.