The-turing-way: Problems with BuildABinderHub entirely on Azure

[arm61]

A few weeks ago I tried to build a binderhub entirely on azure using Azure Container Registry rather than docker hub. Below are my notes annotated to reflect the difficulties found.

Build a BinderHub

For the most part I am going to use the following resourse as a guide: binderhub.readthedocs.io

Azure

First we will try and build on azure and see how it goes.

I googled "azure student" and made a studnet account that gives me $100 of free time, which should be enough for the workshop (but I dunno if it is just students or all with .ac.uk emails).

Start time: 15:40

Kubernetes

First task is to get a Kubernetes Cluster on Azure, there appears to be some Microsoft documentation for this: docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough so lets start there.

Opening the bash shell for azure (clicking the wee shelly button at the top), the first thing we are asked is to create some storage, so lets do that with my "Azure for Students" subscription.

With the shell open we need to create a resource group on which the Kubernetes Cluster will be deployed.

az group create --name binderHubGroup --location ukwest

I chose ukwest as this seemed like the closest to me...the different options can be found with

az account list-locations

With the resource group created, we can now create the AKS (Azure Kubernetes Service)

az aks create --resource-group binderHubGroup --name binderHubCluster --node-count 1 --enable-addons monitoring --generate-ssh-keys

This is a waiting point of about 12 minutes (15:57 to 16:09).

Then we connect to the Kubernetes Cluster

az aks get-credentials --resource-group binderHubGroup --name binderHubCluster

We can check this with

kubectl get nodes

That seems to be the Kubernetes cluster set up so now I will move back to following the binderhub documentation.

It seems that helm is already installed on the azure kubernetes cluster that was build so we can skip that step.

Although installed we still need to initialize helm on the cluster

kubectl get namespaces

kubectl --namespace kube-system create serviceaccount tiller

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

helm init --service-account tiller --wait

kubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'

This final step seems to secure tiller, I am not 100 % sure that is it correct when using azure...

Then check that the client and server have the same version of helm

helm version

the version numbers should be the same.

Container Registry

The binderhub guidence want to set up our container registry on google cloud or dockerhub, but this link seems to be guidence to do it on Azure: docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-prepare-acr

It is not necessary to create another resource group, so we skip the first step and go straight to creating the registry.

az acr create --resource-group binderHubGroup --name binderHubRegistry --sku Basic

This seemed to take about a minute.

Hmm the logging into the Registry means running the docker daemon, which the Azure Cloud Shell on the webpage doesn't support. Therefore I need to install the Azure CLI. Looks like there is a brew recipe for macOS fun!

Having installed the Azure CLI I now need to install docker this involves creating an account...

(we should make sure attendees do this beforehand)

The download of the docker.dmg on my laptop took like 10 minutes...

Finally with Docker AND azure-cli installed locally we are able to login to the registry with

az acr login --name binderHubRegistry

We will need the AcrLoginServer address for later which we can find with the following command

az acr list --resource-group binderHubGroup --query "[].{acrLoginServer:loginServer}" --output table

For me this returned binderhubregistry.azurecr.io

BinderHub

We are back again to following the binderhub docs and on step 3,

First we need to get some passwords and stuff for the Registry

az acr update --name binderHubRegistry --admin-enabled true
az acr credential show --name binderHubRegistry

The first line could be done earlier.

Make the secret.yaml has described in the binderhub docs.

The was taken from github.com/jupyterhub/binderhub/issues/385

My secret yaml looks like This

jupyterhub:
  hub:
    services:
      binder:
        apiToken: "xxxxxx"
  proxy:
    secretToken: "xxxxxx"
registry:
  url: https://binderhubregistry.azurecr.io
  password: |
    {
      "passwords": [
        {
          "name": "password",
          "value": "xxxxxx"
        },
        {
          "name": "password2",
          "value": "xxxxxx"
        }
      ],
      "username": "binderHubRegistry"
    }

my config.yaml looks like this

config:
  BinderHub:
    use_registry: true
    image_prefix: binderhubregistry.azurecr.io/binder-
  DockerRegistry:
      token_url: https://binderhubregistry.azurecr.io/v2/token?service=

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart
helm repo update
helm install jupyterhub/binderhub --version=0.2.0-64bb3c7 --name=bhub --namespace=bhub -f secret.yaml -f config.yaml

kubectl --namespace=bhub get svc proxy-public

We want the EXTERNAL-IP

then add this to the config.yaml

config:
  BinderHub:
    use_registry: true
    image_prefix: binderhubregistry.azurecr.io/binder-
    hub_url: http://52.142.151.108
  DockerRegistry:
    token_url: https://binderhubregistry.azurecr.io/v2/token?service=

helm upgrade bhub jupyterhub/binderhub --version=0.2.0-64bb3c7 -f secret.yaml -f config.yaml

Everytime I tried to run a test repo in the binder webpage (which did build) I got a HTTP401: Unauthorised error showing up in the logs.

I traced this to be a problem in the secret/config.yaml. I also tried a whole bunch of variations with the username and password pairings and having this info in the config.yaml instead .

I wound up in this info doc a lot but the

az acr repository list -n contosoregistry

wouldn't return the JWT that is claimed.

Stoping for the night 19:20

[arm61]

I hope that makes sense, I am a bit swamped this week so have just copy and paste my notes from when I was doing it all

[sgibson91]

@arm61 No worries, thank you so much for this!

@martintoreilly Andrew's difficulties getting ACR working on a BinderHub on Azure

[trallard]

🤔 @arm61 I did not go over everything in the doc you linked in here so apologies if this is documented (I had a quick scan). Just got curious about this:

Every time I tried to run a test repo in the binder webpage (which did build) I got a HTTP401: Unauthorised error showing up in the logs.

Did you ensure your Kubernetes cluster could have access to your Azure registry? It seems this might be the reason why you are getting the unauthorized issue. This seems like an obvious thing especially if they are in the same resource group but you need to explicitly grant access from one another.

[arm61]

Ok fun! we have an azure pro here 🙂

I did not do that (mainly cause I did not know how to). So you may be onto something

[trallard]

Aha! I will be at the Binderhub workshop on Monday so I can write the how-to for this then (I am off work now and off duties)

[arm61]

Awesome! Chat then and enjoy your weekend 🙂

[sgibson91]

@trallard That would be awesome! I think we'll need an ACR for the Turing BinderHub, so cracking this would be great :slightly_smiling_face:

[trallard]

I am on my last 'work' day before going offline until the 28th. I can post that here then (hope this is not too urgent) sorry

[arm61]

No rush from me! Enjoy offline-time :smile:

[sgibson91]

I can definitely do some reading around the permissions as well while you're offline. Have fun! 😊

[trallard]

##################################
# You will need to change the below two lines to match
# your acr
ACR_NAME=<replace>
EMAIL_ADDRESS=<replace>
##################################

SERVICE_PRINCIPAL_NAME=<replace>

# Populate the ACR login server and resource id.
ACR_LOGIN_SERVER=$(az acr show --name $ACR_NAME --query loginServer --output tsv)
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)

# Create a 'Reader' role assignment with a scope of the ACR resource.
SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME \
  --role AcrPull \
  --scopes $ACR_REGISTRY_ID --query password --output tsv)

# Get the service principal client id.
CLIENT_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)

# Output used when creating Kubernetes secret.
echo "Service principal ID: $CLIENT_ID"
echo "Service principal password: $SP_PASSWD"

# Now create the secret in kubernetes that contains this username and password
kubectl create secret docker-registry acr-auth \
  --docker-server $ACR_LOGIN_SERVER \
  --docker-username $CLIENT_ID \
  --docker-password $SP_PASSWD \
  --docker-email $EMAIL_ADDRESS

[sgibson91]

Hi @trallard , is it possible to change the email address associated with the ACR after this ☝️ has been executed?

[trallard]

🤔 I am not sure, it might be that you can change the email address on the ACR and would need to regenerate (or dispose and generate a new role assignment) to match.

Might do some testing after PyCon

[sgibson91]

I think I'll set up Hub23 gmail and docker accounts just in case. Thanks!

[martintoreilly]

@sgibson91 the Turing IT folk will be able to set you up a shared email account for this.

[sgibson91]

I think what we're missing here is another edit to config.yaml to enable imagePullSecret:

jupyterhub:
  hub:
    imagePullSecret:
      enabled: true

This claims to automatically do the kubectl create secret part automatically!

[trallard]

You still need to specify the registry name/server or this defaults to docker hub so you need to add:

jupyterhub:
  hub:
    imagePullSecret:
      enabled: true
    registry: private-docker-registry

and you still need the username and the password as this only automates the kubectl but you need these credentials to access the registry

[trallard]

A common practices is to have something like:

imageCredentials:
  name: credentials-name
  registry: private-docker-registry
  username: user
  password: pass

in a values.yaml
Then pull these using jinja templates

imagePullSecrets:
  - name: {{ .Values.imageCredentials.name }}

see from helm docs

[sgibson91]

Hmmm, there's something still not working! Keep getting Internal Server Errors on the launch page.

[trallard]

what does

kubectl get secret hub-image-credentials --output yaml 

return (the values you expect or something non sensical - do not paste the contents here as these are your access credentials)

1. did you try with the explicit kubectl command?

2. double check that you have in fact a reader or acrpull role assignment created

[sgibson91]

1. what does

kubectl get secret hub-image-credentials --output yaml 

return (the values you expect or something non sensical - do not paste the contents here as these are your access credentials)

Error from server (NotFound): secrets "hub-image-credentials" not found

So the hub hasn't been able to create the secret yet.

1. did you try with the explicit kubectl command?

Not yet. I'm going to try 1 more configuration before I'll give this a go.

1. double check that you have in fact a reader or acrpull role assignment created

When I view the Role Assignments under the Access Control panel of the registry, there's an AcrPush role.

[trallard]

AcrPush should be ok as this gives you implicit pull access

From the docs (you pasted above) it seems to imply that you should have a hub-image-credentials already if you are using imagePullsecret ⬇

# you won't need to run this manually...
kubectl create secret docker-registry hub-image-credentials \
  --docker-server=<REGISTRY> \
  --docker-username=<USERNAME> \
  --docker-email=<EMAIL> \
  --docker-password=<PASSWORD>

[sgibson91]

Yeah, so something's happening that preventing the creation of that secret. How does the email get parsed? I've got nothing with an email in it right now.

@betatim any ideas?

[trallard]

Ah that is what I was going to before you still need the user, password and email 🤔 wait the docs seem to imply you only need one or the other but in the regular practice you provide the three plus the server

[trallard]

again from the helm docs

Image pull secrets are essentially a combination of registry, username, and password. You may need them in an application you are deploying, but to create them requires running base64 a couple of times.

so maybe adding the fourth is more like a precaution.... I always tend to add it anyway

[sgibson91]

Yeah, so this is what my config looks like at the minute:

jupyterhub:
  hub:
    imagePullSecret:
      enabled: true
      registry: http://<ACR_NAME>.azurecr.io
      username: _json_key
      password: |
        {
          "username": "<ACR_NAME>"
          "password": "xxxx",
          "password2": "xxxx"
        }

Do you think adding "email": "<EMAIL>" to that json will work?

[trallard]

add a
name: hub-image-credentials

and see if that way you can do:

kubectl get secret hub-image-credentials --output yaml 

[sgibson91]

Same error as before 😕

[trallard]

🤔 jajaja seems easier with kubectl

yeh add it like so

{
     "username":"registry_user", 👉🏼 above it is ACR_NAME is this the same as your ACR_NAME ?
     "password":"reg_password",
     "email":"something",
 }

[sgibson91]

# you won't need to run this manually...
kubectl create secret docker-registry hub-image-credentials \
  --docker-server=<REGISTRY> \
  --docker-username=<USERNAME> \
  --docker-email=<EMAIL> \
  --docker-password=<PASSWORD>

I'm trying with this command and keep getting the following!

error: exactly one NAME is required, got 4

[trallard]

kubectl create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

definitely correct... unless bash does not like the indentation after the \

[sgibson91]

Didn't use the indentation or \!

[trallard]

I just ran that from my command line 🤯

[sgibson91]

Didn't like the json file containing two passwords apparently

[trallard]

ahhhhh yes that might be it... but in the step above you are not passing the json... do you need 2 passwords?

[sgibson91]

Yeah, but the user has two pass words, password and password2

see https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-491329777

they came from az acr credential show

[sgibson91]

I have no idea what I need! 😅

[trallard]

how did you come to get those two passwords? normally it is only one

[sgibson91]

No idea, I ran az acr credential show -n <ACR_NAME> and there are two passwords

[trallard]

Did you run the stuff as in https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-481674873 ?

This has made me feel a bit confused now...edit run as

az acr credential show --name ACR_NAME "passwords[0].value"

[sgibson91]

I did

az role assignment create --assignee $SERVICE_PRINCIPAL_CLIENT_ID --scope $ACR_ID --role acrpush

I already have a service principal in a file that I can call so a lot of the commands in that comment were superfluous for me. It's also the same SP that deployed the cluster.

[sgibson91]

@arm61 also got 2 passwords in his original post

[trallard]

yes try with the query in https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-491343247 to get just the one pwd you need

[sgibson91]

Got an error: az: error: unrecognized arguments: passwords[0].value

[trallard]

az acr credential show -n MyRegistry --query passwords[0].value oops deleted the --query

[sgibson91]

Still Internal Server Errors!!

[trallard]

Aaaaaaa I have gotten this working on before so not sure what this is

Have you checked your helm template?

I can try and deploy next week when I am back at home. Wanna do a remote pair coding thing?

[sgibson91]

Yeah sounds good!. I haven't changed BinderHub's helm template - it's just what I'm adding to a config.yaml file

[sgibson91]

Hey @trallard want to email me your availability for pair programming and I'll book a zoom call? Also, I'm coming to the sprint next week so we could do some more work together then 😄

[sgibson91]

Ok, so I have managed to push/pull an image to my ACR using the Docker CLI, by doing the following:

docker login <ACR_NAME>.azurecr.io

When asked for a username and password, I parsed the Service Principal Client ID and Secret that Turing IT generated for me to deploy the Kubernetes cluster in the first place.
I used the az role assignment command to give this SP AcrPush role.

az role assignment create --assignee $CLIENT_ID --role acrpush --scope $ACR_ID

Then, get a container, tag it, push it to my registry and locally remove it:

docker pull hello-world
docker tag hello-world <ACR_NAME>.azurecr.io/hello-world:v1
docker push <ACR_NAME>.azurecr.io/hello-world:v1
docker rmi <ACR_NAME>.azurecr.io/hello-world:v1

I ran az acr repository list -n <ACR_NAME> -o table and az acr repository show-tags -n <ACR_NAME> --repository hello-world -o table to verify that the container was in my registry.

Then:

docker run <ACR_NAME>.azurecr.io/hello-world:v1

which gave:

Unable to find image '<ACR_NAME>.azurecr.io/hello-world:v1' locally
v1: Pulling from hello-world
Digest: sha256:<redacted>
Status: Downloaded newer image for <ACR_NAME>.azurecr.io/hello-world:v1

Hello from Docker!
This message shows that your installation appears to be working correctly.

So hooray, I can put an image into my registry! :tada:

So now I know that my username and password are the SP Client ID and Secret, I created the following Kubernetes secret:

kubectl create secret docker-registry hub-image-credentials \
  --docker-server=<ACR_NAME>.azurecr.io \
  --docker-username=$CLIENT_ID \
  --docker-password=$CLIENT_SECRET \
  --docker-email=<an email address>

So now where I'm at is finding the correct place in the BinderHub config.yaml to place:

imagePullSecrets:
  - name: hub-image-credentials

so that BinderHub has the credentials.

[trallard]

Also make sure to set imagePullPolicy to always

[trallard]

This normally goes under a spec section but according to https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/facf1c31972b32ffa1418b1431efb5c7392a9175/jupyterhub/values.yaml#L53 this seems to be under hub directly:

hub:
  imagePullSecrets:
    enabled: false
    registry:
    username:
    email:
    password:

So you might be able to add your

hub:
  imagePullSecrets:
  - name: hub-image-credentials

directly like so

[sgibson91]

Also make sure to set imagePullPolicy to always

Is this just the following:

jupyterhub:
  hub:
    imagePullPolicy: Always

Or is there another key under imagePullPolicy?

Also, do you know what the token_url would be from here?

[sgibson91]

NOTE
When creating a secret, make sure you parse the namespace of the pods you want to be able to access the secret. Secrets are not accessible across namespaces.

kubectl create secret docker-registry <SECRET_NAME> \
  --docker-server=<SERVER_NAME> \
  --docker-username=$CLIENT_ID \
  --docker-password=$CLIENT_SECRET \
  --docker-email=<an-email-address>
  --namespace=<NAMESPACE>

[sgibson91]

secret.yaml

registry:
  url: "http://<ACR_NAME>.azurecr.io"
  username: $CLIENT_ID
  password: $CLIENT_SECRET

Attempted config.yaml files

jupyterhub:
  hub:
    imagePullSecret:
      enabled: true
      registry: <ACR_NAME>.azurecr.io
      username: $CLIENT_ID
      password: $CLIENT_SECRET

The above configuration successfully created the hub-image-credentials secret in the correct namespace but the BinderHub kept giving Internal Server Error when clicking launch.

Used the command in https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-491820157 to create hub-im-creds and parse this directly.

jupyterhub:
  imagePullSecrets:
    - name: hub-im-creds

jupyterhub:
  hub:
    imagePullSecrets:
      - name: hub-im-creds

Still receiving Internal Server Error

Other info

config:
  DockerRegistry:
    token_url: None

I'm not sure what token_url needs to be for ACR.

[sgibson91]

So I manually added the binder-examples/requirements image to the ACR (pulled it from my DockerHub org that was originally linked), and I'm still getting Internal Server Errors.

My guess now is that the Service Principal I have needs user_impersonator API permission to the ACR. Which means another conversation with Turing IT.

[sgibson91]

See https://github.com/jupyterhub/binderhub/issues/800#issuecomment-491923206 for what the DockerRegistry.token_url should be

[sgibson91]

Thanks to @trallard and a Microsoft Azure Pass, I managed to get a BinderHub to push to an ACR not on the Turing infrastructure. Here's what I did:

# logged in with personal account
az login -o none

# select Azure Pass subscription
az account set -s "Azure Pass - Sponsorship" -o none

# create a resource group
az group create -n binderhub --location westeurope -o table

# create an acr
az acr create -g binderhub -n binderhubregistry --sku Basic -o table

# login to the acr
az acr login -n binderhubregistry

# save the acr login server to a variable
acrlogin=$(az acr list -g binderhub --query "[].{acrLoginServer:loginServer}" -o tsv)

# create a service principal - note appID and password from output and save to variables
az ad sp create-for-rbac --skip-assignment

# save acr ID to a variable
acrid=$(az acr show -g binderhub -n binderhubregistry --query "id" -o tsv)

# assign AcrPush role to service principal
az role assignment create --assignee $appID --scope $acrid --role acrpush

# create k8s cluster
az aks create -g binderhub -n hubcluster -c 3 --service-principal $appID --client-secret $password --node-vm-size Standard_D2s_v3 -o table

# get credentials for cluster
az aks get-credentials -g binderhub -n hubcluster

# create tiller service account
kubectl --namespace kube-system create serviceaccount tiller

# create cluster role binding for tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

# initialise helm and tiller
helm init --service-account tiller --wait

# patch tiller
kubectl patch deployment tiller-deploy --namespace=kube-system --type=json --patch='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value": ["/tiller", "--listen=localhost:44134"]}]'

# create tokens for binderhub
mkdir testhub && cd testhub
mkdir secrets
openssl rand -hex 32 > secrets/apiToken.txt
openssl rand -hex 32 > secrets/secretToken.txt

Create secrets/secret.yaml (connect to my DockerHub first for testing):

jupyterhub:
  hub:
    services:
      binder:
        apiToken: $(cat secrets/apiToken.txt)
  proxy:
    secretToken: $(cat secrets/secretToken.txt)

registry:
  username: sgibson91
  password: xxxx

Create secrets/config.yaml:

config:
  BinderHub:
    use_registry: true
    image_prefix: sgibson91/binder-dev-

# pull binderhub helm charts
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart
helm repo update

# install binderhub helm chart
helm install jupyterhub/binderhub --version=0.2.0-3b53fce --name=binderhub --namespace=binderhub -f secrets/secret.yaml -f secrets/config.yaml

# get jupyterhub IP
kubectl get svc proxy-public -n binderhub

Update secrets/config.yaml with JupyterHub IP:

config:
  BinderHub:
    use_registry: true
    image_prefix: sgibson91/binder-dev-
    hub_url: http://xx.xx.xx.xx

# upgrade helm chart
helm upgrade binderhub jupyterhub/binderhub --version=0.2.0-3b53fce -f secrets/secret.yaml -f secrets/config.yaml

# get binder IP
kubectl -n binderhub get svc binder

At this point, I tested the BinderHub was pulling from DockerHub as expected with binder-examples/requirements repo - all good.

Moved on to connecting the ACR.

Updated secrets/secret.yaml:

jupyterhub:
  hub:
    services:
      binder:
        apiToken: $(cat secrets/apiToken.txt)
  proxy:
    secretToken: $(cat secrets/secretToken.txt)

registry:
  url: https://binderhubregistry.azurecr.io
  username: $appID  # service principal app ID
  password: $password  # service principal secret

Update secrets/config.yaml:

config:
  BinderHub:
    use_registry: true
    image_prefix: binderhubregistry.azurecr.io/binder-dev-
    hub_url: http://xx.xx.xx.xx
  DockerRegistry:
    token_url: "https://binderhubregistry.azurecr.io/oauth2/token?service=binderhubregistry.azurecr.io"

jupyterhub:
  hub:
    imagePullSecret:
      enabled: true
      registry: binderhubregistry.azurecr.io
      username: $appID
      password: $password

Upgrade the helm chart again:

helm upgrade binderhub jupyterhub/binderhub --version=0.2.0-3b53fce -f secrets/secret.yaml -f secrets/config.yaml

Get binder IP:

kubectl get svc binder -n binderhub

Test with binder-examples/requirements repo - all good!

---

Next thing to figure out. What's different between Azure Pass deployment and Turing-BinderHub deployment? Is it all in the service principal?

---

Note to self: This comment needs to go into the BinderHub docs.

[KirstieJane]

Oooooooh I love this comment so much. Way to persevere @sgibson91. Thank you @trallard 🌺🌸🌻

[trallard]

Interesting 😑 seems this is an issue with your other account or rather the service principal.

Let's get all this documented on Thursday and next week 🙃 I completely forgot to mention that point about the namespace and the secret 🤦🏻‍♀️

[sgibson91]

SP_PASSWD=$(az ad sp create-for-rbac --name $SERVICE_PRINCIPAL_NAME \
  --role AcrPull \
  --scopes $ACR_REGISTRY_ID --query password --output tsv)

Hey @trallard Is there a query to return the password after the Service Principal has been created? For example, in the scenario where a user has deployed a k8s cluster that has auto-generated a SP and they're trying to access the info after the fact. I'm trying to get the ACR stuff to fit with the flow of the BinderHub documentation. The easiest way to do this programmatically is as I've done in https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-492320069 (build ACR first, then AKS) but that doesn't quite gel with the eco-system of the BinderHub docs (AKS first, then deal with a container registry).

[trallard]

Yes you can grant access to a pre-existing service principal. To do so you have to assign a new role to it like so:

# ACR_NAME is the name of your Azure Container registry
# and the SERVICE_PRINCIPAL_ID is the service principal's 'appId' or
# one of its 'servicePrincipalNames' values.
ACR_NAME=mycontainerregistry
SERVICE_PRINCIPAL_ID=<service-principal-ID>

# Populate value required for subsequent command args
ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)

az role assignment create --assignee $SERVICE_PRINCIPAL_ID --scope $ACR_REGISTRY_ID --role acrpull

That way you can follow a more 'Binder' style flow:

1. Create resource group
2. Create cluster
3. All things tiller, Helm, etc
4. Create ACR
5. Authorize access to the ACR 👆🏼
6. Done

[sgibson91]

Notes from meeting with Tania 16/05/2019:

• rbac permissions probably haven't been added to Turing SP which is why connecting to the ACR won't work. Ask IT to make a new service principal using create-for-rbac command in Azure shell and send the output.
• SP password probably can't be recovered programmatically after creation. For ACR documentation in BinderHub, get users to create a new SP using create-for-rbac.
• At Reactor sprint next week: update docs, look into nodepools for JupyterHub (also a docs PR), set up Azure pipeline to auto-deploy or auto-update a BinderHub with CI, write a blog about my journey with Azure, about a million other cool ideas! 🌻

(Feel free to edit if I've missed anything!)

[trallard]

This all looks good. Re asking IT make sure that they run

az ad sp create-for-rbac --skip-assignment --name <assign_meaningful_name>

better name the SP as well as this makes it easier when auditing your resources

[sgibson91]

So I just tried to build a BinderHub and connect it to an ACR in the flow that BinderHub's documentation would suggest, i.e.:

• deploy Kubernetes cluster, which autogenerates a Service Principal (from Step Zero: Kubernetes on AKS)
• Create an ACR (this would be added to Setup the container registry)
• Create a new Service Principal with RBAC and AcrPush role and scope of the just-built ACR (also added to Setup the container registry docs)
• Connect the ACR to BinderHub through config files (which would be added to Setup BinderHub)

But this didn't work and I got Internal Server Errors.

Either:

• k8s cluster needs to be on the same SP as the registry (see flow in https://github.com/alan-turing-institute/the-turing-way/issues/317#issuecomment-492320069), or;
• I'm distracted and have messed up somewhere

[sgibson91]

I got this to work in the end. The workflow in this comment works and has been added to the BinderHub documentation.