Aks-engine: Permission issue using autoscaler plugin

[Timost]

Describe the bug
I use aks-engine to create a k8s cluster which I want to have the following properties:

• Belong to a dedicated resource group for easier resource management (resource-group-1)
• Be able to query other resources in an existing, custom VNET (main-vnet), which belongs to another resource group (resource-group-main)
• Has 2 VMSS backed node pools:
  
  
  
  • High priority node pool with auto scaling
  
  
  • Low priority node pool with auto scaling
  
  

The cluster is created using a service principal with the following role:

• Owner on the resource-group-1 resource group
• Contributor on the main-vnet VNET

To avoid ip conflicts, I've properly created and assigned two separate subnets in main-vnet:

• one for the kubernetes master node: master-subnet
• one for the other nodes (both node pools): agents-subnet

The cluster-autoscaler fails to scale the low priority node pool because of a permission error:

E1126 17:02:06.863462       1 azure_scale_set.go:213] virtualMachineScaleSetsClient.CreateOrUpdate for scale set "k8s-burst-12135021-vmss" failed: 
compute.VirtualMachineScaleSetsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=403 
Code="LinkedAuthorizationFailed" Message="The client '<k8s-master-service-principal>' with object id '<redacted>' has permission to perform action 
'Microsoft.Compute/virtualMachineScaleSets/write' on scope '/subscriptions/<redacted>
/resourceGroups/<resource-group-1>/providers/Microsoft.Compute/virtualMachineScaleSets/<low-priority-vmss>'; however, it does not have permission to perform action 
'Microsoft.Network/virtualNetworks/subnets/join/action' on the linked scope(s) '/subscriptions
/<redacted>/resourceGroups/<resource-group-main>/providers/Microsoft.Network/virtualNetworks/<main-vnet>/subnets/<agents-subnet>' or the linked scope(s) are invalid."

It seems the Service Principal associated to kubernetes master does not have the permission to add a new machine to the agent subnet.

Steps To Reproduce
The cluster is created using:

 aks-engine deploy -f --resource-group "dask-cluster" --debug\
  --location "westeurope" \
  --subscription-id "<redacted>" \
  --client-id "<redacted>" \
  --client-secret "<redacted>" \
  --api-model "./cluster-definition.json" \
  --set linuxProfile.ssh.publicKeys[0].keyData="<redacted>" \
  --set masterProfile.dnsPrefix="dask-cluster-`date +"%m-%d-%yT-%H-%M-%S"`", \
  --set servicePrincipalProfile.clientId="<redacted>" \
  --set servicePrincipalProfile.secret="<redacted>"

where cluster-definition is:

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.14.7",
      "kubernetesConfig": {
        "useManagedIdentity": true,
         "addons": [
          {
            "name": "cluster-autoscaler",
            "enabled": true,
            "pools": [
              {
                "name": "burst",
                "config": {
                  "min-nodes": "0",
                  "max-nodes": "5"
                }
              },
              {
                "name": "stable",
                "config": {
                  "min-nodes": "1",
                  "max-nodes": "2"
                }
              }
            ],
            "config": {
              "scan-interval": "1m"
            }
          }
        ]
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "dask-cluster",
      "vmSize": "Standard_D4_v3",
      "firstConsecutiveStaticIP": "<redacted>",
      "vnetSubnetId": "/subscriptions/<redacted>/resourceGroups/<resource-group-main>/providers/Microsoft.Network/virtualNetworks/<main-vnet>/subnets/<master-subnet>"
    },
    "agentPoolProfiles": [
      {
        "name": "burst",
        "count": 3,
        "vmSize": "Standard_D4_v3",
        "availabilityProfile": "VirtualMachineScaleSets",
        "scaleSetPriority": "Low",
        "scaleSetEvictionPolicy": "Delete",
        "storageProfile": "ManagedDisks",
        "OSDiskSizeGB": 200,
        "vnetSubnetId": "/subscriptions/<redacted>/resourceGroups/<resource-group-main>/providers/Microsoft.Network/virtualNetworks/<main-vnet>/subnets/<agents-subnet>"
      },
      {
        "name": "stable",
        "count": 1,
        "vmSize": "Standard_D4_v3",
        "availabilityProfile": "VirtualMachineScaleSets",
        "scaleSetEvictionPolicy": "Delete",
        "storageProfile": "ManagedDisks",
        "OSDiskSizeGB": 200,
        "vnetSubnetId": "/subscriptions/<redacted>/resourceGroups/<resource-group-main>/providers/Microsoft.Network/virtualNetworks/<main-vnet>/subnets/<agents-subnet>"
      }
    ],
    "linuxProfile": {
      "adminUsername": "<redacted>",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    }
  }
}

Expected behavior
Since the service principal used to create the cluster has Contributor role on the Vnet I would expect that the k8s-master be able to auto-scale properly.

AKS Engine version
v0.44.0
Kubernetes version
1.14.7
Additional context
I'm not sure this is a bug, it might very well be an issue with the permissions set up on my side. I've done my best trying to figure out which role I need to affect to the service principal, but I've been having a really hard time to find proper details in the docs for each functionality (auto-scaler, custom Vnet, etc...).
Otherwise the cluster works perfectly. Thank you for aks-engine !

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[devigned]

Can you add contributor rights to the SP for the agents-subnet and see if that solves the issue?

[Timost]

OK so adding the Contributor rights to the kubernetes master SP did solve the issue (took some time to propagate). The thing is, I can't know this SP before the cluster creation and I would like not to have to give it rights after cluster creation. I'm trying to automate things as much as possible.
Is there any role that I can give to the SP I use to create the cluster so that the resulting kubernetes master SP can have the right rights "out of the box" ?

[devigned]

What do you think about using Managed Identity or User Assigned Identity?

Why would you not know the SP before cluster creation? It could be passed in as part of cluster creation.

[Timost]

As far as I can tell, I'm already using Managed Identity, I set "useManagedIdentity": true, in the cluster configuration. I've been trying to find extra information on the docs but I only found this: https://github.com/Azure/aks-engine/blob/da5ad7c157c0e172703fff00f834cf7c9e3f28c7/docs/topics/features.md#managed-identity
How can I set the SP to use for the k8s master ? I mean I'm already pass a specific SP when calling aks-engine deploy which should have all the correct rights but it's another one that's used by the k8s master.

I've also tried this:

aks-engine deploy -f --resource-group "<redacted>" \
  --location "westeurope" \
  --subscription-id "<redacted>" \
  --api-model "./cluster-definition.json" \
  --set linuxProfile.ssh.publicKeys[0].keyData="<redacted>" \
  --set masterProfile.dnsPrefix="dask-cluster-`date +"%m-%d-%yT-%H-%M-%S"`"
  --auth-method cli

In the hope that it would create all the correct permissions and service principals but the autoscaling still fails, and I have to manually give the Contributor role to the SP created in the process to make it work.
Thank you for your help !

[devigned]

Hmm... when using Managed Identity or User Assigned Identity, aks-engine should be ensuring the identity which is used in the cluster has access to all of the resources it needs to manipulated (custom vnet, subnets, etc...). We should be adding RBAC role assignments to the deployment template. This is a bug / new feature that is required for having Managed Identities work with user provided resources.

One way you can get around this is by not using Managed Identity, but by creating a Service Principal before deployment, assigning the RBAC roles to the SP, and then running --set to update:
https://github.com/Azure/aks-engine/blob/1cb07950f1a25a16053cb97bf7b18670440db862/examples/kubernetes.json#L29-L32

That should get you a cluster using the service principal you have specified.

[Timost]

Yes I agree.

After some more digging, I've reached the same conclusion and I can confirm that disabling Managed Identity and using a custom SP instead solves all my permissions issues (I also had issues with internal load balancer among other things).

I'll use the custom SP approach until this is fixed.

Thank you again for your help

[devigned]

Thank you for digging into this with me. We'll get this prioritized and fixed.

It would be a lot more fun to use a Managed Identity rather than deal with SP creation and role assignment.

[Timost]

No problem.

Maybe giving more details about what lead me to do this whole setup can be of interest.

I need to run daily TB scale dask-based jobs as fast and cost effectively as possible.
I need auto-scaling because the load varies across the jobs life-cycle.

Because AKS does not offer low-priority node pools I had to use aks-engine to save money essentialy.
However I also need access to other existing azure resources which were created in the past in their dedicated resource groups (VNET etc...). Also having a separate resource group for the jobs makes it easier to manage.

All this requirements lead me to the current setup.

[devigned]

@craiglpeters ^ sounds like an interesting scenario.