Azure-cli: `az aks create` gives `Could not create a role assignment for subnet. Are you an Owner on this subscription? Operation failed with status: 'Bad Request'. Details: Internal server error`

Created on 8 Aug 2019  路  9Comments  路  Source: Azure/azure-cli

This is autogenerated. Please review and update as needed.

Describe the bug

I'm unable to create a kubernetes cluster with the below az cli command, with the settings/inputs given.

Command Name
az aks create

Errors:

Operation failed with status: 'Bad Request'. Details: Internal server error

To Reproduce:

I've written some shellscripts to automate the creation and set-up of the K8s cluster and associated resources (acr/keyvault/etc.).
Below are some relevant files and the bash output of said terminal execution, where you can see me create from scratch all the resources, with actual valid inputs/outputs (I'll run the delete script before I post this though).

setup/_bash_support/debug.sh:

#!/usr/bin/env bash

export PS4='# [${BASH_SOURCE}:${LINENO}]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

set -o errexit  # Stop script on first error (non-zero returncode)
set -o pipefail # Stop script on first error in a piped command (default only checks last pipe-command)
set -o verbose  # Verbose debugprinting of executing scripts
set -o xtrace   # Show commands being executed through debugprint during execution

setup/init_env.sh:

#!/usr/bin/env bash

FILE_DIR=$(dirname $0)
source "${FILE_DIR}"/_bash_support/debug.sh

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/inputs.sh"
fi
set -o nounset  # Throw error on first variable-reference to an unset variable

"${FILE_DIR}/network/network.azcli"
"${FILE_DIR}/keyvault/keyvault.azcli"
"${FILE_DIR}/aks/providers.azcli"
"${FILE_DIR}/aks/k8s.azcli"
"${FILE_DIR}/acr/acr.azcli"
"${FILE_DIR}/helm/helm.sh"
"${FILE_DIR}/rbac/set-access.azcli"
"${FILE_DIR}/ingress/ingress.azcli"

_NB_: The below terminal copy/paste contains the output of my scripts having all sourced the above contents as the first thing they do. 

[2019-08-08 13:13:38] 0 user@xmachine:~/Documents/team-k8s-project $ setup/init_env.sh 
set -o xtrace   # Show commands being executed through debugprint during execution

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/inputs.sh"
fi
# [setup/init_env.sh:6]: main(): [[ -v '' ]]
# [setup/init_env.sh:6]: main(): [[ -z '' ]]
# [setup/init_env.sh:7]: main(): source setup/inputs.sh
#!/usr/bin/env bash

# Throw error on first variable-reference to an unset variable
set -o nounset
## [setup/inputs.sh:4]: source(): set -o nounset


# Commonly used variables
## _MUST_ fill-in variables!
export location="westeurope"
## [setup/inputs.sh:9]: source(): export location=westeurope
## [setup/inputs.sh:9]: source(): location=westeurope
export subscription_name="{REDACTED}"
## [setup/inputs.sh:10]: source(): export subscription_name={REDACTED}
## [setup/inputs.sh:10]: source(): subscription_name={REDACTED}
export team_name="${subscription_name}-${location}-team"
## [setup/inputs.sh:11]: source(): export team_name={REDACTED}-westeurope-team
## [setup/inputs.sh:11]: source(): team_name={REDACTED}-westeurope-team
export product_name="${team_name}-team"
## [setup/inputs.sh:12]: source(): export product_name={REDACTED}-westeurope-team-team
## [setup/inputs.sh:12]: source(): product_name={REDACTED}-westeurope-team-team
export environment='dev'
## [setup/inputs.sh:13]: source(): export environment=dev
## [setup/inputs.sh:13]: source(): environment=dev

## Derivatives
export subscription_name_capitalized="$(tr '[:lower:]' '[:upper:]' <<< ${subscription_name:0:1})${subscription_name:1}"
### [setup/inputs.sh:16]: source(): tr '[:lower:]' '[:upper:]'
## [setup/inputs.sh:16]: source(): export subscription_name_capitalized={REDACTED}
## [setup/inputs.sh:16]: source(): subscription_name_capitalized={REDACTED}
export resource_group="${product_name}-rg"
## [setup/inputs.sh:17]: source(): export resource_group={REDACTED}-westeurope-team-team-rg
## [setup/inputs.sh:17]: source(): resource_group={REDACTED}-westeurope-team-team-rg
export subscription_id=$(az account list \
    --query "[?name=='${subscription_name_capitalized}'].[id]" \
    -o tsv \
)
### [setup/inputs.sh:21]: source(): az account list --query '[?name=='\''{REDACTED}'\''].[id]' -o tsv
## [setup/inputs.sh:21]: source(): export subscription_id=15ea10b3-c25a-4569-b4c2-46054c6f5227
## [setup/inputs.sh:21]: source(): subscription_id=15ea10b3-c25a-4569-b4c2-46054c6f5227

# Resource specific variables
## Azure Container Registry (ACR)
export acr_name="$(echo "${team_name}-acr" | sed -r 's/-//g')"
### [setup/inputs.sh:25]: source(): echo {REDACTED}-westeurope-team-acr
### [setup/inputs.sh:25]: source(): sed -r s/-//g
## [setup/inputs.sh:25]: source(): export acr_name={REDACTED}westeuropeteamacr
## [setup/inputs.sh:25]: source(): acr_name={REDACTED}westeuropeteamacr
export acr_rg="${team_name}-rg"
## [setup/inputs.sh:26]: source(): export acr_rg={REDACTED}-westeurope-team-rg
## [setup/inputs.sh:26]: source(): acr_rg={REDACTED}-westeurope-team-rg
export acr_email="[email protected]"
## [setup/inputs.sh:27]: source(): export [email protected]
## [setup/inputs.sh:27]: source(): [email protected]
export secret_name="acr-auth"
## [setup/inputs.sh:28]: source(): export secret_name=acr-auth
## [setup/inputs.sh:28]: source(): secret_name=acr-auth

## Azure Keyvault (KV)
export kv_name="${team_name}"
## [setup/inputs.sh:31]: source(): export kv_name={REDACTED}-westeurope-team
## [setup/inputs.sh:31]: source(): kv_name={REDACTED}-westeurope-team
export kv_rg="${team_name}-rg"
## [setup/inputs.sh:32]: source(): export kv_rg={REDACTED}-westeurope-team-rg
## [setup/inputs.sh:32]: source(): kv_rg={REDACTED}-westeurope-team-rg
export k8s_rbac_keyname="k8s-rbac-client"
## [setup/inputs.sh:33]: source(): export k8s_rbac_keyname=k8s-rbac-client
## [setup/inputs.sh:33]: source(): k8s_rbac_keyname=k8s-rbac-client

## Azure Kubernetes Service (AKS)
export aks_name="${team_name}-aks"
## [setup/inputs.sh:36]: source(): export aks_name={REDACTED}-westeurope-team-aks
## [setup/inputs.sh:36]: source(): aks_name={REDACTED}-westeurope-team-aks
export aks_rg="${team_name}-rg"
## [setup/inputs.sh:37]: source(): export aks_rg={REDACTED}-westeurope-team-rg
## [setup/inputs.sh:37]: source(): aks_rg={REDACTED}-westeurope-team-rg
export aks_subnet="${aks_name}-subnet"
## [setup/inputs.sh:38]: source(): export aks_subnet={REDACTED}-westeurope-team-aks-subnet
## [setup/inputs.sh:38]: source(): aks_subnet={REDACTED}-westeurope-team-aks-subnet
export k8s_namespace="${product_name}-${environment}"
## [setup/inputs.sh:39]: source(): export k8s_namespace={REDACTED}-westeurope-team-team-dev
## [setup/inputs.sh:39]: source(): k8s_namespace={REDACTED}-westeurope-team-team-dev

## Networking
export vnet="${team_name}"
## [setup/inputs.sh:42]: source(): export vnet={REDACTED}-westeurope-team
## [setup/inputs.sh:42]: source(): vnet={REDACTED}-westeurope-team
export vnet_rg="${team_name}-rg"
## [setup/inputs.sh:43]: source(): export vnet_rg={REDACTED}-westeurope-team-rg
## [setup/inputs.sh:43]: source(): vnet_rg={REDACTED}-westeurope-team-rg
export subnet="${aks_name}-subnet"
## [setup/inputs.sh:44]: source(): export subnet={REDACTED}-westeurope-team-aks-subnet
## [setup/inputs.sh:44]: source(): subnet={REDACTED}-westeurope-team-aks-subnet

# Misc.
## Show my service principals
## az ad sp list --subscription "${subscription_name_capitalized}" --show-mine -o table
export SERVICE_PRINCIPAL_NAME={REDACTED}
## [setup/inputs.sh:49]: source(): export SERVICE_PRINCIPAL_NAME={REDACTED}
## [setup/inputs.sh:49]: source(): SERVICE_PRINCIPAL_NAME={REDACTED}
export SERVICE_PRINCIPAL_ID="http://${SERVICE_PRINCIPAL_NAME}"
## [setup/inputs.sh:50]: source(): export SERVICE_PRINCIPAL_ID=http://{REDACTED}
## [setup/inputs.sh:50]: source(): SERVICE_PRINCIPAL_ID=http://{REDACTED}
set -o nounset  # Throw error on first variable-reference to an unset variable
# [setup/init_env.sh:9]: main(): set -o nounset

"${FILE_DIR}/network/network.azcli"
# [setup/init_env.sh:11]: main(): setup/network/network.azcli
set -o xtrace   # Show commands being executed through debugprint during execution

echo -e "\n--- Network setup start"
# [setup/network/network.azcli:6]: main(): echo -e '\n--- Network setup start'

--- Network setup start

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/../inputs.sh"
fi
# [setup/network/network.azcli:8]: main(): [[ -v {REDACTED} ]]
# [setup/network/network.azcli:8]: main(): [[ -z {REDACTED} ]]
set -o nounset  # Throw error on first variable-reference to an unset variable
# [setup/network/network.azcli:11]: main(): set -o nounset

# Create core network
az group create \
    --location $location \
    --name $vnet_rg \
    --subscription $subscription_id
# [setup/network/network.azcli:14]: main(): az group create --location westeurope --name {REDACTED}-westeurope-team-rg --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227
{
  "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg",
  "location": "westeurope",
  "managedBy": null,
  "name": "{REDACTED}-westeurope-team-rg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": null
}

az network vnet create \
    --name $vnet \
    --resource-group $vnet_rg \
    --location $location \
    --address-prefix 10.40.0.0/16
# [setup/network/network.azcli:19]: main(): az network vnet create --name {REDACTED}-westeurope-team --resource-group {REDACTED}-westeurope-team-rg --location westeurope --address-prefix 10.40.0.0/16
{
  "newVNet": {
    "addressSpace": {
      "addressPrefixes": [
        "10.40.0.0/16"
      ]
    },
    "ddosProtectionPlan": null,
    "dhcpOptions": {
      "dnsServers": []
    },
    "enableDdosProtection": false,
    "enableVmProtection": false,
    "etag": "W/\"8c77e8d4-a1be-498c-9a06-29d3c3cea19a\"",
    "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.Network/virtualNetworks/{REDACTED}-westeurope-team",
    "location": "westeurope",
    "name": "{REDACTED}-westeurope-team",
    "provisioningState": "Succeeded",
    "resourceGroup": "{REDACTED}-westeurope-team-rg",
    "resourceGuid": "c15424c5-e428-4432-b693-29d50905e55e",
    "subnets": [],
    "tags": {},
    "type": "Microsoft.Network/virtualNetworks",
    "virtualNetworkPeerings": []
  }
}

# The name GatewaySubnet does not meet the naming convention because Azure is looking at it!
az network vnet subnet create \
    --resource-group $vnet_rg \
    --vnet-name $vnet \
    --name "GatewaySubnet" \
    --address-prefixes 10.40.0.0/24
# [setup/network/network.azcli:26]: main(): az network vnet subnet create --resource-group {REDACTED}-westeurope-team-rg --vnet-name {REDACTED}-westeurope-team --name GatewaySubnet --address-prefixes 10.40.0.0/24
{
  "addressPrefix": "10.40.0.0/24",
  "addressPrefixes": null,
  "delegations": [],
  "etag": "W/\"e54ef133-b8b5-4eeb-8d16-b267b325fcfc\"",
  "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.Network/virtualNetworks/{REDACTED}-westeurope-team/subnets/GatewaySubnet",
  "ipConfigurationProfiles": null,
  "ipConfigurations": null,
  "name": "GatewaySubnet",
  "natGateway": null,
  "networkSecurityGroup": null,
  "privateEndpoints": null,
  "provisioningState": "Succeeded",
  "purpose": null,
  "resourceGroup": "{REDACTED}-westeurope-team-rg",
  "resourceNavigationLinks": null,
  "routeTable": null,
  "serviceAssociationLinks": null,
  "serviceEndpointPolicies": null,
  "serviceEndpoints": null,
  "type": "Microsoft.Network/virtualNetworks/subnets"
}

az network vnet subnet create \
    --resource-group $vnet_rg \
    --vnet-name $vnet \
    --name $subnet \
    --address-prefixes 10.40.1.0/24
# [setup/network/network.azcli:32]: main(): az network vnet subnet create --resource-group {REDACTED}-westeurope-team-rg --vnet-name {REDACTED}-westeurope-team --name {REDACTED}-westeurope-team-aks-subnet --address-prefixes 10.40.1.0/24
{
  "addressPrefix": "10.40.1.0/24",
  "addressPrefixes": null,
  "delegations": [],
  "etag": "W/\"602169fe-651e-443a-94fd-eae8ca93c4e2\"",
  "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.Network/virtualNetworks/{REDACTED}-westeurope-team/subnets/{REDACTED}-westeurope-team-aks-subnet",
  "ipConfigurationProfiles": null,
  "ipConfigurations": null,
  "name": "{REDACTED}-westeurope-team-aks-subnet",
  "natGateway": null,
  "networkSecurityGroup": null,
  "privateEndpoints": null,
  "provisioningState": "Succeeded",
  "purpose": null,
  "resourceGroup": "{REDACTED}-westeurope-team-rg",
  "resourceNavigationLinks": null,
  "routeTable": null,
  "serviceAssociationLinks": null,
  "serviceEndpointPolicies": null,
  "serviceEndpoints": null,
  "type": "Microsoft.Network/virtualNetworks/subnets"
}

echo "--- Network setup stop"
# [setup/network/network.azcli:38]: main(): echo '--- Network setup stop'
--- Network setup stop
"${FILE_DIR}/keyvault/keyvault.azcli"
# [setup/init_env.sh:12]: main(): setup/keyvault/keyvault.azcli
set -o xtrace   # Show commands being executed through debugprint during execution

echo -e "\n--- Keyvault setup start"
# [setup/keyvault/keyvault.azcli:6]: main(): echo -e '\n--- Keyvault setup start'

--- Keyvault setup start

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/../inputs.sh"
fi
# [setup/keyvault/keyvault.azcli:8]: main(): [[ -v {REDACTED} ]]
# [setup/keyvault/keyvault.azcli:8]: main(): [[ -z {REDACTED} ]]
set -o nounset  # Throw error on first variable-reference to an unset variable
# [setup/keyvault/keyvault.azcli:11]: main(): set -o nounset

if ! az group exists --name "${kv_rg}" --subscription "${subscription_id}"; then
    az group create \
        --location $location \
        --name $kv_rg \
        --subscription $subscription_id \
        --verbose
fi
# [setup/keyvault/keyvault.azcli:13]: main(): az group exists --name {REDACTED}-westeurope-team-rg --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227
true

if ! az keyvault show --name "${kv_name}"; then
    # Create development key vault
    az keyvault create \
        --name "${kv_name}" \
        --location "${location}" \
        --resource-group "${kv_rg}" \
        --subscription "${subscription_id}" \
        --verbose
fi
# [setup/keyvault/keyvault.azcli:21]: main(): az keyvault show --name {REDACTED}-westeurope-team
{
  "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.KeyVault/vaults/{REDACTED}-westeurope-team",
  "location": "westeurope",
  "name": "{REDACTED}-westeurope-team",
  "properties": {
    "accessPolicies": [
      {
        "applicationId": null,
        "objectId": "694534a6-1c88-4bce-9c7e-5b8a77005356",
        "permissions": {
          "certificates": [
            "get",
            "list",
            "delete",
            "create",
            "import",
            "update",
            "managecontacts",
            "getissuers",
            "listissuers",
            "setissuers",
            "deleteissuers",
            "manageissuers",
            "recover"
          ],
          "keys": [
            "get",
            "create",
            "delete",
            "list",
            "update",
            "import",
            "backup",
            "restore",
            "recover"
          ],
          "secrets": [
            "get",
            "list",
            "set",
            "delete",
            "backup",
            "restore",
            "recover"
          ],
          "storage": [
            "get",
            "list",
            "delete",
            "set",
            "update",
            "regeneratekey",
            "setsas",
            "listsas",
            "getsas",
            "deletesas"
          ]
        },
        "tenantId": "0abe8783-2c3e-4c42-9848-54e419bcdeb0"
      }
    ],
    "createMode": null,
    "enablePurgeProtection": null,
    "enableSoftDelete": null,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": null,
    "enabledForTemplateDeployment": null,
    "networkAcls": null,
    "provisioningState": "Succeeded",
    "sku": {
      "name": "standard"
    },
    "tenantId": "0abe8783-2c3e-4c42-9848-54e419bcdeb0",
    "vaultUri": "https://{REDACTED}-westeurope-team.vault.azure.net/"
  },
  "resourceGroup": "{REDACTED}-westeurope-team-rg",
  "tags": {},
  "type": "Microsoft.KeyVault/vaults"
}

echo "--- Keyvault setup stop"
# [setup/keyvault/keyvault.azcli:31]: main(): echo '--- Keyvault setup stop'
--- Keyvault setup stop
"${FILE_DIR}/aks/providers.azcli"
# [setup/init_env.sh:13]: main(): setup/aks/providers.azcli
set -o xtrace   # Show commands being executed through debugprint during execution

echo -e "\n--- Providers setup start"
# [setup/aks/providers.azcli:6]: main(): echo -e '\n--- Providers setup start'

--- Providers setup start

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/../inputs.sh"
fi
# [setup/aks/providers.azcli:8]: main(): [[ -v {REDACTED} ]]
# [setup/aks/providers.azcli:8]: main(): [[ -z {REDACTED} ]]
set -o nounset  # Throw error on first variable-reference to an unset variable
# [setup/aks/providers.azcli:11]: main(): set -o nounset

az provider register \
    --namespace Microsoft.ContainerService \
    --subscription $subscription_id
# [setup/aks/providers.azcli:13]: main(): az provider register --namespace Microsoft.ContainerService --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227

echo "--- Providers setup stop"
# [setup/aks/providers.azcli:17]: main(): echo '--- Providers setup stop'
--- Providers setup stop
"${FILE_DIR}/aks/k8s.azcli"
# [setup/init_env.sh:14]: main(): setup/aks/k8s.azcli
set -o xtrace   # Show commands being executed through debugprint during execution

echo -e "\n--- Kubernetes setup start"
# [setup/aks/k8s.azcli:6]: main(): echo -e '\n--- Kubernetes setup start'

--- Kubernetes setup start

if [[ -v $SERVICE_PRINCIPAL_NAME ]] || [[ -z "${SERVICE_PRINCIPAL_NAME}" ]]; then
    source "${FILE_DIR}/../inputs.sh"
fi
# [setup/aks/k8s.azcli:8]: main(): [[ -v {REDACTED} ]]
# [setup/aks/k8s.azcli:8]: main(): [[ -z {REDACTED} ]]
set -o nounset  # Throw error on first variable-reference to an unset variable
# [setup/aks/k8s.azcli:11]: main(): set -o nounset

# Get k8s rbac credentials
if ! az keyvault secret list --vault-name "${kv_name}" | \
   jq -r '.[] | .id' | \
   awk -F '/' '{print $NF}' | \
   grep -q "${k8s_rbac_keyname}-id"; then
    # If secret does not exist in keyvault, create it
    az keyvault secret set \
    --vault-name "${kv_name}" \
    --name "${k8s_rbac_keyname}-id" \
    --value "$(pwgen --ambiguous --secure 16 1)"
fi
# [setup/aks/k8s.azcli:14]: main(): az keyvault secret list --vault-name {REDACTED}-westeurope-team
# [setup/aks/k8s.azcli:15]: main(): jq -r '.[] | .id'
# [setup/aks/k8s.azcli:16]: main(): awk -F / '{print $NF}'
# [setup/aks/k8s.azcli:17]: main(): grep -q k8s-rbac-client-id
echo -e "\tGet client ID from keyvault"
# [setup/aks/k8s.azcli:24]: main(): echo -e '\tGet client ID from keyvault'
    Get client ID from keyvault
client_id=$(az keyvault secret show \
    --vault-name $kv_name \
    --name "${k8s_rbac_keyname}-id" \
    --query "value" \
    --output tsv)
## [setup/aks/k8s.azcli:29]: main(): az keyvault secret show --vault-name {REDACTED}-westeurope-team --name k8s-rbac-client-id --query value --output tsv
# [setup/aks/k8s.azcli:29]: main(): client_id=kRuebd7khTr39JUq
[ -z "${client_id}" ] && exit 1
# [setup/aks/k8s.azcli:30]: main(): '[' -z kRuebd7khTr39JUq ']'

if ! az keyvault secret list --vault-name "${kv_name}" | \
   jq -r '.[] | .id' | \
   awk -F '/' '{print $NF}' | \
   grep -q "${k8s_rbac_keyname}-secret"; then
    # If secret does not exist in keyvault, create it
    client_id=
    az keyvault secret set \
    --vault-name "${kv_name}" \
    --name "${k8s_rbac_keyname}-secret" \
    --value "$(pwgen --ambiguous --secure 64 1)"
fi
# [setup/aks/k8s.azcli:32]: main(): az keyvault secret list --vault-name {REDACTED}-westeurope-team
# [setup/aks/k8s.azcli:33]: main(): jq -r '.[] | .id'
# [setup/aks/k8s.azcli:34]: main(): awk -F / '{print $NF}'
# [setup/aks/k8s.azcli:35]: main(): grep -q k8s-rbac-client-secret
echo -e "\tGet client secret from keyvault"
# [setup/aks/k8s.azcli:43]: main(): echo -e '\tGet client secret from keyvault'
    Get client secret from keyvault
client_secret=$(az keyvault secret show \
    --vault-name $kv_name \
    --name "${k8s_rbac_keyname}-secret" \
    --query "value" \
    --output tsv)
## [setup/aks/k8s.azcli:48]: main(): az keyvault secret show --vault-name {REDACTED}-westeurope-team --name k8s-rbac-client-secret --query value --output tsv
# [setup/aks/k8s.azcli:48]: main(): client_secret=EsuWzXCWKiuthT3ubrsfP3pxLMfn4feqrWm3hJyyYvNrsTFka9iMywYyHPtmUNdY
[ -z "${client_secret}" ] && exit 1
# [setup/aks/k8s.azcli:49]: main(): '[' -z EsuWzXCWKiuthT3ubrsfP3pxLMfn4feqrWm3hJyyYvNrsTFka9iMywYyHPtmUNdY ']'

# Get k8s eu west 1 subnet id
echo -e "\tGet subnet ID"
# [setup/aks/k8s.azcli:52]: main(): echo -e '\tGet subnet ID'
    Get subnet ID
k8s_subnet_id=$(az network vnet subnet show \
    --resource-group $vnet_rg \
    --vnet-name $vnet \
    --name $aks_subnet \
    --subscription $subscription_id \
    --query id \
    -o tsv)
## [setup/aks/k8s.azcli:59]: main(): az network vnet subnet show --resource-group {REDACTED}-westeurope-team-rg --vnet-name {REDACTED}-westeurope-team --name {REDACTED}-westeurope-team-aks-subnet --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227 --query id -o tsv
# [setup/aks/k8s.azcli:59]: main(): k8s_subnet_id=/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.Network/virtualNetworks/{REDACTED}-westeurope-team/subnets/{REDACTED}-westeurope-team-aks-subnet

# Get tenant id
echo -e "\tGet tenant ID"
# [setup/aks/k8s.azcli:62]: main(): echo -e '\tGet tenant ID'
    Get tenant ID
tenant_id=$(az account show \
    --query tenantId \
    -o tsv)
## [setup/aks/k8s.azcli:65]: main(): az account show --query tenantId -o tsv
# [setup/aks/k8s.azcli:65]: main(): tenant_id=0abe8783-2c3e-4c42-9848-54e419bcdeb0

# Set the current subscription because the explicit argument does NOT working properly
echo -e "\tGet current subscription ID"
# [setup/aks/k8s.azcli:68]: main(): echo -e '\tGet current subscription ID'
    Get current subscription ID
current_subscription=$(az account show \
    --query id \
    -o tsv)
## [setup/aks/k8s.azcli:71]: main(): az account show --query id -o tsv
# [setup/aks/k8s.azcli:71]: main(): current_subscription=15ea10b3-c25a-4569-b4c2-46054c6f5227
echo -e "\tSet subscription to current"
# [setup/aks/k8s.azcli:72]: main(): echo -e '\tSet subscription to current'
    Set subscription to current
az account set \
    --subscription $subscription_id
# [setup/aks/k8s.azcli:73]: main(): az account set --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227

# Create the k8s
echo -e "\tCreate resource group $aks_rg"
# [setup/aks/k8s.azcli:77]: main(): echo -e '\tCreate resource group {REDACTED}-westeurope-team-rg'
    Create resource group {REDACTED}-westeurope-team-rg
az group create \
    --location $location \
    --name $aks_rg \
    --subscription $subscription_id
# [setup/aks/k8s.azcli:78]: main(): az group create --location westeurope --name {REDACTED}-westeurope-team-rg --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227
{
  "id": "/subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg",
  "location": "westeurope",
  "managedBy": null,
  "name": "{REDACTED}-westeurope-team-rg",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": null
}

# NOTE: Make sure the subscription limits are greater than 10 cores
# NOTE: The nodepool-name does not meet the naming convention because the limit of characters
echo -e "\tCreate Kubernetes cluster (this will take a while)"
# [setup/aks/k8s.azcli:85]: main(): echo -e '\tCreate Kubernetes cluster (this will take a while)'
    Create Kubernetes cluster (this will take a while)
az aks create \
    --resource-group $aks_rg \
    --name $aks_name \
    --node-count 3 \
    --generate-ssh-keys \
    --network-plugin kubenet \
    --service-principal $client_id \
    --client-secret $client_secret \
    --service-cidr 192.168.0.0/17 \
    --dns-service-ip 192.168.0.2 \
    --pod-cidr 192.168.128.0/17 \
    --docker-bridge-address 172.17.0.1/16 \
    --vnet-subnet-id $k8s_subnet_id \
    --kubernetes-version 1.13.5 \
    --node-osdisk-size 200 \
    --node-vm-size Standard_DS3_v2 \
    --admin-username kadmin \
    --nodepool-name euwest \
    --subscription $subscription_id \
    --enable-addons monitoring \
    --verbose
# [setup/aks/k8s.azcli:86]: main(): az aks create --resource-group {REDACTED}-westeurope-team-rg --name {REDACTED}-westeurope-team-aks --node-count 3 --generate-ssh-keys --network-plugin kubenet --service-principal kRuebd7khTr39JUq --client-secret EsuWzXCWKiuthT3ubrsfP3pxLMfn4feqrWm3hJyyYvNrsTFka9iMywYyHPtmUNdY --service-cidr 192.168.0.0/17 --dns-service-ip 192.168.0.2 --pod-cidr 192.168.128.0/17 --docker-bridge-address 172.17.0.1/16 --vnet-subnet-id /subscriptions/15ea10b3-c25a-4569-b4c2-46054c6f5227/resourceGroups/{REDACTED}-westeurope-team-rg/providers/Microsoft.Network/virtualNetworks/{REDACTED}-westeurope-team/subnets/{REDACTED}-westeurope-team-aks-subnet --kubernetes-version 1.13.5 --node-osdisk-size 200 --node-vm-size Standard_DS3_v2 --admin-username kadmin --nodepool-name euwest --subscription 15ea10b3-c25a-4569-b4c2-46054c6f5227 --enable-addons monitoring --verbose
Use existing SSH public key file: /home/user/.ssh/id_rsa.pub
Waiting for AAD role to propagate
Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[###                                 ]  10.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[#######                             ]  20.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[##########                          ]  30.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[##############                      ]  40.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[##################                  ]  50.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[#####################               ]  60.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[#########################           ]  70.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[############################        ]  80.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Waiting for AAD role to propagate[################################    ]  90.0000%Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://graph.windows.net/0abe8783-2c3e-4c42-9848-54e419bcdeb0/getObjectsByObjectIds?api-version=1.6
Could not create a role assignment for subnet. Are you an Owner on this subscription?
Operation failed with status: 'Bad Request'. Details: Internal server error
command ran in 124.497 seconds.
[2019-08-08 13:16:20] 1 user@xmachine:~/Documents/team-k8s-project $

Expected Behavior

For the az aks create command not to fail, so that the k8s cluster could be created and the script(s) could continue.

Environment Summary

Linux-4.15.0-55-generic-x86_64-with-debian-buster-sid
Python 3.6.5
Shell: bash

azure-cli 2.0.70

Additional Context


AKS Service Attention customer-reported
Source
picture x10an14
👍1

Most helpful comment

I've also come across this and found that many of the az aks commands use the Azure Active Directory Graph API which can cause this issue. In my case, the Azure CLI commands appear to be performing a lookup of the "Object ID" for the Service Principal being used to run the command before validating a Role Assignment against the dependent resource.

I found multiple situations where I got a warning or error similar to the following:

Waiting for AAD role to propagate[################################ ] 90.0000%Could not create a role assignment for {{ resource_name }} . Are you an Owner on this subscription?

This doesn't always cause the job to fail. For example, when creating a new AKS Cluster attached to an existing Subnet which has the appropriate Role Assignments, this will be treated as a warning and the command will continue to run. By contrast, trying to connect the AKS Cluster to an Azure Container Registry using az aks update ... --attach-acr will fail.

Looking at the debug output, I was getting an authentication error (HTTP 403) when trying to connect to:

https://graph.windows.net:443

To resolve this, I needed to grant permission to the Service Principal for the permission:

https://graph.windows.net/Application.ReadWrite.OwnedBy

This is in addition to assigning the Owner role on the Subscription, or ensuring the Service Principal has suitable Role Assignments on each dependent resource.

Unfortunately granting permissions to these APIs isn't always possible due to internal security policies, so it would be good to find an alternative way to handle this in the Azure CLI but hopefully this piece of information will help anyone else who comes across this.

picture krowlandson on 28 May 2020
👍3

All 9 comments

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @mjconnection

msftbot[bot] picture msftbot[bot] on 13 Aug 2019

I am getting the same error trying to update an existing cluster to attach an ACR:

az aks update --name nathan1 --resource-group nathan --attach-acr /subscriptions/<id>/resourceGroups/nathan/providers/Microsoft.ContainerRegistry/registries/nathandfdc

generates the error

Waiting for AAD role to propagate[################################    ]  90.0000%Could not create a role assignment for ACR. Are you an Owner on this subscription?

I have the "global admin" role.

griffinator76 picture griffinator76 on 29 Oct 2019

Hi @griffinator76 ,

Global Admin is an AAD role not a subscription level role. Owner is a subscription role and assigned via the subscription blade:

Subscriptions -> Select SubscriptionID-> Access Control (IAM) -> Role assignments screen.

Alternatively you can provide Global Admins with the ability to assign subscription level permissions, how to do that is detailed on this Microsoft page (Access Management for Azure resources is what you want).

Hope that helps.

TitaniumDave picture TitaniumDave on 2 Dec 2019

I'm facing the same problem with the Owner role

GersonDias picture GersonDias on 13 Dec 2019

@GersonDias do you have a commands ran list and output?

palma21 picture palma21 on 16 Dec 2019

I've had the same problem come up recently.
i tried to create a AKS-cluster with this command:
az aks create --resource-group AKS-Group --name gp-demo-cluster --node-count 1 --generate-ssh-keys --attach-acr duala --kubernetes-version 1.17.0 --location westeurope --node-vm-size Standard_B2ms

and i got

Waiting for AAD role to propagate[################################ ] 90.0000%Could not create a role assignment for ACR. Are you an Owner on this subscription?

I am Service Administrator for this development subcription so this should be working. Even after i gave me the owner role is got this error message.

image

image

In the past i have successfully created aks cluster with "-attach-acr" on this account.

I'm not sure what's wrong.

i tried before to create this aks-cluster with the installed aks-preview extensions where at the moment --attach-acr is broken (https://github.com/Azure/azure-cli-extensions/issues/1147), so i decided to remove the preview extension but now i got this error.

Herr-Sepp picture Herr-Sepp on 14 Jan 2020

i found the problem and the solution.
az aks was using an old serviceprincipal that i had delete some time ago.

to fix this you must delete the file "aksServicePrincipal.json" under

C:\Users\XXXXXX.azure\

after that the az aks create command run through without an error.
But the Error message make it difficult to find it.

Maybe you can check if the Service principal is valid before you try to assign a role to it?

Herr-Sepp picture Herr-Sepp on 14 Jan 2020
👍2

Thanks for chasing this @Herr-Sepp we'll take a look at that error message to see if we can clarify it.

palma21 picture palma21 on 17 Jan 2020

I've also come across this and found that many of the az aks commands use the Azure Active Directory Graph API which can cause this issue. In my case, the Azure CLI commands appear to be performing a lookup of the "Object ID" for the Service Principal being used to run the command before validating a Role Assignment against the dependent resource.

I found multiple situations where I got a warning or error similar to the following:

Waiting for AAD role to propagate[################################ ] 90.0000%Could not create a role assignment for {{ resource_name }} . Are you an Owner on this subscription?

This doesn't always cause the job to fail. For example, when creating a new AKS Cluster attached to an existing Subnet which has the appropriate Role Assignments, this will be treated as a warning and the command will continue to run. By contrast, trying to connect the AKS Cluster to an Azure Container Registry using az aks update ... --attach-acr will fail.

Looking at the debug output, I was getting an authentication error (HTTP 403) when trying to connect to:

https://graph.windows.net:443

To resolve this, I needed to grant permission to the Service Principal for the permission:

https://graph.windows.net/Application.ReadWrite.OwnedBy

This is in addition to assigning the Owner role on the Subscription, or ensuring the Service Principal has suitable Role Assignments on each dependent resource.

Unfortunately granting permissions to these APIs isn't always possible due to internal security policies, so it would be good to find an alternative way to handle this in the Azure CLI but hopefully this piece of information will help anyone else who comes across this.

krowlandson picture krowlandson on 28 May 2020
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cicorias picture cicorias  路  3Comments
mmacy picture mmacy  路  3Comments
Kannaj picture Kannaj  路  3Comments
ambakshi picture ambakshi  路  3Comments
jsturtevant picture jsturtevant  路  3Comments