Aks-engine: Azure aad not working with 1.17+

[damyan90]

Describe the bug
Created AAD applications according the doc: https://docs.microsoft.com/en-us/azure/aks/azure-ad-integration-cli (up until AKS related steps)
Then added the following in the aks-engine deployment:

"aadProfile": {
    "serverAppID": "",
    "clientAppID": "",
    "adminGroupID": "";
    "tenantID": ""
}

Then when I try to use the kubeconfig I get:

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code xxxxxxx to authenticate.
error: You must be logged in to the server (Unauthorized)

Even though the login to MS app is successful. I can see "Success" entry in the "Sign-ins" section of the app in Azure.

In the logs of kube-apiserver I can see
authentication.go:53] Unable to authenticate the request due to an error: [invalid bearer token, oidc: authenticator not initialized]

Those are the OIDC related settings set on kube-apiserver:
- --oidc-client-id=spn:54da1e36-089b-4c78-91b8-a239399305e3 - --oidc-groups-claim=groups - --oidc-issuer-url=https://sts.windows.net/${tenantID}/ - --oidc-username-claim=oid
ClusterRoleBinding is also created and looks fine.
Steps To Reproduce

{
  "apiVersion": "vlabs",
  "location": "northeurope",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.18.4",
      "kubernetesConfig": {
        "enableRbac": true,
        "privateCluster": {
          "enabled": true
        },
        "etcdDiskSizeGB": "128",
        "networkPolicy": "none",
        "dnsServiceIP": "10.241.0.10",
        "serviceCidr": "10.241.0.0/16",
        "clusterSubnet": "10.240.0.0/16",
        "loadBalancerSku": "Standard",
        "excludeMasterFromStandardLB": true,
        "addons": [
          {
            "name": "cluster-autoscaler",
            "enabled": true,
            "pools": [
              {
                "name": "pool1",
                "config": {
                  "min-nodes": "1",
                  "max-nodes": "10"
                }
              }
            ]
          }
        ]
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "prefix-k8s184",
      "osDiskSizeGB": 30,
      "vmSize": "Standard_D2s_v3",
      "vnetSubnetId": "xxxxxx",
      "firstConsecutiveStaticIP": "xxxxxx",
      "vnetCidr": "10.240.0.0/15"
    },
    "agentPoolProfiles": [
      {
        "name": "pool1",
        "count": 1,
        "osDiskSizeGB": 30,
        "vmSize": "Standard_F4s_v2",
        "vnetSubnetId": "xxxxxx",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vmssOverProvisioningEnabled": true,
        "enableVMSSNodePublicIP": false
      }
    ],
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    },
    "aadProfile": {
      "serverAppID": "1234",
      "clientAppID": "1234",
      "tenantID": "1234",
      "adminGroupID": "1234"
    }
  }
}

Expected behavior
I get authorized and can access the cluster.
AKS Engine version
v0.53.0
Kubernetes version
v1.18.4
Additional context
Works fine with Kubernetes 1.16.11

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[chreichert]

Same here on a cluster, upgraded from 1.16.11 to 1.17.7 with AKS-Engine 0.53.0:

Kubectl access with AAD on 1.16.11 worked fine. After Upgrade to 1.17.7 it fails with: error: You must be logged in to the server (Unauthorized). Kubectl version 1.18.6.

Same error in apiserver log as above:
E0724 07:56:56.505824 1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, oidc: authenticator not initialized]
I0724 07:56:56.505972

Additionally:
E0724 07:56:59.926776 1 oidc.go:232] oidc authenticator: initializing plugin: Get https://sts.windows.net/**/.well-known/openid-configuration: x509: certificate signed by unknown authority

It even happened during the upgrade, had to switch to use local kubeconfig with certificate authentication.

[jackfrancis]

@chreichert @damyan90 are you able to build from this PR to see if mounting the certs directory is all we need to fix?:

https://github.com/Azure/aks-engine/pull/3782

[chreichert]

@jackfrancis Sorry, I am not able to build AKS-Engine binaries.
I could test, if someone can provide me with a linux-amd64 binary.

[chewong]

@chreichert I helped you build a binary based on #3782 to test - aks-engine-d02d8be6d5-linux-amd64.tar.gz

[chreichert]

@jackfrancis Just tested an upgrade from 1.16.14 to 1.17.11 with this PR version. The error is still occuring.

Found the following in the apiserver log:

E0904 07:48:52.545225       1 oidc.go:232] oidc authenticator: initializing plugin: Get https://sts.windows.net/ecaa386b-c8df-4ce0-ad01-740cbdb5ba55/.well-known/openid-configuration: x509: certificate signed by unknown authority

and:

E0904 07:50:29.077747       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, oidc: authenticator not initialized]

thanks @chewong for providing the binary.

[msaurabhee]

I tested with AKS engine 0.54.1 with Kubernetes 1.18.6 Issue exists.

[msaurabhee]

Any way to bypass this manually?

[jackfrancis]

Can anyone still unable to solve this try with this new release of AKS Engine:

https://github.com/Azure/aks-engine/releases/tag/v0.55.2

The above includes a change that mounts the k8s certs directory in the apiserver container.

[chreichert]

@jackfrancis I just tested a clean install of k8s 1.17.11 with AKS-Engine v0.55.3. The problem is still occuring, unless the error in the Apiserver log is now different:

E0909 11:17:17.078363       1 authentication.go:104] Unable to authenticate the request due to an error: [invalid bearer token, oidc: verify token: oidc: expected audience "spn:29******-07**-4d**-b9**-21**********" got ["29******-07**-4d**-b9**-21**********"]]

[chreichert]

@jackfrancis Got it working now. I cleaned up my kubectl config and did a clean login, which fixed it for me now.
Thanks again for your help.

[jackfrancis]

3800 was the ultimate fix. This release has it:

https://github.com/Azure/aks-engine/releases/tag/v0.55.3

[damyan90]

The AAD integration works for me too. Thanks!