Autoscaler: [Azure] [AKS] Cluster not Scaling up with error "Node group agentpool is not ready for scaleup"

[abizake]

Failed to scale up: failed to increase node group size: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="PropertyChangeNotAllowed" Message="Changing property 'enableRBAC' is not allowed."

Additional info
There are currently 2 nodes in the cluster.
I created the Cluster Auto Scaler with Min=3 and Max =5
I also scaled the pods because of which there is one pods unscheduled.

Kubernetes Version : 1.10.5
Cluster Autoscaler version : V1.2.2
Cloud Provider :- Azure (AKS)

Please Advise.

[abizake]

Also tried the latest version of AKS(1.10.6) and Cluster Auto Scaler (1.3.1).
Still facing the issue.
logs :-
containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="" Message=""
W0801 16:02:23.327898 1 clusterstate.go:223] Disabling scale-up for node group agentpool until 2018-08-01 16:07:23.327890748 +0000 UTC m=+1003.285999979
E0801 16:02:23.327976 1 static_autoscaler.go:283] Failed to scale up: failed to increase node group size: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="" Message=""

[gcampanella]

I'm seeing the very same issue with Kubernetes 1.10.6 and Cluster Autoscaler 1.2.2 set up as per this guide:

Scale-up failed for group nodepool1: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="PropertyChangeNotAllowed" Message="Changing property 'enableRBAC' is not allowed."

Resource Explorer is showing "enableRBAC": true.

[iMartyn]

Also happening for me on 1.10.6 and 1.3.1. I've raised a support ticket with Azure to see if it's something their end.

[feiskyer]

There is an internal bug related with this issue. The fix should be released soon.

[iMartyn]

@feiskyer Can we get an idea of when "soon" is? Days? Weeks? Months? It would help to feed into our planning sessions, as this is an essential requirement for our setup. Is there an alternative image we can help test?

[gabrielmcf]

Waiting anxiously for this bug fix!

[feiskyer]

New CA releases with AKS API 2018-03-31: https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.2.3-beta.1 and https://github.com/kubernetes/autoscaler/releases/tag/cluster-autoscaler-1.3.2-beta.1.

@gabrielmcf @iMartyn @gcampanella @abizake Could you check whether new releases fix the issue on your cluster?

[iMartyn]

@feiskyer No go I'm afraid.

I0816 11:39:57.820992       1 scale_up.go:249] Pod default/nginx-56fd48c675-6lr47 is unschedulable
I0816 11:39:57.821000       1 scale_up.go:249] Pod default/nginx-56fd48c675-6nhgs is unschedulable
I0816 11:39:58.018025       1 scale_up.go:383] Best option to resize: nodepool1
I0816 11:39:58.018125       1 scale_up.go:387] Estimated 2 nodes needed in nodepool1
I0816 11:39:58.087642       1 scale_up.go:466] Final scale-up plan: [{nodepool1 3->5 (max: 10)}]
I0816 11:39:58.087822       1 scale_up.go:536] Scale-up: setting group nodepool1 size to 5
I0816 11:39:58.246026       1 azure_container_service_pool.go:304] Setting size for cluster ("mi-platform-qa-aks") with new count (5)
I0816 11:39:58.320713       1 azure_container_service_pool.go:184] Current size: 3, Target size requested: 5
E0816 11:39:58.321092       1 azure_container_service_pool.go:193] Failed to update AKS cluster (%q): %vmi-platform-qa-akscontainerservice.ManagedClustersClient#CreateOrUpdate: Invalid input: autorest/validation: validation failed: parameter=parameters.ManagedClusterProperties.AadProfile.ServerAppSecret constraint=Null value=(*string)(nil) details: value can not be null; required parameter
W0816 11:39:58.321151       1 clusterstate.go:223] Disabling scale-up for node group nodepool1 until 2018-08-16 11:44:58.321145962 +0000 UTC m=+596.575624501
E0816 11:39:58.321196       1 static_autoscaler.go:283] Failed to scale up: failed to increase node group size: containerservice.ManagedClustersClient#CreateOrUpdate: Invalid input: autorest/validation: validation failed: parameter=parameters.ManagedClusterProperties.AadProfile.ServerAppSecret constraint=Null value=(*string)(nil) details: value can not be null; required parameter

The ClientSecret is set in the configmap as per these instructions : https://docs.microsoft.com/en-us/azure/aks/autoscaler so I think this is still a bug :(

[feiskyer]

@iMartyn Thanks for feedback, let me have a check

[feiskyer]

@iMartyn Could you send the error logs together with (subscription ID, resource group, aks cluster name) to [email protected]? We need to dive into what's wrong.

[iMartyn]

I'd deleted that cluster, I'm recreating it now for you to capture the error.

[iMartyn]

So now I have no idea as creating a new cluster with aks on k8s 1.11.1 and cas 1.3.2-beta.1 worked.

I have mixed feelings about this because truly I was still seeing the above error but because of all the EUWest problems yesterday, we destroyed the test cluster.

[feiskyer]

@iMartyn Thanks.

[gabrielmcf]

Turns out it was a permission problem with our user that was fixed by us! Sorry for the anxiety but it seemed at the time a bug!

[feiskyer]

@gabrielmcf Thanks for the updates.

@abizake Have you checked whether the new releases solved your issue?

[feiskyer]

Fixed in the https://github.com/kubernetes/autoscaler/pull/1416.

[feiskyer]

/close

[k8s-ci-robot]

@feiskyer: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.