Azure-cli: Change max pods per node on cli

[GustavoAmerico]

Change max pods per node on cli

Problem

i'm facing problem in my AKS, as i found max pods number is 30 per node, when i checked the deployed pods I found +-10 pods are related to AKS system itself, i think max pods number should not include AKS system pods as this reduced my limit to 20 not 30 as mentioned in MS doc.

Expectative:

run az aks nodepool update --cluster-name <you aks cluster name> --name <pool name> --max-pods=$maxpodsize (currently it doesn't work)

Workaround:

1. I created an new node pool
   az aks nodepool add --cluster-name <you aks cluster name> -g <your-aks-resource-group> --max-pods $maxpodsize --name newpool --enable-cluster-autoscaler --min-count 1 --max-count 1 -c 1 -s <machine size>

2. Scale down the node of agentpool one per one and scale up newpool as necessary

Workaround problem:

I tried to remove the default pool, but the cli show an error

The message is: Operation failed with status: 'Bad Request'. Details: There has to be at least one system agent pool.

Workaround Expectative:

Cleaning the default pool if other pool was work

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm.

[yonzhan]

aks

[yungezz]

routing to appropriate team

[jluk]

This is by design, the pods per node have to represent the actual count of pods viewed from k8s. The system pod count is dependent on the cluster config, if you add multiple addons the "system-pods" will increase and while these pods are managed by the AKS service, they still reside in user-space so you can debug and transparently view them as needed.

It is also by design that there must be 1 system pool, you can delete the whole cluster if you need all the system pools removed.

Closing as a result that this is by design, but comment if this doesnt' clarify and we can revisit.

[GustavoAmerico]

I understand, displaying the system nodes in the total of pods makes sense, but, I did not understand how you determine that the maximum pod in the node is 30. A cluster with nodes DS12-V2 only run 30 pods equals the cluster with B2MS?

This problem should not be closed without action, I reported 2 problems with the CLI.
Problems:

1. az aks nodepool update --max-pods not working

1. It is not possible to delete 1 pool, even if there is another one.

[jluk]

The max pod setting per node is user defined and done so at create time only for each agent pool. There is no update functionality for it as seen from the possible commands on update:

    az aks nodepool update : Update a node pool to enable/disable cluster-autoscaler or change min-
    count or max-count.

Arguments
    --cluster-name       [Required] : The cluster name.
    --name -n            [Required] : The node pool name.
    --resource-group -g  [Required] : Name of resource group. You can configure the default group
                                      using `az configure --defaults group=<name>`.
    --disable-cluster-autoscaler -d : Disable cluster autoscaler.
    --enable-cluster-autoscaler -e  : Enable cluster autoscaler.
    --max-count                     : Maximum nodes count used for autoscaler, when "--enable-
                                      cluster-autoscaler" specified. Please specify the value in the
                                      range of [1, 100].
    --min-count                     : Minimum nodes count used for autoscaler, when "--enable-
                                      cluster-autoscaler" specified. Please specify the value in the
                                      range of [1, 100].
    --mode                          : The mode for a node pool which defines a node pool's primary
                                      function. If set as "System", AKS prefers system pods
                                      scheduling to node pools with mode `System`. Learn more at
                                      https://aka.ms/aks/nodepool/mode.  Allowed values: System,
                                      User.
    --no-wait                       : Do not wait for the long-running operation to finish.
    --tags                          : Space-separated tags: key[=value] [key[=value] ...]. Use '' to
                                      clear existing tags.
    --update-cluster-autoscaler -u  : Update min-count or max-count for cluster autoscaler.

Clusters require at least 1 system pool, as returned in err msg. This is detailed here:
https://docs.microsoft.com/en-us/azure/aks/use-system-pools

[pvorb]

But why can't you update the maximum number of pods for an existing nodepool? Could this be added to the cli, please?

[palma21]

Because that would involve re-wiring the pod CIDR which require a node reboot across the node pool at which point it makes the scenario to create a new pool and delete the previous one a bit more of controlled one.

Happy to evaluate your scenario is it's a major pain for you to do this.

[pvorb]

Okay, I fully understand. My original problem is that the AKS collection for Ansible doesn't support specifying --max-pods upon cluster creation: ansible-collections/azure#308.

So now I'm trying to build a workaround:

1. set up cluster with system nodepool via ansible
2. increase --max-pods for the existing cluster via az cli

But now I think I should wait for an action on the other issue.