Autoscaler: CA tries to add pods to master instead of scaling a new node

[pkelleratwork]

I'm expecting new nodes to scale up instead of adding pods to master. I want to avoid adding pods to master. I am getting reason: PodToleratesNodeTaints when it tries to add to master.

What am i missing to scale up nodes?

kops ing
```apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-05-04T20:31:55Z
labels:
kops.k8s.io/cluster: .k8s.local
name: nodes
spec:
image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08
machineType: t2.medium
maxSize: 9
minSize: 3
nodeLabels:
kops.k8s.io/instancegroup: nodes
role: Node
subnets:

• us-east-2a`

NAME STATUS ROLES AGE VERSION
ip-172-xx-xx-xxx.us-xx-xx.compute.internal Ready node 2d v1.9.3
ip-172-xx-xx-xxx.us-xx-xx.compute.internal Ready node 2d v1.9.3
ip-172-xx-xx-xxx.us-xx-xx.compute.internal Ready node 2d v1.9.3
ip-172-xx-xx-xxx.us-xx-xx.compute.internal Ready master 2d v1.9.3

I0507 16:25:08.160505 1 static_autoscaler.go:108] Starting main loop
I0507 16:25:08.312549 1 utils.go:436] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0507 16:25:08.312573 1 static_autoscaler.go:240] Filtering out schedulables
I0507 16:25:08.313112 1 static_autoscaler.go:250] No schedulable pods
I0507 16:25:08.313139 1 scale_up.go:54] Pod default/qafeature2-api-7d68478c9b-86s6z is unschedulable
I0507 16:25:08.313193 1 scale_up.go:54] Pod default/qafeature2-api-7d68478c9b-76g2s is unschedulable
I0507 16:25:08.313232 1 scale_up.go:54] Pod default/qafeature2-web-d478cd9b9-ssxjj is unschedulable
I0507 16:25:08.313245 1 scale_up.go:54] Pod default/qafeature2-web-d478cd9b9-dswvl is unschedulable
I0507 16:25:08.313249 1 scale_up.go:54] Pod default/qafeature2-bff-7f7489cf84-r4j2d is unschedulable
I0507 16:25:08.313253 1 scale_up.go:54] Pod default/qafeature2-bff-7f7489cf84-grgqm is unschedulable
I0507 16:25:08.313277 1 scale_up.go:54] Pod redis/qafeature2-redis-redis-6859d6646f-hk2v7 is unschedulable
I0507 16:25:08.341165 1 scale_up.go:86] Upcoming 0 nodes
I0507 16:25:08.360344 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-api-7d68478c9b-86s6z on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360370 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-api-7d68478c9b-76g2s on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360386 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-web-d478cd9b9-ssxjj on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360400 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-web-d478cd9b9-dswvl on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360419 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-bff-7f7489cf84-r4j2d on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360432 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put default/qafeature2-bff-7f7489cf84-grgqm on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360447 1 scale_up.go:146] Scale-up predicate failed: PodToleratesNodeTaints predicate mismatch, cannot put redis/qafeature2-redis-redis-6859d6646f-hk2v7 on template-node-for-master-us-east-2a.masters..k8s.local-9128099766993471710, reason: PodToleratesNodeTaints
I0507 16:25:08.360464 1 scale_up.go:175] No pod can fit to master-us-east-2a.masters..k8s.local
I0507 16:25:08.360477 1 scale_up.go:180] No expansion options
I0507 16:25:08.360547 1 static_autoscaler.go:299] Calculating unneeded nodes
I0507 16:25:08.361467 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-web-d478cd9b9-dswvl", UID:"d8401c7a-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476122", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.361516 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-bff-7f7489cf84-r4j2d", UID:"d8d3596d-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476139", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.361605 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-bff-7f7489cf84-grgqm", UID:"d8d43c86-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476140", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.361624 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"redis", Name:"qafeature2-redis-redis-6859d6646f-hk2v7", UID:"d4482e1c-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476078", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.361636 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-api-7d68478c9b-86s6z", UID:"d7a6002f-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476101", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.361647 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-api-7d68478c9b-76g2s", UID:"d7a8c070-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476103", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.362476 1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"qafeature2-web-d478cd9b9-ssxjj", UID:"d83e44fd-5212-11e8-b554-022b7a31eac4", APIVersion:"v1", ResourceVersion:"476119", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0507 16:25:08.391999 1 utils.go:408] Skipping ip-172-xx-xx-xx.xx-xx-xx.compute.internal - node group min size reached
I0507 16:25:08.392018 1 utils.go:399] Skipping ip-172-xx-xx-xx.xx-xx-xx.compute.internal - no node group config
I0507 16:25:08.392052 1 utils.go:399] Skipping ip-172-xx-xx-xx.xx-xx-xx.compute.internal - no node group config
I0507 16:25:08.392063 1 utils.go:399] Skipping ip-172-xx-xx-xx.xx-xx-xx.compute.internal - no node group config
I0507 16:25:08.393659 1 static_autoscaler.go:329] Scale down status: unneededOnly=false lastScaleUpTime=2018-05-05 14:53:03.343699447 +0000 UTC lastScaleDownDeleteTime=2018-05-05 14:53:03.343699997 +0000 UTC lastScaleDownFailTime=2018-05-05 14:53:03.343700358 +0000 UTC schedulablePodsPresent=false isDeleteInProgress=false
I0507 16:25:08.393684 1 static_autoscaler.go:332] Starting scale down
I0507 16:25:08.443351 1 scale_down.go:446] No candidates for scale down`

[bskiba]

The log line "Skipping ip-172-20-51-175.us-east-2.compute.internal - no node group config" tells you that there is a node group in a cluster that autoscaler sees but it doesn't know about so it won't touch it.

In general you need to tell Cluster Autoscaler which node groups you want to scale. How you do it depends on your cloud provider and the way you deploy Cluster Autoscaler.

[pkelleratwork]

@bskiba - thanks you for you quick reply

I created cluster
1) using kops on AWS
2) ran master node setup kubectl apply -f examples/cluster-autoscaler-run-on-master.yaml

Is this my mistake? Is that what is making it try to deploy to master node. I understood it as putting the cluster-autoscaler pod in master, not the application pods.

Should i have used kubectl apply -f examples/cluster-autoscaler-one-asg.yaml ?

[bskiba]

Looks like it. Note that you need to modify it here with your node group name.

[pkelleratwork]

awesome - i'll give it a shot now! thanks!

[pkelleratwork]

@bskiba IT WORKS!!!!!!! Thanks!!!!!