Autoscaler: [AKS] [Autoscaler]Failed to find readiness information for agentpool / segmentation violation

Created on 24 Sep 2018  路  17Comments  路  Source: kubernetes/autoscaler

Hello everyone,

Cluster autoscaler service seems to have some problem to get nodegroup health status however our cluster works fine.

Pod scheduling works fine but when I exceed cluster limit autoscaler don't scale the cluster an the pod panic.

Here is more information:

Kubectl get node:
'''
_NAME STATUS ROLES AGE VERSION LABELS
aks-agentpool-33865844-0 Ready agent 14d v1.11.2 agentpool=agentpool,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/insta nce-type=Standard_D16s_v3,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=northeurope,failure-domain.beta.kubernetes.io/ zone=0,kubernetes.azure.com/cluster=MC_RG-APPLI-HPROD_AKS-LHA-HPROD02_northeurope,kubernetes.io/hostname=aks-agentpool-33865844-0, kubernetes.io/role=agent,storageprofile=managed,storagetier=Premium_LRS
aks-agentpool-33865844-1 Ready agent 14d v1.11.2 agentpool=agentpool,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/insta nce-type=Standard_D16s_v3,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=northeurope,failure-domain.beta.kubernetes.io/ zone=1,kubernetes.azure.com/cluster=MC_RG-APPLI-HPROD_AKS-LHA-HPROD02_northeurope,kubernetes.io/hostname=aks-agentpool-33865844-1, kubernetes.io/role=agent,storageprofile=managed,storagetier=Premium_LRS
'''_

Kubernetes 1.11.2
Cluster-autoscaler: 1.3.3

cluster-autoscaler-status
_'''
kubectl -n kube-system describe configmaps cluster-autoscaler-status
...
Cluster-autoscaler status at 2018-09-24 11:24:47.973812405 +0000 UTC:
Cluster-wide:
Health: Healthy (ready=6 unready=0 notStarted=0 longNotStarted=0 registered=6 longUnregistered=0)
LastProbeTime: 2018-09-24 11:24:47.436701137 +0000 UTC m=+5227.476091680
LastTransitionTime: 2018-09-24 09:58:13.148928229 +0000 UTC m=+33.188318872
ScaleUp: NoActivity (ready=6 registered=6)
LastProbeTime: 2018-09-24 11:24:47.436701137 +0000 UTC m=+5227.476091680
LastTransitionTime: 2018-09-24 09:58:13.148928229 +0000 UTC m=+33.188318872
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2018-09-24 11:24:47.436701137 +0000 UTC m=+5227.476091680
LastTransitionTime: 2018-09-24 09:58:13.148928229 +0000 UTC m=+33.188318872

NodeGroups:
Name: agentpool
Health: Unhealthy (ready=0 unready=0 notStarted=0 longNotStarted=0 registered=0 longUnregistered=0 cloudProviderTarget=6 (minSize=5, maxSize=10))
LastProbeTime: 0001-01-01 00:00:00 +0000 UTC
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
ScaleUp: Backoff (ready=0 cloudProviderTarget=6)
LastProbeTime: 0001-01-01 00:00:00 +0000 UTC
LastTransitionTime: 0001-01-01 00:00:00 +0000 UTC
ScaleDown: NoCandidates (candidates=0)
LastProbeTime: 2018-09-24 11:24:47.436701137 +0000 UTC m=+5227.476091680
LastTransitionTime: 2018-09-24 09:58:13.148928229 +0000 UTC m=+33.188318872
'''_

begining of autoscaler startup logs
'''
_I0924 09:57:41.508847 1 flags.go:52] FLAG: --address=":8085"
I0924 09:57:41.508897 1 flags.go:52] FLAG: --alsologtostderr="false"
I0924 09:57:41.508904 1 flags.go:52] FLAG: --application-metrics-count-limit="100"
I0924 09:57:41.508912 1 flags.go:52] FLAG: --azure-container-registry-config=""
I0924 09:57:41.508918 1 flags.go:52] FLAG: --balance-similar-node-groups="false"
I0924 09:57:41.508922 1 flags.go:52] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I0924 09:57:41.508997 1 flags.go:52] FLAG: --cloud-config=""
I0924 09:57:41.509005 1 flags.go:52] FLAG: --cloud-provider="azure"
I0924 09:57:41.509011 1 flags.go:52] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0924 09:57:41.509022 1 flags.go:52] FLAG: --cluster-name=""
I0924 09:57:41.509044 1 flags.go:52] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I0924 09:57:41.509051 1 flags.go:52] FLAG: --containerd="unix:///var/run/containerd.sock"
I0924 09:57:41.509067 1 flags.go:52] FLAG: --cores-total="0:320000"
I0924 09:57:41.509073 1 flags.go:52] FLAG: --docker="unix:///var/run/docker.sock"
I0924 09:57:41.509104 1 flags.go:52] FLAG: --docker-env-metadata-whitelist=""
I0924 09:57:41.509110 1 flags.go:52] FLAG: --docker-only="false"
I0924 09:57:41.509125 1 flags.go:52] FLAG: --docker-root="/var/lib/docker"
I0924 09:57:41.509131 1 flags.go:52] FLAG: --docker-tls="false"
I0924 09:57:41.509137 1 flags.go:52] FLAG: --docker-tls-ca="ca.pem"
I0924 09:57:41.509157 1 flags.go:52] FLAG: --docker-tls-cert="cert.pem"
I0924 09:57:41.509164 1 flags.go:52] FLAG: --docker-tls-key="key.pem"
I0924 09:57:41.509169 1 flags.go:52] FLAG: --enable-load-reader="false"
I0924 09:57:41.509175 1 flags.go:52] FLAG: --estimator="binpacking"
I0924 09:57:41.509197 1 flags.go:52] FLAG: --event-storage-age-limit="default=0"
I0924 09:57:41.509204 1 flags.go:52] FLAG: --event-storage-event-limit="default=0"
I0924 09:57:41.509209 1 flags.go:52] FLAG: --expander="random"
I0924 09:57:41.509215 1 flags.go:52] FLAG: --expendable-pods-priority-cutoff="0"
I0924 09:57:41.509237 1 flags.go:52] FLAG: --gke-api-endpoint=""
I0924 09:57:41.509243 1 flags.go:52] FLAG: --global-housekeeping-interval="1m0s"
I0924 09:57:41.509264 1 flags.go:52] FLAG: --google-json-key=""
I0924 09:57:41.509270 1 flags.go:52] FLAG: --gpu-total="[]"
I0924 09:57:41.509292 1 flags.go:52] FLAG: --housekeeping-interval="10s"
I0924 09:57:41.509298 1 flags.go:52] FLAG: --httptest.serve=""
I0924 09:57:41.509304 1 flags.go:52] FLAG: --kubeconfig=""
I0924 09:57:41.509327 1 flags.go:52] FLAG: --kubernetes=""
I0924 09:57:41.509344 1 flags.go:52] FLAG: --leader-elect="true"
I0924 09:57:41.509352 1 flags.go:52] FLAG: --leader-elect-lease-duration="15s"
I0924 09:57:41.509374 1 flags.go:52] FLAG: --leader-elect-renew-deadline="10s"
I0924 09:57:41.509394 1 flags.go:52] FLAG: --leader-elect-resource-lock="endpoints"
I0924 09:57:41.509399 1 flags.go:52] FLAG: --leader-elect-retry-period="2s"
I0924 09:57:41.509405 1 flags.go:52] FLAG: --log-backtrace-at=":0"
I0924 09:57:41.509425 1 flags.go:52] FLAG: --log-cadvisor-usage="false"
I0924 09:57:41.509431 1 flags.go:52] FLAG: --log-dir=""
I0924 09:57:41.509451 1 flags.go:52] FLAG: --log-flush-frequency="5s"
I0924 09:57:41.509456 1 flags.go:52] FLAG: --logtostderr="true"
I0924 09:57:41.509461 1 flags.go:52] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I0924 09:57:41.509467 1 flags.go:52] FLAG: --max-autoprovisioned-node-group-count="15"
I0924 09:57:41.509486 1 flags.go:52] FLAG: --max-empty-bulk-delete="10"
I0924 09:57:41.509491 1 flags.go:52] FLAG: --max-failing-time="15m0s"
I0924 09:57:41.509496 1 flags.go:52] FLAG: --max-graceful-termination-sec="600"
I0924 09:57:41.509501 1 flags.go:52] FLAG: --max-inactivity="10m0s"
I0924 09:57:41.509520 1 flags.go:52] FLAG: --max-node-provision-time="15m0s"
I0924 09:57:41.509525 1 flags.go:52] FLAG: --max-nodes-total="0"
I0924 09:57:41.509530 1 flags.go:52] FLAG: --max-total-unready-percentage="45"
I0924 09:57:41.509536 1 flags.go:52] FLAG: --memory-total="0:6400000"
I0924 09:57:41.509556 1 flags.go:52] FLAG: --min-replica-count="0"
I0924 09:57:41.509561 1 flags.go:52] FLAG: --namespace="kube-system"
I0924 09:57:41.509566 1 flags.go:52] FLAG: --node-autoprovisioning-enabled="false"
I0924 09:57:41.509570 1 flags.go:52] FLAG: --node-group-auto-discovery="[]"
I0924 09:57:41.509590 1 flags.go:52] FLAG: --nodes="[5:10:agentpool]"
I0924 09:57:41.509596 1 flags.go:52] FLAG: --ok-total-unready-count="3"
I0924 09:57:41.509601 1 flags.go:52] FLAG: --regional="false"
I0924 09:57:41.509606 1 flags.go:52] FLAG: --scale-down-candidates-pool-min-count="50"
I0924 09:57:41.509624 1 flags.go:52] FLAG: --scale-down-candidates-pool-ratio="0.1"
I0924 09:57:41.509630 1 flags.go:52] FLAG: --scale-down-delay-after-add="10m0s"
I0924 09:57:41.509635 1 flags.go:52] FLAG: --scale-down-delay-after-delete="10s"
I0924 09:57:41.509639 1 flags.go:52] FLAG: --scale-down-delay-after-failure="3m0s"
I0924 09:57:41.509652 1 flags.go:52] FLAG: --scale-down-enabled="true"
I0924 09:57:41.509657 1 flags.go:52] FLAG: --scale-down-non-empty-candidates-count="30"
I0924 09:57:41.509661 1 flags.go:52] FLAG: --scale-down-unneeded-time="10m0s"
I0924 09:57:41.509666 1 flags.go:52] FLAG: --scale-down-unready-time="20m0s"
I0924 09:57:41.509670 1 flags.go:52] FLAG: --scale-down-utilization-threshold="0.5"
I0924 09:57:41.509691 1 flags.go:52] FLAG: --scan-interval="10s"
I0924 09:57:41.509696 1 flags.go:52] FLAG: --skip-nodes-with-local-storage="false"
I0924 09:57:41.509700 1 flags.go:52] FLAG: --skip-nodes-with-system-pods="true"
I0924 09:57:41.509705 1 flags.go:52] FLAG: --stderrthreshold="2"
I0924 09:57:41.509725 1 flags.go:52] FLAG: --storage-driver-buffer-duration="1m0s"
I0924 09:57:41.509730 1 flags.go:52] FLAG: --storage-driver-db="cadvisor"
I0924 09:57:41.509735 1 flags.go:52] FLAG: --storage-driver-host="localhost:8086"
I0924 09:57:41.509739 1 flags.go:52] FLAG: --storage-driver-password="root"
I0924 09:57:41.509744 1 flags.go:52] FLAG: --storage-driver-secure="false"
I0924 09:57:41.509761 1 flags.go:52] FLAG: --storage-driver-table="stats"
I0924 09:57:41.509766 1 flags.go:52] FLAG: --storage-driver-user="root"
I0924 09:57:41.509771 1 flags.go:52] FLAG: --test.bench=""
I0924 09:57:41.509775 1 flags.go:52] FLAG: --test.benchmem="false"
I0924 09:57:41.509795 1 flags.go:52] FLAG: --test.benchtime="1s"
I0924 09:57:41.509800 1 flags.go:52] FLAG: --test.blockprofile=""
I0924 09:57:41.509804 1 flags.go:52] FLAG: --test.blockprofilerate="1"
I0924 09:57:41.509809 1 flags.go:52] FLAG: --test.count="1"
I0924 09:57:41.509813 1 flags.go:52] FLAG: --test.coverprofile=""
I0924 09:57:41.509833 1 flags.go:52] FLAG: --test.cpu=""
I0924 09:57:41.509837 1 flags.go:52] FLAG: --test.cpuprofile=""
I0924 09:57:41.509842 1 flags.go:52] FLAG: --test.failfast="false"
I0924 09:57:41.509846 1 flags.go:52] FLAG: --test.list=""
I0924 09:57:41.509868 1 flags.go:52] FLAG: --test.memprofile=""
I0924 09:57:41.509873 1 flags.go:52] FLAG: --test.memprofilerate="0"
I0924 09:57:41.509877 1 flags.go:52] FLAG: --test.mutexprofile=""
I0924 09:57:41.509882 1 flags.go:52] FLAG: --test.mutexprofilefraction="1"
I0924 09:57:41.509886 1 flags.go:52] FLAG: --test.outputdir=""
I0924 09:57:41.509902 1 flags.go:52] FLAG: --test.parallel="16"
I0924 09:57:41.509907 1 flags.go:52] FLAG: --test.run=""
I0924 09:57:41.509951 1 flags.go:52] FLAG: --test.short="false"
I0924 09:57:41.509959 1 flags.go:52] FLAG: --test.testlogfile=""
I0924 09:57:41.509982 1 flags.go:52] FLAG: --test.timeout="0s"
I0924 09:57:41.509988 1 flags.go:52] FLAG: --test.trace=""
I0924 09:57:41.509994 1 flags.go:52] FLAG: --test.v="false"
I0924 09:57:41.509999 1 flags.go:52] FLAG: --v="6"
I0924 09:57:41.510005 1 flags.go:52] FLAG: --version="false"
I0924 09:57:41.510025 1 flags.go:52] FLAG: --vmodule=""
I0924 09:57:41.510031 1 flags.go:52] FLAG: --write-status-configmap="true"
I0924 09:57:41.510054 1 main.go:311] Cluster Autoscaler 1.3.3_

'''

other cluster-autoscaler log extract:
'''
_I0924 12:04:47.228074 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0924 12:04:47.228242 1 utils.go:451] Skipping aks-agentpool-33865844-0 - no node group config
I0924 12:04:47.228300 1 utils.go:451] Skipping aks-agentpool-33865844-1 - no node group config
I0924 12:04:47.228333 1 utils.go:451] Skipping aks-agentpool-33865844-2 - no node group config
I0924 12:04:47.228406 1 utils.go:451] Skipping aks-agentpool-33865844-3 - no node group config
I0924 12:04:47.228493 1 utils.go:451] Skipping aks-agentpool-33865844-4 - no node group config
I0924 12:04:47.228559 1 utils.go:451] Skipping aks-agentpool-33865844-5 - no node group config
I0924 12:04:47.228893 1 static_autoscaler.go:331] Scale down status: unneededOnly=true lastScaleUpTime=2018-09-24 12:03:33.3416408 +0000 UTC m=+22.111044501 lastScaleDownDeleteTime=2018-09-24 12:03:33.3416409 +0000 UTC m=+22.111044701 lastScaleDownFailTime=2018-09-24 12:03:33.3416411 +0000 UTC m=+22.111044801 scaleDownForbidden=false isDeleteInProgress=false_
''''

Got a runtime panic: as soon as autoscaler found unshedulable pods

'''
_
W0924 12:13:55.585498 1 clusterstate.go:539] Readiness for node group agentpool not found
I0924 12:13:55.585718 1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0924 12:13:55.586012 1 static_autoscaler.go:244] Filtering out schedulables
I0924 12:13:55.593844 1 static_autoscaler.go:254] No schedulable pods
I0924 12:13:55.593973 1 scale_up.go:249] Pod test/testdep2-5b68d598f5-v8wwq is unschedulable
I0924 12:13:55.593989 1 scale_up.go:249] Pod test/testdep2-5b68d598f5-jf8ff is unschedulable
I0924 12:13:55.593996 1 scale_up.go:249] Pod test/testdep2-5b68d598f5-8bbbh is unschedulable
I0924 12:13:55.594001 1 scale_up.go:249] Pod test/testdep2-5b68d598f5-r25fz is unschedulable

W0924 12:13:55.766106 1 clusterstate.go:321] Failed to find readiness information for agentpool
W0924 12:13:55.766120 1 clusterstate.go:377] Failed to find readiness information for agentpool
W0924 12:13:55.766127 1 clusterstate.go:321] Failed to find readiness information for agentpool
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xa056e4]

goroutine 138 [running]:
k8s.io/autoscaler/cluster-autoscaler/utils/errors.ToAutoscalerError(0x38487c6, 0x12, 0x0, 0x0, 0x5cc1b40, 0xc42135a000)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/utils/errors/errors.go:74 +0x74
k8s.io/autoscaler/cluster-autoscaler/core.calculateScaleUpCoresMemoryTotal(0xc42135a100, 0x1, 0x1, 0xc42115b500, 0xc4207b5800, 0x6, 0x8, 0xc4204ac800, 0xc4223b0b48, 0x7f2e43f3e000, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/scale_up.go:119 +0x1a4
k8s.io/autoscaler/cluster-autoscaler/core.computeScaleUpResourcesLeftLimits(0xc42135a100, 0x1, 0x1, 0xc42115b500, 0xc4207b5800, 0x6, 0x8, 0xc4202eb730, 0x0, 0x0, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/scale_up.go:56 +0x8e
k8s.io/autoscaler/cluster-autoscaler/core.ScaleUp(0xc422d601c0, 0xc4207b5080, 0xc421902000, 0xc421582800, 0x89, 0x100, 0xc4218943c0, 0x6, 0x8, 0xc42045ed90, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/scale_up.go:273 +0x57a
k8s.io/autoscaler/cluster-autoscaler/core.(StaticAutoscaler).RunOnce(0xc4211b8b40, 0xbee253209e970f90, 0x3ed22bb7d, 0x5c9ef20, 0x0, 0x0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/static_autoscaler.go:278 +0x1d86
main.run(0xc420843860)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:288 +0x48d
main.main.func2(0xc420054ae0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:369 +0x2a
created by k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection.(LeaderElector).Run
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:155 +0x92
_
'''

Does anyone have an idea ?
Thanks

Regards
Stephane

areprovideazure cluster-autoscaler
Source
picture stroon2

Most helpful comment

Great! Thank you ! Tags were missing !

Autoscaler works now :-)

PS: All VM's tags have been deteled by an inappropriate post-install script.
I used the command "az ressource tag... " which replaces tags instead of adding to existing ones.

Thanks everyone for your help.

picture stroon2 on 26 Oct 2018
👍2 ❤1

All 17 comments

The direct cause of panic is bug in scale_up.go.
We try to dereference err which is nil in that case.
But underlying issue is that we are not getting NodeInfo for presumably autoscaled node group - which suggests configuration issue. Will get back to that later today.

losipiuk picture losipiuk on 25 Sep 2018

This panic has been fixed on master already. What probably remains to be fixed is entering err != nil branch with nil error (probably caused by Golang's nil interface gotcha * sigh *)

aleksandra-malinowska picture aleksandra-malinowska on 25 Sep 2018

Ok - from what I see the Azure cloud provider does not support scale-up from empty node groups.
Please ensure that all autoscaled agentpools are non-empty and the problem should stop to surface.

cc: @feiskyer

losipiuk picture losipiuk on 25 Sep 2018

from what I see the Azure cloud provider does not support scale-up from empty node groups.

This is right except VMSS. Only VMSS support zero nodes on Azure now.

@stroon2 From the logs, CA failed to get node group for all AKS nodes. Are there any error logs before Skipping aks-agentpool-33865844-0 - no node group config? And have you set NodeResourceGroup to MC_RG-APPLI-HPROD_AKS-LHA-HPROD02_northeurope (case sensitive)?

cc @kkmsft

feiskyer picture feiskyer on 26 Sep 2018

Here is a complete Autoscaler pod logs (loglevel=5)
I replaced few strings by *

I tried with loglevel 6 to see it was any connexion problems but I haven't found anything.
I've double checked the NodeResourceGroup name.

The deployment has been made following AKS documentation https://docs.microsoft.com/en-us/azure/aks/autoscaler. Is there something missing ?

'''
_I0926 08:31:43.905949 1 flags.go:52] FLAG: --address=":8085"
I0926 08:31:43.906005 1 flags.go:52] FLAG: --alsologtostderr="false"
I0926 08:31:43.906026 1 flags.go:52] FLAG: --application-metrics-count-limit="100"
I0926 08:31:43.906029 1 flags.go:52] FLAG: --azure-container-registry-config=""
I0926 08:31:43.906033 1 flags.go:52] FLAG: --balance-similar-node-groups="false"
I0926 08:31:43.906035 1 flags.go:52] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
I0926 08:31:43.906038 1 flags.go:52] FLAG: --cloud-config=""
I0926 08:31:43.906041 1 flags.go:52] FLAG: --cloud-provider="azure"
I0926 08:31:43.906043 1 flags.go:52] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0926 08:31:43.906053 1 flags.go:52] FLAG: --cluster-name=""
I0926 08:31:43.906055 1 flags.go:52] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
I0926 08:31:43.906058 1 flags.go:52] FLAG: --containerd="unix:///var/run/containerd.sock"
I0926 08:31:43.906061 1 flags.go:52] FLAG: --cores-total="0:320000"
I0926 08:31:43.906063 1 flags.go:52] FLAG: --docker="unix:///var/run/docker.sock"
I0926 08:31:43.906066 1 flags.go:52] FLAG: --docker-env-metadata-whitelist=""
I0926 08:31:43.906068 1 flags.go:52] FLAG: --docker-only="false"
I0926 08:31:43.906070 1 flags.go:52] FLAG: --docker-root="/var/lib/docker"
I0926 08:31:43.906073 1 flags.go:52] FLAG: --docker-tls="false"
I0926 08:31:43.906075 1 flags.go:52] FLAG: --docker-tls-ca="ca.pem"
I0926 08:31:43.906078 1 flags.go:52] FLAG: --docker-tls-cert="cert.pem"
I0926 08:31:43.906098 1 flags.go:52] FLAG: --docker-tls-key="key.pem"
I0926 08:31:43.906101 1 flags.go:52] FLAG: --enable-load-reader="false"
I0926 08:31:43.906103 1 flags.go:52] FLAG: --estimator="binpacking"
I0926 08:31:43.906106 1 flags.go:52] FLAG: --event-storage-age-limit="default=0"
I0926 08:31:43.906129 1 flags.go:52] FLAG: --event-storage-event-limit="default=0"
I0926 08:31:43.906132 1 flags.go:52] FLAG: --expander="random"
I0926 08:31:43.906135 1 flags.go:52] FLAG: --expendable-pods-priority-cutoff="0"
I0926 08:31:43.906138 1 flags.go:52] FLAG: --gke-api-endpoint=""
I0926 08:31:43.906141 1 flags.go:52] FLAG: --global-housekeeping-interval="1m0s"
I0926 08:31:43.906144 1 flags.go:52] FLAG: --google-json-key=""
I0926 08:31:43.906147 1 flags.go:52] FLAG: --gpu-total="[]"
I0926 08:31:43.906204 1 flags.go:52] FLAG: --housekeeping-interval="10s"
I0926 08:31:43.906209 1 flags.go:52] FLAG: --httptest.serve=""
I0926 08:31:43.906212 1 flags.go:52] FLAG: --kubeconfig=""
I0926 08:31:43.906215 1 flags.go:52] FLAG: --kubernetes=""
I0926 08:31:43.906218 1 flags.go:52] FLAG: --leader-elect="true"
I0926 08:31:43.906224 1 flags.go:52] FLAG: --leader-elect-lease-duration="15s"
I0926 08:31:43.906229 1 flags.go:52] FLAG: --leader-elect-renew-deadline="10s"
I0926 08:31:43.906233 1 flags.go:52] FLAG: --leader-elect-resource-lock="endpoints"
I0926 08:31:43.906237 1 flags.go:52] FLAG: --leader-elect-retry-period="2s"
I0926 08:31:43.906241 1 flags.go:52] FLAG: --log-backtrace-at=":0"
I0926 08:31:43.906248 1 flags.go:52] FLAG: --log-cadvisor-usage="false"
I0926 08:31:43.906252 1 flags.go:52] FLAG: --log-dir=""
I0926 08:31:43.906256 1 flags.go:52] FLAG: --log-flush-frequency="5s"
I0926 08:31:43.906260 1 flags.go:52] FLAG: --logtostderr="true"
I0926 08:31:43.906264 1 flags.go:52] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
I0926 08:31:43.906268 1 flags.go:52] FLAG: --max-autoprovisioned-node-group-count="15"
I0926 08:31:43.906272 1 flags.go:52] FLAG: --max-empty-bulk-delete="10"
I0926 08:31:43.906276 1 flags.go:52] FLAG: --max-failing-time="15m0s"
I0926 08:31:43.906279 1 flags.go:52] FLAG: --max-graceful-termination-sec="600"
I0926 08:31:43.906282 1 flags.go:52] FLAG: --max-inactivity="10m0s"
I0926 08:31:43.906286 1 flags.go:52] FLAG: --max-node-provision-time="15m0s"
I0926 08:31:43.906289 1 flags.go:52] FLAG: --max-nodes-total="0"
I0926 08:31:43.906294 1 flags.go:52] FLAG: --max-total-unready-percentage="45"
I0926 08:31:43.906299 1 flags.go:52] FLAG: --memory-total="0:6400000"
I0926 08:31:43.906302 1 flags.go:52] FLAG: --min-replica-count="0"
I0926 08:31:43.906307 1 flags.go:52] FLAG: --namespace="kube-system"
I0926 08:31:43.906310 1 flags.go:52] FLAG: --node-autoprovisioning-enabled="false"
I0926 08:31:43.906314 1 flags.go:52] FLAG: --node-group-auto-discovery="[]"
I0926 08:31:43.906317 1 flags.go:52] FLAG: --nodes="[5:10:agentpool]"
I0926 08:31:43.906322 1 flags.go:52] FLAG: --ok-total-unready-count="3"
I0926 08:31:43.906326 1 flags.go:52] FLAG: --regional="false"
I0926 08:31:43.906329 1 flags.go:52] FLAG: --scale-down-candidates-pool-min-count="50"
I0926 08:31:43.906332 1 flags.go:52] FLAG: --scale-down-candidates-pool-ratio="0.1"
I0926 08:31:43.906338 1 flags.go:52] FLAG: --scale-down-delay-after-add="10m0s"
I0926 08:31:43.906342 1 flags.go:52] FLAG: --scale-down-delay-after-delete="10s"
I0926 08:31:43.906345 1 flags.go:52] FLAG: --scale-down-delay-after-failure="3m0s"
I0926 08:31:43.906348 1 flags.go:52] FLAG: --scale-down-enabled="true"
I0926 08:31:43.906351 1 flags.go:52] FLAG: --scale-down-non-empty-candidates-count="30"
I0926 08:31:43.906355 1 flags.go:52] FLAG: --scale-down-unneeded-time="10m0s"
I0926 08:31:43.906358 1 flags.go:52] FLAG: --scale-down-unready-time="20m0s"
I0926 08:31:43.906363 1 flags.go:52] FLAG: --scale-down-utilization-threshold="0.5"
I0926 08:31:43.906366 1 flags.go:52] FLAG: --scan-interval="10s"
I0926 08:31:43.906370 1 flags.go:52] FLAG: --skip-nodes-with-local-storage="false"
I0926 08:31:43.906373 1 flags.go:52] FLAG: --skip-nodes-with-system-pods="true"
I0926 08:31:43.906376 1 flags.go:52] FLAG: --stderrthreshold="2"
I0926 08:31:43.906379 1 flags.go:52] FLAG: --storage-driver-buffer-duration="1m0s"
I0926 08:31:43.906383 1 flags.go:52] FLAG: --storage-driver-db="cadvisor"
I0926 08:31:43.906386 1 flags.go:52] FLAG: --storage-driver-host="localhost:8086"
I0926 08:31:43.906390 1 flags.go:52] FLAG: --storage-driver-password="root"
I0926 08:31:43.906394 1 flags.go:52] FLAG: --storage-driver-secure="false"
I0926 08:31:43.906397 1 flags.go:52] FLAG: --storage-driver-table="stats"
I0926 08:31:43.906400 1 flags.go:52] FLAG: --storage-driver-user="root"
I0926 08:31:43.906404 1 flags.go:52] FLAG: --test.bench=""
I0926 08:31:43.906407 1 flags.go:52] FLAG: --test.benchmem="false"
I0926 08:31:43.906412 1 flags.go:52] FLAG: --test.benchtime="1s"
I0926 08:31:43.906415 1 flags.go:52] FLAG: --test.blockprofile=""
I0926 08:31:43.906418 1 flags.go:52] FLAG: --test.blockprofilerate="1"
I0926 08:31:43.906421 1 flags.go:52] FLAG: --test.count="1"
I0926 08:31:43.906425 1 flags.go:52] FLAG: --test.coverprofile=""
I0926 08:31:43.906428 1 flags.go:52] FLAG: --test.cpu=""
I0926 08:31:43.906431 1 flags.go:52] FLAG: --test.cpuprofile=""
I0926 08:31:43.906435 1 flags.go:52] FLAG: --test.failfast="false"
I0926 08:31:43.906438 1 flags.go:52] FLAG: --test.list=""
I0926 08:31:43.906442 1 flags.go:52] FLAG: --test.memprofile=""
I0926 08:31:43.906445 1 flags.go:52] FLAG: --test.memprofilerate="0"
I0926 08:31:43.906448 1 flags.go:52] FLAG: --test.mutexprofile=""
I0926 08:31:43.906451 1 flags.go:52] FLAG: --test.mutexprofilefraction="1"
I0926 08:31:43.906454 1 flags.go:52] FLAG: --test.outputdir=""
I0926 08:31:43.906457 1 flags.go:52] FLAG: --test.parallel="16"
I0926 08:31:43.906461 1 flags.go:52] FLAG: --test.run=""
I0926 08:31:43.906464 1 flags.go:52] FLAG: --test.short="false"
I0926 08:31:43.906468 1 flags.go:52] FLAG: --test.testlogfile=""
I0926 08:31:43.906471 1 flags.go:52] FLAG: --test.timeout="0s"
I0926 08:31:43.906475 1 flags.go:52] FLAG: --test.trace=""
I0926 08:31:43.906478 1 flags.go:52] FLAG: --test.v="false"
I0926 08:31:43.906481 1 flags.go:52] FLAG: --v="5"
I0926 08:31:43.906486 1 flags.go:52] FLAG: --version="false"
I0926 08:31:43.906492 1 flags.go:52] FLAG: --vmodule=""
I0926 08:31:43.906496 1 flags.go:52] FLAG: --write-status-configmap="true"
I0926 08:31:43.906501 1 main.go:311] Cluster Autoscaler 1.3.2
I0926 08:31:44.220243 1 leaderelection.go:185] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0926 08:31:44.233111 1 leaderelection.go:253] lock is held by cluster-autoscaler-6b75bcfccc-5jrzq and has not yet expired
I0926 08:31:44.233151 1 leaderelection.go:190] failed to acquire lease kube-system/cluster-autoscaler
I0926 08:31:46.673409 1 leaderelection.go:253] lock is held by cluster-autoscaler-6b75bcfccc-5jrzq and has not yet expired
I0926 08:31:46.673434 1 leaderelection.go:190] failed to acquire lease kube-system/cluster-autoscaler
I0926 08:31:50.292821 1 leaderelection.go:253] lock is held by cluster-autoscaler-6b75bcfccc-5jrzq and has not yet expired
I0926 08:31:50.292843 1 leaderelection.go:190] failed to acquire lease kube-system/cluster-autoscaler
I0926 08:31:53.308750 1 leaderelection.go:253] lock is held by cluster-autoscaler-6b75bcfccc-5jrzq and has not yet expired
I0926 08:31:53.308775 1 leaderelection.go:190] failed to acquire lease kube-system/cluster-autoscaler
I0926 08:31:56.603702 1 leaderelection.go:253] lock is held by cluster-autoscaler-6b75bcfccc-5jrzq and has not yet expired
I0926 08:31:56.603729 1 leaderelection.go:190] failed to acquire lease kube-system/cluster-autoscaler
I0926 08:32:00.922966 1 leaderelection.go:194] successfully acquired lease kube-system/cluster-autoscaler
I0926 08:32:00.923282 1 factory.go:33] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cluster-autoscaler", UID:"f41a11be-bdb0-11e8-92bf-bedee911215a", APIVersion:"v1", ResourceVersion:"2569976", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster-autoscaler-6b75bcfccc-qs67m became leader
I0926 08:32:00.925751 1 predicates.go:126] Using predicate PodFitsResources
I0926 08:32:00.925968 1 predicates.go:126] Using predicate GeneralPredicates
I0926 08:32:00.926005 1 predicates.go:126] Using predicate PodToleratesNodeTaints
I0926 08:32:00.926042 1 predicates.go:126] Using predicate MaxEBSVolumeCount
I0926 08:32:00.926084 1 predicates.go:126] Using predicate MaxGCEPDVolumeCount
I0926 08:32:00.926116 1 predicates.go:126] Using predicate NoDiskConflict
I0926 08:32:00.926154 1 reflector.go:202] Starting reflector *v1.Service (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.926219 1 reflector.go:240] Listing and watching *v1.Service from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.926154 1 predicates.go:126] Using predicate NoVolumeZoneConflict
I0926 08:32:00.927320 1 reflector.go:202] Starting reflector *v1beta1.ReplicaSet (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927345 1 reflector.go:240] Listing and watching *v1beta1.ReplicaSet from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927372 1 reflector.go:202] Starting reflector *v1beta1.StatefulSet (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927336 1 predicates.go:126] Using predicate CheckNodeMemoryPressure
I0926 08:32:00.927454 1 predicates.go:126] Using predicate CheckNodePIDPressure
I0926 08:32:00.927464 1 predicates.go:126] Using predicate ready
I0926 08:32:00.927471 1 predicates.go:126] Using predicate CheckNodeCondition
I0926 08:32:00.927476 1 predicates.go:126] Using predicate CheckNodeDiskPressure
I0926 08:32:00.927483 1 predicates.go:126] Using predicate CheckVolumeBinding
I0926 08:32:00.927477 1 reflector.go:202] Starting reflector *v1.PersistentVolumeClaim (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927493 1 reflector.go:202] Starting reflector *v1.Node (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927521 1 reflector.go:240] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927486 1 predicates.go:126] Using predicate MatchInterPodAffinity
I0926 08:32:00.927602 1 predicates.go:126] Using predicate MaxAzureDiskVolumeCount
I0926 08:32:00.927498 1 reflector.go:240] Listing and watching *v1.PersistentVolumeClaim from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927728 1 reflector.go:202] Starting reflector *v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:174
I0926 08:32:00.927758 1 reflector.go:240] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:174
I0926 08:32:00.927759 1 reflector.go:202] Starting reflector *v1beta1.PodDisruptionBudget (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927777 1 reflector.go:240] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927807 1 reflector.go:202] Starting reflector *v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0926 08:32:00.927832 1 reflector.go:240] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0926 08:32:00.927831 1 reflector.go:202] Starting reflector *v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:149
I0926 08:32:00.927750 1 reflector.go:202] Starting reflector *v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:239
I0926 08:32:00.927865 1 reflector.go:240] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:149
I0926 08:32:00.927869 1 reflector.go:240] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:239
I0926 08:32:00.927928 1 reflector.go:202] Starting reflector *v1.Pod (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927960 1 reflector.go:240] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927837 1 reflector.go:202] Starting reflector *v1beta1.DaemonSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:293
I0926 08:32:00.928032 1 reflector.go:240] Listing and watching *v1beta1.DaemonSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:293
I0926 08:32:00.927192 1 reflector.go:202] Starting reflector *v1.ReplicationController (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.928114 1 reflector.go:240] Listing and watching *v1.ReplicationController from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927437 1 reflector.go:240] Listing and watching *v1beta1.StatefulSet from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:00.927846 1 reflector.go:202] Starting reflector *v1beta1.PodDisruptionBudget (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:266
I0926 08:32:00.928543 1 reflector.go:240] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:266
I0926 08:32:01.003583 1 reflector.go:202] Starting reflector *v1.PersistentVolume (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:01.003612 1 reflector.go:240] Listing and watching *v1.PersistentVolume from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:01.003682 1 reflector.go:202] Starting reflector *v1.StorageClass (0s) from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:01.003727 1 reflector.go:240] Listing and watching *v1.StorageClass from k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/informers/factory.go:130
I0926 08:32:01.204843 1 request.go:485] Throttling request took 201.052101ms, request: GET:https://172.17.80.1:443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0
I0926 08:32:01.503531 1 request.go:485] Throttling request took 297.620906ms, request: POST:https://172.17.80.1:443/api/v1/namespaces/kube-system/configmaps
I0926 08:32:01.705230 1 cloud_provider_builder.go:72] Building azure cloud provider.
I0926 08:32:01.705266 1 cloud_provider_builder.go:158] Creating Azure Manager with default configuration.
I0926 08:32:01.705300 1 azure_manager.go:163] Starting azure manager with subscription ID "581c1d56-05c5-4f65-b4dd-9908d82eed29"
I0926 08:32:01.705339 1 azure_client.go:440] azure: using client_id+client_secret to retrieve access token
I0926 08:32:01.705376 1 azure_client.go:476] Created scale set client with authorizer: &{{{{0xc42044dd50 0xc420a9acf0 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 compute/2017-12-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705425 1 azure_client.go:479] Created scale set vm client with authorizer: &{{{{0xc42044dda0 0xc420a9ad80 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 compute/2017-12-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705487 1 azure_client.go:482] Created vm client with authorizer: &{{{{0xc42044ddf0 0xc420a9ae10 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 compute/2017-12-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705507 1 azure_client.go:485] Created deployments client with authorizer: &{{{{0xc42044de40 0xc420a9aea0 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 resources/2017-05-10; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705533 1 azure_client.go:488] Created interfaces client with authorizer: &{{{{0xc42044de90 0xc420a9af30 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 network/2017-09-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705556 1 azure_client.go:491] Created storage accounts client with authorizer: &{{{{0xc42044dee0 0xc420a9afc0 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 storage/2017-10-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705580 1 azure_client.go:494] Created disks client with authorizer: &{{{{0xc42044df30 0xc420a9b050 5000000000 900000000000 3 30000000000 Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 compute/2017-12-01; cluster-autoscaler/v0.0.0-master+$Format:%h$ false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}}
I0926 08:32:01.705606 1 azure_client.go:501] Created Container services client with authorizer: {{{0xc42044df80 0xc420a9b110 5s 15m0s 3 30s Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 containerservice/2018-03-31 false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}
I0926 08:32:01.803629 1 azure_client.go:508] Created Managed Container services client with authorizer: {{{0xc42044dfb0 0xc420a9b200 5s 15m0s 3 30s Go/go1.10.2 (amd64-linux) go-autorest/v10.12.0 Azure-SDK-For-Go/v14.6.0 containerservice/2018-03-31 false} https://management.azure.com/ 581c1d56-05c5-4f65-b4dd-9908d82eed29}}
I0926 08:32:01.803737 1 azure_cache.go:78] Registering ASG "agentpool"
I0926 08:32:01.803763 1 azure_cache.go:85] Invalidating unowned instance cache
I0926 08:32:02.968483 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:05.003113 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:07.103785 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:08.974319 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:08.974345 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:08.974351 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:08.974357 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:08.974363 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:08.974367 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:08.974391 1 azure_manager.go:261] Refreshed ASG list, next refresh after 2018-09-26 08:33:08.974385628 +0000 UTC m=+86.850830442
I0926 08:32:08.974679 1 main.go:239] Registered cleanup signal handler
I0926 08:32:09.059547 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:09.059572 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:09.059578 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:09.059583 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:09.059586 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:09.059590 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:09.166934 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:11.212910 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:13.244616 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:15.303706 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:17.340178 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:19.060759 1 static_autoscaler.go:131] Starting main loop
I0926 08:32:19.293082 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:19.380329 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:19.392025 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.392050 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.392057 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.392061 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.392065 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.392069 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.466514 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.466537 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.466544 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.466549 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.466555 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.466560 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.558731 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.558754 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.558759 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.558764 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.558767 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.558771 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.627884 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.627904 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.627928 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.627931 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.627935 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.627938 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.739297 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.739345 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.739355 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.739360 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.739365 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.739370 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.836399 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.836424 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.836431 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.836437 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.836459 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.836483 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
I0926 08:32:19.930145 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:19.930167 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:19.930172 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:19.930177 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:19.930182 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:19.930187 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
W0926 08:32:19.930223 1 clusterstate.go:539] Readiness for node group agentpool not found
I0926 08:32:19.930286 1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0926 08:32:19.930306 1 static_autoscaler.go:244] Filtering out schedulables
I0926 08:32:19.930492 1 static_autoscaler.go:254] No schedulable pods
I0926 08:32:19.930512 1 static_autoscaler.go:258] No unschedulable pods
I0926 08:32:19.930538 1 static_autoscaler.go:305] Calculating unneeded nodes
I0926 08:32:20.097530 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:20.097559 1 utils.go:451] Skipping aks-agentpool-33865844-4 - no node group config
I0926 08:32:20.097567 1 utils.go:451] Skipping aks-agentpool-33865844-5 - no node group config
I0926 08:32:20.097574 1 utils.go:451] Skipping aks-agentpool-33865844-0 - no node group config
I0926 08:32:20.097580 1 utils.go:451] Skipping aks-agentpool-33865844-1 - no node group config
I0926 08:32:20.097587 1 utils.go:451] Skipping aks-agentpool-33865844-2 - no node group config
I0926 08:32:20.097613 1 utils.go:451] Skipping aks-agentpool-33865844-3 - no node group config
I0926 08:32:20.097904 1 static_autoscaler.go:331] Scale down status: unneededOnly=true lastScaleUpTime=2018-09-26 08:32:08.974422828 +0000 UTC m=+26.850867742 lastScaleDownDeleteTime=2018-09-26 08:32:08.974422928 +0000 UTC m=+26.850867742 lastScaleDownFailTime=2018-09-26 08:32:08.974423028 +0000 UTC m=+26.850867842 scaleDownForbidden=false isDeleteInProgress=false
W0926 08:32:20.097938 1 clusterstate.go:321] Failed to find readiness information for agentpool
W0926 08:32:20.097951 1 clusterstate.go:377] Failed to find readiness information for agentpool
W0926 08:32:20.097985 1 clusterstate.go:321] Failed to find readiness information for agentpool
I0926 08:32:21.449247 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:23.489311 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:25.614936 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:27.648393 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:29.732529 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:30.159100 1 static_autoscaler.go:131] Starting main loop
I0926 08:32:30.327920 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:30.419248 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:30.419284 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:30.419289 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:30.419292 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:30.419295 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:30.419299 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
W0926 08:32:30.419359 1 clusterstate.go:539] Readiness for node group agentpool not found
I0926 08:32:30.419473 1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0926 08:32:30.419547 1 static_autoscaler.go:244] Filtering out schedulables
I0926 08:32:30.419706 1 static_autoscaler.go:254] No schedulable pods
I0926 08:32:30.419727 1 static_autoscaler.go:258] No unschedulable pods
I0926 08:32:30.419738 1 static_autoscaler.go:305] Calculating unneeded nodes
I0926 08:32:30.564779 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:30.564897 1 utils.go:451] Skipping aks-agentpool-33865844-3 - no node group config
I0926 08:32:30.564921 1 utils.go:451] Skipping aks-agentpool-33865844-4 - no node group config
I0926 08:32:30.564928 1 utils.go:451] Skipping aks-agentpool-33865844-5 - no node group config
I0926 08:32:30.564933 1 utils.go:451] Skipping aks-agentpool-33865844-0 - no node group config
I0926 08:32:30.564970 1 utils.go:451] Skipping aks-agentpool-33865844-1 - no node group config
I0926 08:32:30.564976 1 utils.go:451] Skipping aks-agentpool-33865844-2 - no node group config
I0926 08:32:30.565290 1 static_autoscaler.go:331] Scale down status: unneededOnly=true lastScaleUpTime=2018-09-26 08:32:08.974422828 +0000 UTC m=+26.850867742 lastScaleDownDeleteTime=2018-09-26 08:32:08.974422928 +0000 UTC m=+26.850867742 lastScaleDownFailTime=2018-09-26 08:32:08.974423028 +0000 UTC m=+26.850867842 scaleDownForbidden=false isDeleteInProgress=false
W0926 08:32:30.565326 1 clusterstate.go:321] Failed to find readiness information for agentpool
W0926 08:32:30.565340 1 clusterstate.go:377] Failed to find readiness information for agentpool
W0926 08:32:30.565361 1 clusterstate.go:321] Failed to find readiness information for agentpool
I0926 08:32:31.784654 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:33.830656 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:35.935314 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:37.993505 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:40.023410 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:40.605158 1 static_autoscaler.go:131] Starting main loop
I0926 08:32:40.752466 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:40.840784 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:40.840812 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:40.840820 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:40.840825 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:40.840831 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:40.840836 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
W0926 08:32:40.840870 1 clusterstate.go:539] Readiness for node group agentpool not found
I0926 08:32:40.840973 1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0926 08:32:40.840987 1 static_autoscaler.go:244] Filtering out schedulables
I0926 08:32:40.841147 1 static_autoscaler.go:254] No schedulable pods
I0926 08:32:40.841170 1 static_autoscaler.go:258] No unschedulable pods
I0926 08:32:40.841181 1 static_autoscaler.go:305] Calculating unneeded nodes
I0926 08:32:40.997177 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:40.997212 1 utils.go:451] Skipping aks-agentpool-33865844-3 - no node group config
I0926 08:32:40.997220 1 utils.go:451] Skipping aks-agentpool-33865844-4 - no node group config
I0926 08:32:40.997227 1 utils.go:451] Skipping aks-agentpool-33865844-5 - no node group config
I0926 08:32:40.997234 1 utils.go:451] Skipping aks-agentpool-33865844-0 - no node group config
I0926 08:32:40.997239 1 utils.go:451] Skipping aks-agentpool-33865844-1 - no node group config
I0926 08:32:40.997246 1 utils.go:451] Skipping aks-agentpool-33865844-2 - no node group config
I0926 08:32:40.997535 1 static_autoscaler.go:331] Scale down status: unneededOnly=true lastScaleUpTime=2018-09-26 08:32:08.974422828 +0000 UTC m=+26.850867742 lastScaleDownDeleteTime=2018-09-26 08:32:08.974422928 +0000 UTC m=+26.850867742 lastScaleDownFailTime=2018-09-26 08:32:08.974423028 +0000 UTC m=+26.850867842 scaleDownForbidden=false isDeleteInProgress=false
W0926 08:32:40.997593 1 clusterstate.go:321] Failed to find readiness information for agentpool
W0926 08:32:40.997607 1 clusterstate.go:377] Failed to find readiness information for agentpool
W0926 08:32:40.997618 1 clusterstate.go:321] Failed to find readiness information for agentpool
I0926 08:32:42.069437 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:44.103828 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:46.240472 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:48.303791 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:50.338827 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:51.066705 1 static_autoscaler.go:131] Starting main loop
I0926 08:32:51.214144 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:51.287330 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0
I0926 08:32:51.287363 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-1, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-1
I0926 08:32:51.287376 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-2, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-2
I0926 08:32:51.287453 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-3, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-3
I0926 08:32:51.287458 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-4, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-4
I0926 08:32:51.287467 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-5, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-5
W0926 08:32:51.287516 1 clusterstate.go:539] Readiness for node group agentpool not found
I0926 08:32:51.287593 1 utils.go:503] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
I0926 08:32:51.287620 1 static_autoscaler.go:244] Filtering out schedulables
I0926 08:32:51.287791 1 static_autoscaler.go:254] No schedulable pods
I0926 08:32:51.287810 1 static_autoscaler.go:258] No unschedulable pods
I0926 08:32:51.287819 1 static_autoscaler.go:305] Calculating unneeded nodes
I0926 08:32:51.430680 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
I0926 08:32:51.430711 1 utils.go:451] Skipping aks-agentpool-33865844-0 - no node group config
I0926 08:32:51.430719 1 utils.go:451] Skipping aks-agentpool-33865844-1 - no node group config
I0926 08:32:51.430723 1 utils.go:451] Skipping aks-agentpool-33865844-2 - no node group config
I0926 08:32:51.430728 1 utils.go:451] Skipping aks-agentpool-33865844-3 - no node group config
I0926 08:32:51.430734 1 utils.go:451] Skipping aks-agentpool-33865844-4 - no node group config
I0926 08:32:51.430738 1 utils.go:451] Skipping aks-agentpool-33865844-5 - no node group config
I0926 08:32:51.431139 1 static_autoscaler.go:331] Scale down status: unneededOnly=true lastScaleUpTime=2018-09-26 08:32:08.974422828 +0000 UTC m=+26.850867742 lastScaleDownDeleteTime=2018-09-26 08:32:08.974422928 +0000 UTC m=+26.850867742 lastScaleDownFailTime=2018-09-26 08:32:08.974423028 +0000 UTC m=+26.850867842 scaleDownForbidden=false isDeleteInProgress=false
W0926 08:32:51.431181 1 clusterstate.go:321] Failed to find readiness information for agentpool
W0926 08:32:51.431196 1 clusterstate.go:377] Failed to find readiness information for agentpool
W0926 08:32:51.431203 1 clusterstate.go:321] Failed to find readiness information for agentpool
I0926 08:32:52.431519 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler
I0926 08:32:54.479071 1 leaderelection.go:209] successfully renewed lease kube-system/cluster-autoscaler_
'''

stroon2 picture stroon2 on 26 Sep 2018

Hi @stroon2 - Thank you for using the AKS autoscaler.

Looks like there is a mismatch:

The tag has MC_RG-APPLI-HPROD_AKS-LHA-HPROD02_northeurope
but the NodeResourceGroup provided:

I0926 08:32:51.287330 1 azure_container_service_pool.go:402] Node Name: aks-agentpool-33865844-0, ID: /subscriptions/581c1d56-05c5-4f65-b4dd-9908d82eed29/resourceGroups/MC_RG-APPLI-LES_HALLES-HPROD_AKS-LHA-HPROD02_northeurope/providers/Microsoft.Compute/virtualMachines/aks-agentpool-33865844-0

has "LES_HALLES" as extra term. Can you please change this to match the right resource group from the tag?

kkmsft picture kkmsft on 26 Sep 2018

Thank you for your help but it's not a problem with NodeResourceGroup (I missed my obfuscation!).

I wonder which request is made before getting "Failed to find readiness information for agentpool"? It's a request made to Azure or to Kubernetes ? Is the information missing ou unavailable ?

Regards

stroon2 picture stroon2 on 27 Sep 2018

@stroon2 : One thing which needs fixing is the name of the agent pool. While the information is collected in the - https://docs.microsoft.com/en-us/azure/aks/autoscaler#gather-information the agent pool name(which is usually nodepool1) needs to be the value of the label.

From the configmap, the node pool name is showing up as agentpool instead of nodepool1
Can you please fix this (in --nodes option) and retry.

kkmsft picture kkmsft on 23 Oct 2018

Thanks for your suggestion.

I've just tried to change the --nodes options to --nodes=5:10:nodepool1, but got another error.

I1023 07:10:13.739740 1 azure_container_service_pool.go:80] AKS AgentPool profile name: agentpool
E1023 07:10:13.739775 1 static_autoscaler.go:415] Failed to update node registry: could not find pool with name: {nodepool1}

I use "agentpool" as it is what I found in node labels (this configuration works fine in another cluster).

kubectl get node --show-labels

NAME STATUS ROLES AGE VERSION LABELS
aks-agentpool-33865844-0 Ready agent 19d v1.11.2 agentpool=agentpool,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D16s_v3.../....

stroon2 picture stroon2 on 23 Oct 2018

@stroon2 : Thank you for checking the label and parameter match.

Can you please run the following command and provide with the tags part of the output:

az vm show -n (one of the nodes) -g (the mc resource group) -o json

The tags here is used to match the nodes which belong to the AKS cluster.

kkmsft picture kkmsft on 24 Oct 2018

Great! Thank you ! Tags were missing !

Autoscaler works now :-)

PS: All VM's tags have been deteled by an inappropriate post-install script.
I used the command "az ressource tag... " which replaces tags instead of adding to existing ones.

Thanks everyone for your help.

stroon2 picture stroon2 on 26 Oct 2018
👍2 ❤1

Something seems to have changed around this for recent k8s versions. For a node from 1.15.5, the kubectl describe node returns:

bash [kops]$ ktl describe node aks-agentpool-11151811-0 | grep ProviderID
ProviderID: azure:///subscriptions//resourceGroups/mc_abhiresourceinfa_testamadavk8s_westus/providers/Microsoft.Compute/virtualMachines/aks-agentpool-11151811-0

This is with the node-resource-group in lower case. The node labels, however are created with upper case chars.

bash [kops]$ ktl describe node aks-agentpool-11151811-0 | grep cluster kubernetes.azure.com/cluster=MC_abhiresourceinfa_testamadavk8s_westus

This is causing regression for cluster-autoscaler which relies on secret created with NodeResourceGroup in caps alike the labels.

amadav picture amadav on 7 Feb 2020

CC: @kkmsft

amadav picture amadav on 7 Feb 2020

@amadav are you using cluster-autoscaler with versions matched with k8s version? (refer matrix here)

feiskyer picture feiskyer on 9 Feb 2020

Hello there, same problem here, using AKS with VMSS, Kubernetes 1.15.7, cluster-autoscaler 1.15.4 deployed using Helm from stable/cluster-autoscaler.

ProviderID: azure:///subscriptions/SUBSCRIPTION ID/resourceGroups/mc_xkom-kube-rg_xkom-kube-ops_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-19524067-vmss/virtualMachines/0
Tag: kubernetes.azure.com/cluster=MC_xkom-kube-rg_xkom-kube-ops_westeurope
Actual Azure RG name: MC_xkom-kube-rg_xkom-kube-ops_westeurope

Let me know if you'd like some more data.

rakkhin picture rakkhin on 10 Feb 2020

Hello there, same problem here, using AKS with VMSS, Kubernetes 1.15.7, cluster-autoscaler 1.15.4 deployed using Helm from stable/cluster-autoscaler.

ProviderID: azure:///subscriptions/SUBSCRIPTION ID/resourceGroups/mc_xkom-kube-rg_xkom-kube-ops_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-19524067-vmss/virtualMachines/0
Tag: kubernetes.azure.com/cluster=MC_xkom-kube-rg_xkom-kube-ops_westeurope
Actual Azure RG name: MC_xkom-kube-rg_xkom-kube-ops_westeurope

Let me know if you'd like some more data.

Is there any workaround ?

jederfreitas picture jederfreitas on 26 Oct 2020

Hello there, same problem here, using AKS with VMSS, Kubernetes 1.15.7, cluster-autoscaler 1.15.4 deployed using Helm from stable/cluster-autoscaler.
ProviderID: azure:///subscriptions/SUBSCRIPTION ID/resourceGroups/mc_xkom-kube-rg_xkom-kube-ops_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-19524067-vmss/virtualMachines/0
Tag: kubernetes.azure.com/cluster=MC_xkom-kube-rg_xkom-kube-ops_westeurope
Actual Azure RG name: MC_xkom-kube-rg_xkom-kube-ops_westeurope
Let me know if you'd like some more data.

Is there any workaround ?

Is there any work around? Is autoscaler usable in azure currently?

mohamedalsherif picture mohamedalsherif on 25 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

davidquarles picture davidquarles  路  7Comments
bpinske picture bpinske  路  5Comments
whereisaaron picture whereisaaron  路  7Comments
johanneswuerbach picture johanneswuerbach  路  5Comments
pkelleratwork picture pkelleratwork  路  5Comments