Kubespray: HorizontalPodAutoscaler FailedGetResourceMetric, "unable to get metrics for resource cpu"

[rafi]

Horizontal Pod Autoscaler (HPA) fails to collect metrics after setup, for example:

$ kubectl get --all-namespaces event
NAMESPACE      LAST SEEN   FIRST SEEN   COUNT     NAME                                    KIND                      SUBOBJECT   TYPE      REASON                         SOURCE                      MESSAGE
istio-system   1m          11d          34210     istio-egressgateway.153e74efecb3de28    HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   6m          11d          34200     istio-ingress.153e74eff8ffd9e9          HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   1m          11d          34210     istio-ingress.153e74eff95336d2          HorizontalPodAutoscaler               Warning   FailedComputeMetricsReplicas   horizontal-pod-autoscaler   failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
istio-system   1m          11d          34210     istio-ingressgateway.153e74f0059bb205   HorizontalPodAutoscaler               Warning   FailedGetResourceMetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

BUG REPORT

Environment:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:59:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: VMWare

• OS:

Linux 3.10.0-862.el7.x86_64 x86_64
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Applications"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"

• Version of Ansible (ansible --version): 2.5.5

Kubespray version (commit): 0b939a49

Network plugin used: Calico

Copy of your inventory file:

[all]
vrnv02ax04315    ansible_host=10.213.236.115 ip=10.213.236.115
vrnv02ax04316    ansible_host=10.213.236.116 ip=10.213.236.116
vrnv02ax04317    ansible_host=10.213.236.117 ip=10.213.236.117
vrnv02ax04318    ansible_host=10.213.236.118 ip=10.213.236.118

[kube-master]
vrnv02ax04315    
vrnv02ax04316    

[kube-node]
vrnv02ax04315    
vrnv02ax04316    
vrnv02ax04317    
vrnv02ax04318    

[etcd]
vrnv02ax04315    
vrnv02ax04316    
vrnv02ax04317    

[k8s-cluster:children]
kube-node    
kube-master      

[calico-rr]

[vault]
vrnv02ax04315    
vrnv02ax04316    
vrnv02ax04317    

Command used to invoke ansible:

ansible-playbook -i inventory/stg/hosts.ini cluster.yml -b -u kube --private-key=~/.ssh/kube_rsa -v

[monkeydaichan]

You deployed "metrics-server"?
https://github.com/cloudfoundry-incubator/kubo-deployment/issues/263

$ git clone https://github.com/kubernetes-incubator/metrics-server.git
$ cd metrics-server
$ kubectl create -f deploy/1.8+/

[rafi]

@monkeydaichan yes.

It started working when I upgraded to Istio 1.0.0