Minikube: HPA does not work; HPA current always shows <waiting>

[raghur]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Minikube version (use minikube version): 0.16.0

Environment:

• OS (e.g. from /etc/os-release): Windows 8.1
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): Virtualbox
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep ISO): minikube-v1.0.2.iso
• Install tools:
• Others:

- heapster: enabled
- ingress: disabled
- registry-creds: disabled
- addon-manager: enabled
- dashboard: enabled
- kube-dns: enabled

What happened: Wanted to try out HPA on minikube.

1. enabled heapster
2. Confirmed that heapster is working (grafana dashboard displays and also displays graphs for my pod)
3. set up hpa for a deployment.
4. kubectl get hpa

NAME         REFERENCE               TARGET    CURRENT     MINPODS   MAXPODS   AGE
webapi-app   Deployment/webapi-app   50%       <waiting>   1         3         5m

1. Minikube logs shows that it's unable to get metrics

Feb 06 04:05:35 minikube localkube[3278]: W0206 04:05:35.848786    3278 horizontal.go:104] Failed to reconcile webapi-app: failed to compute desired number of replicas based on CPU utilization for Deployment/default/webapi-app: failed to get CPU utilization: unable to get metrics for resource cpu: no metrics returned from heapster

What you expected to happen:
Should be able to get metrics and scale the service

How to reproduce it (as minimally and precisely as possible):

1. Enable heapster
2. Deploy pod
3. autoscale pod targetting deployment

Anything else do we need to know:
No.

[dlorenc]

We'll try to repro this.

[bobhenkel]

HPA works for me. Though it took a bit before the pods scaled out, docs day 30 seconds but it seemed like it was more like 1.5 to 2.5 minutes maybe more(though watching things for more than 15 seconds starts to feel like a really long time)

1. I installed heapster via this https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/standalone. You will need to update the controller yaml with image tag v1.3.0-beta.1 the version they have in the file is no longer in the google container reg.

1. I used this to test hpa https://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/walkthrough/.

My Setup:
minikube version: v0.16.0
My OS = OSX 10.12.3
minikube-v1.0.6.iso
DriverName = virtualbox

[aaron-prindle]

I have tested hpa and it seems that it does work properly but takes some time for it to come up. I even saw the same error that you did (which is probably worth looking into) but it still seemed to work:
Following the instructions here:
https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/standalone

$ minikube start
$ minikube addons enable heapster
#Create all pods from instructions, no load test yet
$ kubectl get hpa
NAME         REFERENCE               TARGET    CURRENT   MINPODS   MAXPODS   AGE
php-apache   Deployment/php-apache   50%       <waiting>      1         10        1m

#I saw the same error that you saw, kept waiting on kubectl get hpa...
$ minikube logs | grep heapster
Feb 14 18:30:27 minikube localkube[3435]: W0214 18:30:27.400478    3435 horizontal.go:104] Failed to reconcile php-apache: failed to compute desired number of replicas based on CPU utilization for Deployment/default/php-apache: failed to get CPU utilization: unable to get metrics for resource cpu: no metrics returned from heapster

#  Wait ~5 mins
$ kubectl get hpa
NAME         REFERENCE               TARGET    CURRENT   MINPODS   MAXPODS   AGE
php-apache   Deployment/php-apache   50%       0%      1         10        5m

# Start load test
$ kubectl get hpa
NAME         REFERENCE               TARGET    CURRENT   MINPODS   MAXPODS   AGE
php-apache   Deployment/php-apache   50%       7%      1         10        8m

$ kubectl get hpa
NAME         REFERENCE               TARGET    CURRENT   MINPODS   MAXPODS   AGE
php-apache   Deployment/php-apache   50%       471%      1         10        8m

# See that pods have scaled out
aprindle@aprindle:~$ kubectl get po
NAME                              READY     STATUS    RESTARTS   AGE
load-generator-1930141919-nqp12   1/1       Running   0          2m
php-apache-3815965786-b7zbb       1/1       Running   0          11m
php-apache-3815965786-cg0nb       1/1       Running   0          1m
php-apache-3815965786-hl2gs       1/1       Running   0          1m
php-apache-3815965786-wjsgm       1/1       Running   0          1m

# Stop load test
$ kubectl get hpa
NAME         REFERENCE               TARGET    CURRENT   MINPODS   MAXPODS   AGE
php-apache   Deployment/php-apache   50%       0%        1         10        12m

# We can see that pods have scaled down
aprindle@aprindle:~$ kubectl get po
NAME                              READY     STATUS    RESTARTS   AGE
load-generator-1930141919-nqp12   1/1       Running   1          9m
php-apache-3815965786-b7zbb       1/1       Running   0          12m

[raghur]

In my case, the nodes were never able to get current cpu...
Just kept seeing the same error in minikube logs.. I'm pretty sure I waited more than an hour. Are there any other logs that'll help?

[aaron-prindle]

For custom metrics to work for HPA, minikube has to be started with the kubelet.EnableCustomMetrics flag:
minikube start --extra-config kubelet.EnableCustomMetrics=true

[raghur]

I tried with minikube start --extra-config kubelet.EnableCustomMetrics=true - but still no current cpu utilization - still the same error minikube logs output

[aaron-prindle]

How are you defining the /etc/custom-metrics/definition.json file? See:
https://medium.com/@marko.luksa/kubernetes-autoscaling-based-on-custom-metrics-without-using-a-host-port-b783ed6241ac#.2t3jt4psv

It seems there are some caveats regarding the host network namespace.

[raghur]

@aaron-prindle I'm a bit confused😕 - I'm just trying to get HPA running on minikube with the default CPU utilization metric. I'm not using any custom metric for HPA scaling (though thanks for the link - that'll be the next step after I get basic HPA going).
On an Azure cluster, HPA worked without any additional tweaks.

[puja108]

Just to make sure, do you have a CPU request on your pod? If not HPA is going to stay in waiting as it works on "percentage of requested CPU". Without a request it fails to calculate that percentage.

[raghur]

Ah... that was it... I didn't have resource limits in there :(... Once I added them in, HPA works. Thanks for the pointer!

[ccwillia]

I have a CPU request set for my pod and i am facing this issue.

Requests:
  cpu:      100m
  memory:       1G

TARGET          CURRENT
 85%            <waiting>

[andyp1per]

This is really not working for me on OSX with xhyve:

minikube addons enable heapster
minikube start --vm-driver=xhyve --v=5 --show-libmachine-logs
kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
kubectl get hpa

gives:

NAME         REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown> / 50%   1         10        0          7s

It never gets the CPU. I tried deploying https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/standalone but that gives the error:

The Service "heapster" is invalid: spec.ports[0].nodePort: Invalid value: 31572: may not be used when `type` is 'ClusterIP'

[praparn]

From testing step below. I'm still facing this problem on OSX with virtualbox. Please kindly suggestion if i wrong or miss some step.

Environment
MAC OS: 10.12.5 (Sierra)
Minikube: v0.19.0
Driver: VirtualBox

---

webtest_deploy_hpa.yml

spec:
containers:
- name: webtest
image: labdocker/cluster:webservicelite_v1
resources:
requests:
memory: "16Mi"
cpu: 100m
limits:
memory: "32Mi"
cpu: "1"
ports:
- containerPort: 5000

protocol: TCP

Step to Test

1. minikube start --vm-driver=virtualbox profile=minikubelab1 --iso-url=https://storage.googleapis.com/minikube/iso/minikube-v0.20.0.iso --extra-config kubelet.EnableCustomMetrics=true

1. kubectl create -f heapster-controller.yaml
   Result:
   serviceaccount "heapster" created
   deployment "heapster" created
   service "heapster" created

2. minikube addons enable heapster
   Result:
   heapster was successfully enabled

3. kubectl create -f webtest_deploy_hpa.yml --record
   Result:
   service "webtest" created
   deployment "webtest" created

4. kubectl get pods
   Result:
   NAME READY STATUS RESTARTS AGE
   webtest-162143909-6khbs 1/1 Running 0 6m

5. kubectl get svc
   Result:
   NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
   kubernetes 10.0.0.1 443/TCP 12h
   webtest 10.0.0.108 5000:32500/TCP 7m

6. kubectl autoscale deployment webtest --cpu-percent=15 --min=1 --max=5
   Result:
   deployment "webtest" autoscaled

7. kubectl get hpa
   Result:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
webtest Deployment/webtest <unknown> / 15% 1 5 0 6m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
webtest Deployment/webtest <unknown>/ 15% 1 5 0 7m
...
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
webtest Deployment/webtest <unknown> / 15% 1 5 1 14m