Minikube: HPA not working, unknown target CPU

[devarajchidambaram]

Im trying to auto scale the kuber pods but not able to get the hpa cpu metrics . so its failing to auto scale.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080 

Ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-tutorial-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-tutorial-cluster-ip
          servicePort: 80

Services.yaml

apiVersion: v1
kind: Service
metadata:
  name: kubernetes-tutorial-cluster-ip
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: kubernetes-tutorial-deployment
  type: ClusterIP

CMD output

D:\docker\kubernetes-tutorial>kubectl get all
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/kubernetes-tutorial-deployment-7b78fb65c7-5dvz2   1/1     Running   0          2d18h

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes                       ClusterIP   10.96.0.1        <none>        443/TCP   2d21h
service/kubernetes-tutorial-cluster-ip   ClusterIP   10.108.184.150   <none>        80/TCP    2d20h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubernetes-tutorial-deployment   1/1     1            1           2d20h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/kubernetes-tutorial-deployment-7b78fb65c7   1         1         1       2d18h

NAME                                                                 REFERENCE                                   TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   <unknown>/2%   1         4         1          122m

Here horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment showing target is UNKNOWN.

This is the error I got

D:\docker\kubernetes-tutorial>kubectl describe hpa kubernetes-tutorial-deployment
Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48 +0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Im new to the kubernetes . so I don't know , how to proceed further.

[afbjorklund]

Did you deploy metrics-server ?

[devarajchidambaram]

Yes @afbjorklund ...

[devarajchidambaram]

Finally I did like this ...now its working

https://stackoverflow.com/questions/54106725/docker-kubernetes-mac-autoscaler-unable-to-find-metrics

[tstromberg]

It's unclear what the issue is with minikube is here? Are we providing a broken metrics-server?

[afbjorklund]

Seems like there is some kind of issue with certificates and/or IP, workaround is disabling those:

--kubelet-insecure-tls

--kubelet-preferred-address-types=InternalIP

Could be related to a broken DNS (somehow), or could be our broken InternalIP (vs. hostname) ?

[salsaverde123]

I have same problem but the solution of @devarajchidambaram didn't work for me.

[nikhils5501]

The deployment file should contain resource limits in order for the metrics-server to calculate the CPU utilization.
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
Edit your deployment.yaml and add the resource limits part in order to fix this.

To read more on this, refer to the autoscaler algorithm:
Please note that if some of the pod’s containers do not have the relevant resource request set, CPU utilization for the pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

[highway900]

@nikhils5501 Thanks, this solved it for me. More specifically I added the metrics section to the config for service-pod.yml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: {{ .Values.name }}
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ .Values.name }}
  minReplicas: {{ .Values.replicaCount }}
  maxReplicas: 16
  targetCPUUtilizationPercentage: 70
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 70

[cristian04]

I can reproduce this on:
minikube version: v1.5.2
Running k8s v1.16.2

I tried adding
--kubelet-insecure-tls --kubelet-preferred-address-types=InternalIP

but it didn't work. Minikube keeps restoring metrics-server to the default values.

[medyagh]

@cristian04
does the solution mentioned above help ?
https://github.com/kubernetes/minikube/issues/4456#issuecomment-558436618

[cristian04]

@cristian04
does the solution mentioned above help ?
#4456 (comment)

Not really. I am having this issue when I add metrics into the hpa.

* : Invalid value: "The edited file failed validation": ValidationError(HorizontalPodAutoscaler): unknown field "metrics" in io.k8s.api.autoscaling.v1.HorizontalPodAutoscaler

I think is related to this bug

[nikhils5501]

@cristian04 You need to have metrics-server installed on your cluster. I hope that should solve it for you. Metrics-server is the alternative for Heapster.

[cristian04]

@cristian04 You need to have metrics-server installed on your cluster. I hope that should solve it for you. Metrics-server is the alternative for Heapster.

Yes, I am using metric-server and it's running on my cluster. Not using heapster as it is deprecated.

Updates:

I've retested on:

Minikube v1.6.1 + metric-server
K8S= 1.17.0
Result: Error Working 😄
Error: Issue starting minikube to start a 1.7.0 cluster "💣 Unable to parse "": Version string empty"

Metrics started working after 5 minutes of seeing this:

W1217 19:09:09.037972   12866 top_pod.go:266] Metrics not available for pod default/nginx-f5f948997-
6cw7s, age: 2m28.037948s
error: Metrics not available for pod default/nginx
-f5f948997-6cw7s, age: 2m28.037948s

---

Minikube v1.5.2 + metric-server:
K8S: v1.16.2

Result: Not working
Error: Metrics are not available:

kubectl top pod
W1217 18:13:05.325281   40088 top_pod.go:266] Metrics not available for pod default/nginx-f5f948997-9
qn9f, age: 7m7.32526s
error: Metrics not available for pod default/nginx-f5f948997-9qn9f, age: 7m7.32526s

HPA with unknown status:

kubectl get hpa 
NAME    REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx   Deployment/nginx   <unknown>/50%   2         10        2          6m54s

metric-server log

I1217 21:18:13.159257       1 reststorage.go:93] No metrics for pod default/nginx-f5f948997-76zrd

---

Minikube v1.3.0 + metrics-server
K8s v1.15.2

Result: Working without workarounds

---

Steps

• minikube start --memory 4096 --cpus 2 --vm-driver=hyperkit
• minikube addons enable metrics-server
• kubectl run nginx --image nginx --port 80
• kubectl set resources deployment nginx -c=nginx --limits=cpu=100m,memory=64Mi
• kubectl autoscale deployment nginx --max=10 --cpu-percent=50 --min=5

Verify with:

• kubectl top pod && kubectl top node && kubectl get hpa

[cristian04]

Fix for Minikube v1.5.2 + K8S v1.16.2

Disable the metric-server addon and install from https://github.com/kubernetes-sigs/metrics-server

$ minikube addons disable metrics-server
$ kubectl create -f deploy/1.8+/
# edit metric-server deployment to add the flags
# args:
# - --kubelet-insecure-tls
# - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
$ kubectl edit deploy -n kube-system metrics-server

I think we can close the issue with that

[medyagh]

@cristian04 @devarajchidambaram
@dshepherd @infra-ddeore

we just merged a PR that improves the metrics-server addon, (it is on the master and will be in the next release)

do you mind checking the minikube that would help with this issue ?

https://github.com/kubernetes/minikube/pull/6322

[cristian04]

@medyagh Yeah, Im fine with that. Feel free to assign this ticket to me if required.

[govargo]

I think this issue had been resolved.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-tutorial-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-tutorial-deployment
  template:
    metadata:
      labels:
        app: kubernetes-tutorial-deployment
    spec:
      containers:
      - name: kubernetes-tutorial-application
        image: devarajc/tomcat_helloworld
        resources:
          requests:
             cpu: 500m
        ports:
          - containerPort: 8080 
EOF

and I confirmed HPA is working.

$ kubectl autoscale deployment kubernetes-tutorial-deployment --cpu-percent=50 --min=1 --max=4
horizontalpodautoscaler.autoscaling/kubernetes-tutorial-deployment autoscaled

$ kubectl get hpa
NAME                             REFERENCE                                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
kubernetes-tutorial-deployment   Deployment/kubernetes-tutorial-deployment   1%/50%    1         4         1          22s

[ryuheechul]

In case it's not obvious how to fix unknown, what you most likely be missing is following:

kubectl set resources deployment nginx -c=nginx --limits=cpu=100m,memory=64Mi