Metrics-server: HPA 'FailedGetScale' no matches for kind "Deployment" in group "extensions"

[winds6206]

Hi

I install metric-server into my Minikube, everything is OK but kubectl get hpa the target always unknown and cause it can't autoscaling.

By the way ,kubectl top pod is in order.

Minikube version: v1.5.2
Metric-server version: 0.3.6

Here is my step when I install metric-server
step1:

kubectl create -f deploy/1.8+/

step2:

kubectl edit deploy -n kube-system metrics-server

Add below

args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

like this

...
    spec:
      containers:
      - args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        name: metrics-server
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
...

My test YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpu-demo-deployment
  namespace: default
  labels:
    app: pod-cpu-demo
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2
  replicas: 2
  progressDeadlineSeconds: 600
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: pod-cpu-demo
  template:
    metadata:
      labels:
        app: pod-cpu-demo
    spec:
      restartPolicy: Always
      containers:
      - name: cpu-demo-ctr
        image: polinux/stress
        imagePullPolicy: Always
        resources:
          limits:
            cpu: "200m"
          requests:
            cpu: "100m"
        command: ["stress"]
        args:
        - --cpu
        - "1"
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-demo-hpa
spec:
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: cpu-demo-deployment
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 30

Is this a bug ?

[serathius]

If kubectl top works, but you HPA has unknown means that your controller-manager has problems with accessing Metrics API.

Can you provide logs from your controller-manager?

[winds6206]

I use the command kubectl logs -n kube-system kube-controller-manager-minikube to fetch logs.

Here is my logs from controller-manager

E1216 12:50:42.622336       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:50:42.622603       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:50:57.895694       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:50:57.896850       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:51:12.897177       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:51:12.897595       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:51:28.153332       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:51:28.153584       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:51:43.154668       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:51:43.155138       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:51:58.409147       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:51:58.409301       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:52:13.409957       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:52:13.410159       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:52:28.666706       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:52:28.666941       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"
E1216 12:52:43.667582       1 horizontal.go:214] unable to determine resource for scale target reference: no matches for kind "Deployment" in group "extensions"
I1216 12:52:43.667841       1 event.go:255] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"cpu-demo-hpa", UID:"3f39c57c-ade6-4dee-9d23-88cbb8b91761", APIVersion:"autoscaling/v2beta2", ResourceVersion:"17511", FieldPath:""}): type: 'Warning' reason: 'FailedGetScale' no matches for kind "Deployment" in group "extensions"

[serathius]

This points to problem with your HPA. Looks there is some API version mismatch between your cpu-demo-deployment deployment, apiserver and controller manager. It's not related to Metrics Server.

You should check your k8s version and when Deployment was moved to apps/v1. I don't know much about compatibility guarantees in this API

[winds6206]

Thank for your help!

Now the HPA is in order.

I modify the hpa YAML , I change the API version from apiVersion: extensions/v1beta1 to apiVersion: apps/v1.

That would be the same as deployment API version.

Here is my HPA YAML after I modified:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-demo-hpa
spec:
  scaleTargetRef:
#    apiVersion: extensions/v1beta1
    apiVersion: apps/v1
    kind: Deployment
    name: cpu-demo-deployment
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 30