Azure-docs: unable to get metrics

[hadr10]

I have got this error message: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 6ab10c96-1f2f-e48f-a084-b1715156e33f
• Version Independent ID: a113b8e9-259a-2a65-2e54-a2661c78ddb5
• Content: Kubernetes on Azure tutorial - Scale Application
• Content Source: articles/aks/tutorial-kubernetes-scale.md
• Service: container-service
• Product: unspecified
• GitHub Login: @dlepow
• Microsoft Alias: danlep

[mimckitt]

@hadr10 Thanks for the feedback! We are currently investigating and will update you shortly.

[mimckitt]

@hadr10 at this step in the document are you getting this error?

[hadr10]

I set autoscale as you write here (kubectl autoscale deployment azure-vote-front --cpu-percent=50 --min=3 --max=10) but nothing happens after one hour, so I tried to investigate problem. When I run command kubectl describe hpa I get this error. This error message I can see also in dashboard for pods

[mimckitt]

Got it. @dlepow or @neilpeterson could either if you help on this one?

[neilpeterson]

@had10 can you paste in the error message that you are receiving when inspecting the HPA resource?

Can you also past in the results of:

kubectl get pods --all-namespaces

I will attempt to reproduce the issue ASAP.

[hadr10]

@neilpeterson I recreated the environment today and here are the logs:
kubectl describe hpa
Name: azure-vote-front
Namespace: default
Labels:
Annotations:
CreationTimestamp: Tue, 29 May 2018 08:18:18 +0200
Reference: Deployment/azure-vote-front
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): / 50%
Min replicas: 3
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics
for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.
k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 25s (x13 over 6m) horizontal-pod-autoscaler unable to get metrics for resourc
e cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedComputeMetricsReplicas 25s (x13 over 6m) horizontal-pod-autoscaler failed to get cpu utilization: un
able to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource
(get pods.metrics.k8s.io)
PS C:\Azure\powershell scripts> kubectl describe hpa
Name: azure-vote-front
Namespace: default
Labels:
Annotations:
CreationTimestamp: Tue, 29 May 2018 08:18:18 +0200
Reference: Deployment/azure-vote-front
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): / 50%
Min replicas: 3
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics
for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.
k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 19m (x31 over 34m) horizontal-pod-autoscaler failed to get cpu utilization: u
nable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resourc
e (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 4m (x61 over 34m) horizontal-pod-autoscaler unable to get metrics for resour
ce cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
azure-vote-front Deployment/azure-vote-front / 50% 3 10 5 35m

kubectl get pods
NAME READY STATUS RESTARTS AGE
azure-vote-back-655476c7f7-5f9pk 1/1 Running 0 37m
azure-vote-front-78b9c759df-6fhlw 1/1 Running 0 36m
azure-vote-front-78b9c759df-b7k5b 1/1 Running 0 36m
azure-vote-front-78b9c759df-c94c4 1/1 Running 0 36m
azure-vote-front-78b9c759df-lrx8g 1/1 Running 0 37m
azure-vote-front-78b9c759df-wgvww 1/1 Running 0 36m

kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default azure-vote-back-655476c7f7-5f9pk 1/1 Running 0 38m
default azure-vote-front-78b9c759df-6fhlw 1/1 Running 0 36m
default azure-vote-front-78b9c759df-b7k5b 1/1 Running 0 36m
default azure-vote-front-78b9c759df-c94c4 1/1 Running 0 36m
default azure-vote-front-78b9c759df-lrx8g 1/1 Running 0 38m
default azure-vote-front-78b9c759df-wgvww 1/1 Running 0 36m
kube-system heapster-7cd8dd888b-fb9c4 1/2 CrashLoopBackOff 73 3d
kube-system kube-dns-v20-7c556f89c5-fgsdf 3/3 Running 3 3d
kube-system kube-dns-v20-7c556f89c5-w24ps 3/3 Running 3 3d
kube-system kube-proxy-7wdxr 1/1 Running 1 3d
kube-system kube-svc-redirect-w82pn 1/1 Running 1 3d
kube-system kubernetes-dashboard-546f987686-t7qbg 1/1 Running 1 3d
kube-system omsagent-w6wb8 1/1 Running 3 3d
kube-system tunnelfront-694975d99d-qh2xp 1/1 Running 1 3d

[neilpeterson]

@hadr10 - thanks for the info. This looks like a know issue and is being tracked here:

https://github.com/Azure/AKS/issues/318

I am going to close this issue in favor of #318. Can you proceed to follow issue #318 as well?

Thanks

please-close

[neilpeterson]

@hadr10 - to follow up on this, I've spoken with the product group. As expected, the issue is related to the crashing Heapster pod. A fix has been merged and should roll out shortly.

@MicahMcKittrick-MSFT - can you assign this one to myself?

Thanks

[mimckitt]

@neilpeterson done and done. Let me know if you need it reopened or prefer it to be closed.

[neilpeterson]

@MicahMcKittrick-MSFT - nope, issue can remain closed. Just wanted to keep it in my bucket for easy retrieval / tracking.

Thanks

[zixo]

Hi, is this fixed? I have same problem :/

Thanks

[neilpeterson]

@zixo - can you provide the output of kubectl gt pods --all-namespaces?

Thanks

[zixo]

kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
azure-vote-front Deployment/azure-vote-front /50% 3 10 5 31m

kubectl describe hpa
Name: azure-vote-front
Namespace: default
Labels:
Annotations:
CreationTimestamp: Mon, 11 Jun 2018 17:24:31 +0200
Reference: Deployment/azure-vote-front
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): / 50%
Min replicas: 3
Max replicas: 10
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 23m (x13 over 29m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 4m (x51 over 29m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)

kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default azure-vote-back-655476c7f7-wjclc 1/1 Running 0 1h
default azure-vote-front-656bdb6b48-264hv 1/1 Running 0 42m
default azure-vote-front-656bdb6b48-npthp 1/1 Running 0 42m
default azure-vote-front-656bdb6b48-qsfd4 1/1 Running 0 1h
default azure-vote-front-656bdb6b48-tbz8h 1/1 Running 0 42m
default azure-vote-front-656bdb6b48-tp5zk 1/1 Running 0 42m
kube-system azureproxy-79c5db744-qcqcx 1/1 Running 2 2h
kube-system heapster-55f855b47-4hs79 2/2 Running 0 2h
kube-system kube-dns-v20-7c556f89c5-x52t7 3/3 Running 0 2h
kube-system kube-dns-v20-7c556f89c5-z2t7w 3/3 Running 0 2h
kube-system kube-proxy-5js8x 1/1 Running 0 58m
kube-system kube-proxy-cdhb9 1/1 Running 0 2h
kube-system kube-proxy-rwgnk 1/1 Running 0 59m
kube-system kube-svc-redirect-58stf 1/1 Running 0 2h
kube-system kube-svc-redirect-6m4xl 1/1 Running 0 58m
kube-system kube-svc-redirect-9dbqj 1/1 Running 0 59m
kube-system kubernetes-dashboard-546f987686-t5qq5 1/1 Running 3 2h
kube-system tunnelfront-86d67d7754-gt78c 1/1 Running 0 2h

[neilpeterson]

Seems like this is a know issue and is being tracked by the product group here - https://github.com/Azure/AKS/issues/318. There are some workarounds in this issue as well.

[NKUCodingCat]

Hey guys,

It seems I met something seems similar here

Is it what AKS#318 met? or just caused by mis-configurate in somewhere? I am a literally newbie of AKS & K8s

$ kubectl describe hpa
Error from server (NotFound): the server could not find the requested resource


$ kubectl get hpa
NAME               REFERENCE                     TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
azure-vote-front   Deployment/azure-vote-front   <unknown>/50%   1         10        1          26m

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
default       azure-vote-back-655476c7f7-jclw7       1/1     Running   0          37m
default       azure-vote-front-5fc94b6696-h987b      1/1     Running   0          37m
kube-system   heapster-779db6bd48-9hpjd              2/2     Running   0          2d23h
kube-system   kube-dns-autoscaler-5b94d96445-gkqjz   1/1     Running   0          2d23h
kube-system   kube-dns-v20-6df5d5c657-8q6xf          4/4     Running   0          2d23h
kube-system   kube-dns-v20-6df5d5c657-nlzt7          4/4     Running   0          2d23h
kube-system   kube-proxy-qn9md                       1/1     Running   0          2d23h
kube-system   kube-svc-redirect-4fgxp                2/2     Running   0          2d23h
kube-system   kubernetes-dashboard-f49587769-zrt6f   1/1     Running   1          2d23h
kube-system   metrics-server-9b5f7d9bb-whb48         1/1     Running   0          64m
kube-system   omsagent-mdthk                         1/1     Running   0          2d23h
kube-system   omsagent-rs-56dd67c559-s2pcn           1/1     Running   0          2d23h
kube-system   tunnelfront-59b5fbc568-n8kz2           1/1     Running   0          2d23h