Aks-engine: kubectl top nodes: error: metrics not available yet

[szwed]

Describe the bug

➜ kubectl top nodes
error: metrics not available yet

Steps To Reproduce

➜ kubectl top nodes
error: metrics not available yet

metrics server is running:
➜ kubectl get pods --all-namespaces |grep metrics
kube-system        metrics-server-6df44d5569-r2hj2                           1/1     Running            0          2d21h

Expected behavior

Metrics are provided.

AKS Engine version

v1.16.9

Kubernetes version

v1.16.9

Additional context

This is default AKS deployment. Should work out of the box. Networking: Calico.

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[pjoe]

Seeing something similar (also on 1.16.9) ... metrics-server log is full of: no metrics known for pod entries.

I'll try upgrading to 1.17.5 and see if that helps

[pjoe]

Unfortunately didn't help :S This was a cluster that was upgraded from 1.15.11. I'll try tearing it all down and bringing up again directly on 1.16.9 ... to be continued

[szwed]

metrics-server logs:
"Unauthorized", unable to fully scrape metrics from source kubelet_summary:aks-agentpool-57656757-vmss000006: unable to fetch metrics from Kubelet aks-agentpool-57656757-vmss000006 (10.101.0.128): request failed - "401 Unauthorized", response: "Unauthorized", unable to fully scrape metrics from source kubelet_summary:aks-agentpool-57656757-vmss000000: unable to fetch metrics from Kubelet aks-agentpool-57656757-vmss000000 (10.101.0.4): request failed - "401 Unauthorized", response: "Unauthorized"]

[pjoe]

FWIW: After taking the whole cluster down and creating a new one directly on 1.16.9 things are working again

[jackfrancis]

Is this for AKS only, or for AKS Engine as well?

[pjoe]

I'm just using pulumi to bring up a standard AKS cluster

[jackfrancis]

Would you like to file an issue in the AKS github queue? This issue queue is only for AKS Engine.

https://github.com/Azure/AKS/issues

[djsly]

I got hit by the same issue... Found the source, and the fix, but since I'm using AKS I cannot try the fix...

https://github.com/kubernetes-sigs/metrics-server/issues/316

The issue is that when you get a single pod marked as duplicate by the metric-server, nothing gets stored anymore.

IF you restart metrics-server, you are doomed, since you end up with nothing at all. (vs partial metrics when the duplicate pod started)

so we would need metric-server 0.3.6 at least. in AKS Engine followed by AKS

[djsly]

AKS tracking: https://github.com/Azure/AKS/issues/1668

[jackfrancis]

Thanks @djsly, incoming!

[djsly]

pleasure is mine @jackfrancis