Kops: Kubectl top nodes not working with the metrics server

[minasys]

1. What kops version are you running? The command kops version, will display
this information.
Version 1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.10.6

3. What cloud provider are you using?
aws

4. What commands did you run? What is the simplest way to reproduce this issue?

5. What happened after the commands executed?
kubectl top nodes

6. What did you expect to happen?
to see top nodes with their cpu consumtion

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
creationTimestamp: 2018-12-04T02:19:00Z
name: EXAMPLE.com
spec:
additionalPolicies:
master: "\n[ \n\n { \n\n "Action": [ \n\n "route53:ChangeResourceRecordSets"\n\n
\ ], \n\n "Resource": [ md5-c0f0ee7eb1c879b136a7f6d33d6d40ebarn:aws:route53:::hostedzone/" ], \n\n
\ "Effect": "Allow" \n\n }, \n\n { \n\n "Action": [ \n\n "route53:ListHostedZones",
\n\n "route53:ListResourceRecordSets"\n\n ], \n\n "Resource":
[ "" ], \n\n "Effect": "Allow" \n\n }\n\n]\n"
node: "\n[ \n { \n\n "Action": [ \n\n "route53:ChangeResourceRecordSets"\n\n
\ ], \n\n "Resource": [ md5-c0f0ee7eb1c879b136a7f6d33d6d40ebarn:aws:route53:::hostedzone/" ], \n\n
\ "Effect": "Allow" \n\n }, \n\n { \n\n "Action": [ \n\n "route53:ListHostedZones",\n\n
\ "route53:ListResourceRecordSets"\n\n ], \n\n "Resource": [
"" ], \n\n "Effect": "Allow" \n\n },\n\n { \n\n "Action": [
\n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebacm:DescribeCertificate", \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebacm:ListCertificates", \n\n
\ md5-c0f0ee7eb1c879b136a7f6d33d6d40ebacm:GetCertificate" \n\n ], \n\n "Resource": [ "" ], \n\n
\ "Effect": "Allow" \n\n }, \n\n { \n\n "Action": [ \n\n "ec2:AuthorizeSecurityGroupIngress",
\n\n "ec2:CreateSecurityGroup", \n\n "ec2:CreateTags", \n\n "ec2:DeleteTags",
\n\n "ec2:DeleteSecurityGroup", \n\n "ec2:DescribeInstances",
\n\n "ec2:DescribeInstanceStatus", \n\n "ec2:DescribeSecurityGroups",
\n\n "ec2:DescribeSubnets", \n\n "ec2:DescribeTags", \n\n "ec2:DescribeVpcs",
\n\n "ec2:ModifyInstanceAttribute", \n\n "ec2:ModifyNetworkInterfaceAttribute",
\n\n "ec2:RevokeSecurityGroupIngress" \n\n ], \n\n "Resource":
[ "" ], \n\n "Effect": "Allow" \n\n }, \n\n { \n\n "Effect":
"Allow", \n\n "Action": [ \n\n "elasticloadbalancing:AddTags",
\n\n "elasticloadbalancing:CreateListener", \n\n "elasticloadbalancing:CreateLoadBalancer",
\n\n "elasticloadbalancing:CreateRule", \n\n "elasticloadbalancing:CreateTargetGroup",
\n\n "elasticloadbalancing:DeleteListener", \n\n "elasticloadbalancing:DeleteLoadBalancer",
\n\n "elasticloadbalancing:DeleteRule", \n\n "elasticloadbalancing:DeleteTargetGroup",
\n\n "elasticloadbalancing:DeregisterTargets", \n\n "elasticloadbalancing:DescribeListeners",
\n\n "elasticloadbalancing:DescribeLoadBalancers", \n\n "elasticloadbalancing:DescribeLoadBalancerAttributes",
\n\n "elasticloadbalancing:DescribeRules", \n\n "elasticloadbalancing:DescribeSSLPolicies",
\n\n "elasticloadbalancing:DescribeTags", \n\n "elasticloadbalancing:DescribeTargetGroups",
\n\n "elasticloadbalancing:DescribeTargetGroupAttributes", \n\n "elasticloadbalancing:DescribeTargetHealth",
\n\n "elasticloadbalancing:ModifyListener", \n\n "elasticloadbalancing:ModifyLoadBalancerAttributes",
\n\n "elasticloadbalancing:ModifyRule", \n\n "elasticloadbalancing:ModifyTargetGroup",
\n\n "elasticloadbalancing:ModifyTargetGroupAttributes", \n\n "elasticloadbalancing:RegisterTargets",
\n\n "elasticloadbalancing:RemoveTags", \n\n "elasticloadbalancing:SetIpAddressType",
\n\n "elasticloadbalancing:SetSecurityGroups", \n\n "elasticloadbalancing:SetSubnets",
\n\n "elasticloadbalancing:SetWebACL" \n\n ], \n\n "Resource":
[""] \n\n }, \n\n { \n\n "Effect": "Allow", \n\n "Action":
[ \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebiam:GetServerCertificate", \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebiam:ListServerCertificates"
\n\n ], \n\n "Resource": [""] \n\n }, \n\n { \n\n "Effect":
"Allow", \n\n "Action": [ \n\n "waf-regional:GetWebACLForResource",
\n\n "waf-regional:GetWebACL", \n\n "waf-regional:AssociateWebACL",
\n\n "waf-regional:DisassociateWebACL" \n\n ], \n\n "Resource":
[""] \n\n }, \n\n { \n\n "Effect": "Allow", \n\n "Action":
[ \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebtag:GetResources", \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebtag:TagResources" \n\n ],
\n\n "Resource": [""] \n\n }, \n\n { \n\n "Effect": "Allow",
\n\n "Action": [ \n\n md5-c0f0ee7eb1c879b136a7f6d33d6d40ebwaf:GetWebACL" \n\n ], \n\n "Resource":
["*"] \n\n } \n\n] \n"
api:
dns: {}
authorization:
rbac: {}
channel: stable
cloudProvider: aws
configBase: s3://BUCKET_NAME/SUFFIX
dnsZone: DOMAIN.com
etcdClusters:

• etcdMembers:
  
  
  
  • instanceGroup: master-us-east-1a
    
    name: a
    
    name: main
  
  
• etcdMembers:
  
  
  
  • instanceGroup: master-us-east-1a
    
    name: a
    
    name: events
    
    iam:
    
    allowContainerRegistry: true
    
    legacy: false
    
    kubeAPIServer:
    
    runtimeConfig:
    
    autoscaling/v2beta1: "true"
    
    kubeControllerManager:
    
    horizontalPodAutoscalerUseRestClients: true
    
    kubelet:
    
    enableCustomMetrics: true
    
    kubernetesApiAccess:
  
  
• 0.0.0.0/0
  kubernetesVersion: 1.10.6
  masterInternalName: api.internal.k8s-clusters.orioneclipse.com
  masterPublicName: api.k8s-clusters.orioneclipse.com
  networkCIDR: 10.0.0.0/16
  networkID: vpc-XXXXX
  networking:
  kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
• 0.0.0.0/0
  subnets:
• cidr: 10.0.0.0/23
  id: subnet-XXXX
  name: us-east-1a
  type: Public
  zone: us-east-1a
• cidr: 10.0.4.0/23
  id: subnet-XXXXX
  name: us-east-1b
  type: Public
  zone: us-east-1b
  topology:
  dns:
  type: Public
  masters: public
  nodes: public

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-12-04T02:19:00Z
labels:
kops.k8s.io/cluster: k8s-clusters.DOMAIN.com
name: master-us-east-1a
spec:
image: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17
machineType: m3.medium
maxSize: 1
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: master-us-east-1a
role: Master
subnets:

• us-east-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
creationTimestamp: 2018-12-04T02:19:00Z
labels:
kops.k8s.io/cluster: k8s-DOMAIN.com
name: nodes
spec:
image: kope.io/k8s-1.10-debian-jessie-amd64-hvm-ebs-2018-08-17
machineType: t2.medium
maxSize: 2
minSize: 1
nodeLabels:
kops.k8s.io/instancegroup: nodes
role: Node
subnets:

• us-east-1a
• us-east-1b

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

kubectl top nodes -v 10

I1204 19:56:14.859714 4920 loader.go:359] Config loaded from file /home/centos/.kube/config
I1204 19:56:14.862137 4920 loader.go:359] Config loaded from file /home/centos/.kube/config
I1204 19:56:14.862665 4920 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7" -H "Authorization: Basic YWRtaW46RzBRaDlvdDlaS25oVEkxaE5nV0ltczMwdTZYb2tSa2E=" -H "Accept: application/json, /" 'https://api.k8s-clusters.orioneclipse.com/api?timeout=32s'
I1204 19:56:14.897824 4920 round_trippers.go:405] GET https://api.k8s-clusters.orioneclipse.com/api?timeout=32s 200 OK in 35 milliseconds
I1204 19:56:14.897843 4920 round_trippers.go:411] Response Headers:
I1204 19:56:14.897850 4920 round_trippers.go:414] Content-Type: application/json
I1204 19:56:14.897856 4920 round_trippers.go:414] Content-Length: 134
I1204 19:56:14.897862 4920 round_trippers.go:414] Date: Tue, 04 Dec 2018 19:56:14 GMT
I1204 19:56:14.897897 4920 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"10.0.0.247:443"}]}
I1204 19:56:14.898118 4920 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7" -H "Authorization: Basic YWRtaW46RzBRaDlvdDlaS25oVEkxaE5nV0ltczMwdTZYb2tSa2E=" 'https://api.k8s-clusters.orioneclipse.com/apis?timeout=32s'
I1204 19:56:14.899655 4920 round_trippers.go:405] GET https://api.k8s-clusters.orioneclipse.com/apis?timeout=32s 200 OK in 1 milliseconds
I1204 19:56:14.899666 4920 round_trippers.go:411] Response Headers:
I1204 19:56:14.899672 4920 round_trippers.go:414] Content-Type: application/json
I1204 19:56:14.899676 4920 round_trippers.go:414] Content-Length: 3746
I1204 19:56:14.899763 4920 round_trippers.go:414] Date: Tue, 04 Dec 2018 19:56:14 GMT
I1204 19:56:14.900242 4920 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"},{"groupVersion":"authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"},{"groupVersion":"autoscaling/v2beta1","version":"v2beta1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"},{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"},"serverAddressByClientCIDRs":null},{"name":"admissionregistration.k8s.io","versions":[{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":null}]}
I1204 19:56:14.900611 4920 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "Authorization: Basic YWRtaW46RzBRaDlvdDlaS25oVEkxaE5nV0ltczMwdTZYb2tSa2E=" -H "User-Agent: kubectl/v1.12.2 (linux/amd64) kubernetes/17c77c7" 'https://api.k8s-clusters.orioneclipse.com/api/v1/namespaces/kube-system/services/http:heapster:/proxy/apis/metrics/v1alpha1/nodes?labelSelector='
I1204 19:56:14.903106 4920 round_trippers.go:405] GET https://api.k8s-clusters.orioneclipse.com/api/v1/namespaces/kube-system/services/http:heapster:/proxy/apis/metrics/v1alpha1/nodes?labelSelector= 404 Not Found in 2 milliseconds
I1204 19:56:14.903119 4920 round_trippers.go:411] Response Headers:
I1204 19:56:14.903124 4920 round_trippers.go:414] Content-Type: application/json
I1204 19:56:14.903128 4920 round_trippers.go:414] Content-Length: 192
I1204 19:56:14.903133 4920 round_trippers.go:414] Date: Tue, 04 Dec 2018 19:56:14 GMT
I1204 19:56:14.903152 4920 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services "heapster" not found","reason":"NotFound","details":{"name":"heapster","kind":"services"},"code":404}
I1204 19:56:14.903299 4920 helpers.go:201] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server could not find the requested resource (get services http:heapster:)",
"reason": "NotFound",
"details": {
"name": "http:heapster:",
"kind": "services",
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "unknown"
}
]
},
"code": 404
}]
F1204 19:56:14.903327 4920 helpers.go:119] Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

9. Anything else do we need to know?
I added the command lines to the metrics-server deployment and it started to collect metrics after I was getting an error

---

apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server

namespace: kube-system

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.1
imagePullPolicy: Always
volumeMounts:
- name: tmp-dir
mountPath: /tmp
command:
- /metrics-server
- --kubelet-insecure-tls
- --v=10
- --kubelet-preferred-address-types=InternalIP

and here is the tail log of the metrics server taht shows that it is working:
"workingSetBytes": 1695744,
"rssBytes": 1478656,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2018-12-04T19:59:43Z",
"name": "eth0",
"rxBytes": 939458,
"rxErrors": 0,
"txBytes": 6643165,
"txErrors": 0,
"interfaces": [
{
"name": "eth0",
"rxBytes": 939458,
"rxErrors": 0,
"txBytes": 6643165,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2018-12-04T04:54:34Z",
"availableBytes": 2073206784,
"capacityBytes": 2073219072,
"usedBytes": 12288,
"inodesFree": 506148,
"inodes": 506157,
"inodesUsed": 9,
"name": "default-token-vlg59"
}
],
"ephemeral-storage": {
"time": "2018-12-04T19:59:38Z",
"availableBytes": 119596871680,
"capacityBytes": 128767627264,
"usedBytes": 249856,
"inodesFree": 33353038,
"inodes": 33554432,
"inodesUsed": 28
}
}
]
}
I1204 19:59:43.981894 1 manager.go:150] ScrapeMetrics: time: 26.036208ms, nodes: 2, pods: 20
I1204 19:59:43.981909 1 manager.go:115] ...Storing metrics...
I1204 19:59:43.981925 1 manager.go:126] ...Cycle complete
I1204 20:00:06.144731 1 reflector.go:428] github.com/kubernetes-incubator/metrics-server/vendor/k8s.io/client-go/informers/factory.go:130: Watch close - v1.Node total 80 items received
I1204 20:00:06.144886 1 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtZXRyaWNzLXNlcnZlci10b2tlbi05a3M3cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjQ3ODViNTIwLWY3ZjItMTFlOC1iNmY0LTBhMjI5OTc2M2NlZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTptZXRyaWNzLXNlcnZlciJ9.rWW5R48HHu5Hwy10QoqancU2rCoswdXr9wrUVYkjnSjQIAQqOFnev4GyqfUduk7OMR-o7PaaPoRhFoqNro7au0ay7XDeV_WSGzPNBSTIjO2SYt1AxvrN3WGX_OARI8QoCKiogKuQ-mZHFZ5PfZc4urTKaynPClTHzsWJtrOKr6UXFMPY84hUxlfn2hwz39mB3iGmd_7xZqQW1JT94PdDkhfZw8uyfK-DFOOx1GtpEVRRHD8jajHfx1Xcgk4YivTZ997P0ZedmQiOUeFYSAl333iuJVkGU1Tdep2F_kzfd0VDxwN0M8x4OVo6yiJvv5zdPWyTpsL9xmJgM5ah-1i_ag" 'https://100.64.0.1:443/api/v1/nodes?resourceVersion=106401&timeoutSeconds=583&watch=true'
I1204 20:00:06.157894 1 round_trippers.go:405] GET https://100.64.0.1:443/api/v1/nodes?resourceVersion=106401&timeoutSeconds=583&watch=true 200 OK in 12 milliseconds
I1204 20:00:06.157921 1 round_trippers.go:411] Response Headers:
I1204 20:00:06.157927 1 round_trippers.go:414] Content-Type: application/json
I1204 20:00:06.157931 1 round_trippers.go:414] Date: Tue, 04 Dec 2018 20:00:06 GMT

But I am still not able to run top command and when creating an autoscale, I see the target continue with unknown/70% forever.

[minasys]

any news about this bug

[myspotontheweb]

I pulled the following out or your report, which looks like your code is calling the older heapster metrics API:

I1204 19:56:14.903106 4920 round_trippers.go:405] GET https://api.k8s-clusters.orioneclipse.com/api/v1/namespaces/kube-system/services/http:heapster:/proxy/apis/metrics/v1alpha1/nodes?labelSelector= 404 Not Found in 2 milliseconds

In my case kubectl using the newer /apis/metrics.k8s.io/v1beta1/nodes API

$ kubectl top nodes -v 10
..
I0212 19:21:44.109180    6652 round_trippers.go:438] GET https://api-REDACTED.elb.amazonaws.com/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 138 milliseconds
I0212 19:21:44.109239    6652 round_trippers.go:444] Response Headers:
I0212 19:21:44.109244    6652 round_trippers.go:447]     Content-Type: application/json
..

I see you've configured your cluster to use the custom metrics API, so now I'm wondering what version of the metrics API is installed and configured on your cluster?

For example in my case:

$ kubectl get APIService v1beta1.metrics.k8s.io | grep metrics
v1beta1.metrics.k8s.io   2019-02-10T10:05:20Z

I noticed you have only included a Deployment specification for the Metrics server. The kops add-on includes other objects, I suspect you're missing this:

• APIService

Additional: How I've installed metrics server

I have installed metrics server, using helm

$ helm ls metrics-server
NAME            REVISION    UPDATED                     STATUS      CHART                   APP VERSION NAMESPACE  
metrics-server  1           REDACTED                      DEPLOYED  metrics-server-2.1.1    0.3.1       kube-system

with the following values

$ helm get values metrics-server
args:
- --logtostderr
- --kubelet-insecure-tls=true
resources:
  limits:
    cpu: 100m
    memory: 50Mi

The insecure tls setting is a work-around an open issue :-(

[robwithhair]

Anyone know how I might get this working?