Metrics-server: Kubernetes 1.18.8 Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

[godomainz]

I'm getting this error --> Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

When I run kubectl top node, kubectl top po

What am I missing here ?

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"

kubectl get pods -o wide --all-namespaces --watch
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE    IP                NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-5448995c8b-87g58         1/1     Running   1          2d2h   10.30.192.1       worker-2   <none>           <none>
kube-system   coredns-5448995c8b-dzsn9         1/1     Running   1          2d2h   10.30.0.2         worker-1   <none>           <none>
kube-system   metrics-server-987bf9856-68mdh   1/1     Running   0          13m    10.30.192.2       worker-2   <none>           <none>
kube-system   weave-net-9f4ss                  2/2     Running   2          2d2h   192.168.111.136   worker-1   <none>           <none>
kube-system   weave-net-bqsp4                  2/2     Running   2          2d2h   192.168.111.137   worker-2   <none>           <none>

metric-server.yaml

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: view-metrics
rules:
- apiGroups:
    - metrics.k8s.io
  resources:
    - pods
    - nodes
  verbs:
    - get
    - list
    - watch

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: view-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view-metrics
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:kube-proxy

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      # hostNetwork: true
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - /metrics-server
          - --kubelet-preferred-address-types=InternalIP
          - --kubelet-insecure-tls
          - --cert-dir=/tmp
          - --secure-port=4443
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

[serathius]

Looks like kubectl doesn't detect metrics apiservice
Please run

kubectl describe apiservice v1beta1.metrics.k8s.io

[casey-robertson]

I run that command and get:

 kubectl describe apiservice v1beta1.metrics.k8s.io
Error from server (NotFound): apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" not found

[hrishikeshnerurkar]

The above command needs to check for "v1.metrics.k8s.io" as that's name of the apiservice. However, upon running the describe command as kubectl describe apiservice v1.metrics.k8s.io, I see following output:

Name:         v1.metrics.k8s.io
Namespace:
Labels:       <none>
Annotations:  API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2020-12-18T23:27:18Z
  Resource Version:    2136344
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1.metrics.k8s.io
  UID:                 6ce3bda2-f876-46dd-8760-eb5cc3cac8ff
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       monitoring
    Port:            443
  Version:           v1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2020-12-18T23:27:18Z
    Message:               failing or missing response from https://10.102.203.170:443/apis/metrics.k8s.io/v1: bad status from https://10.102.203.170:443/apis/metrics.k8s.io/v1: 404
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

Not sure what's the issue.

[xueshanf]

I am using bitnami/metrics-server helm chart and running into same problem with chart v5.3.3, app version 0.4.1. It seems that I need to enable v1beta1.metrics.k8s.io apiService to make it work. This may not apply to your case though since you are using apiregistration.k8s.io/v1.

[serathius]

Ok, I see what's the problem. Metrics API was never graduated to v1, but Service provided uses v1.metrics.k8s.io. This api will not work as kubectl and Metrics Server don't implement it.

Please downgrade to v1beta1. How apiservice should look like:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

[serathius]

Closing issue as resolved, feel free to reopen if problem still persists.