Minikube: kubectl api-resources broken after disabling metrics-server addon: server is currently unable to handle the request

Created on 23 Jan 2019  路  10Comments  路  Source: kubernetes/minikube

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

Please provide the following details:

Environment: minikube version: v0.33.1 on Linux ubuntu16 4.4.0-133

Minikube version (use minikube version): minikube version: v0.33.1 on Linux ubuntu16 4.4.0-133

  • OS (e.g. from /etc/os-release): ubuntu16 4.4.0-133
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): None
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
  • Install tools:
  • Others:
    $kubectl version
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:28:14Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}

What happened:
I use minikube addons enable metrics-server to deploy metrics-server in my env, but can't get the metrics.

$ sudo kubectl get pods.metrics.k8s.io
error: the server doesn't have a resource type "pods"

$ kubectl get 鈥搑aw apis/metrics.k8s.io/v1beta1/nodes
error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'kubectl get resource/<resource_name>' instead of 'kubectl get resource resource/<resource_name>'

$ kubectl get pods -n kube-system
NAME                                       READY   STATUS             RESTARTS   AGE
coredns-86c58d9df4-5wbgj                   0/1     CrashLoopBackOff   4          2m23s
coredns-86c58d9df4-zl5bm                   0/1     CrashLoopBackOff   4          2m23s
default-http-backend-6f855497bd-j8fm6      1/1     Running            0          2m18s
etcd-minikube                              1/1     Running            0          91s
kube-addon-manager-minikube                1/1     Running            0          98s
kube-apiserver-minikube                    1/1     Running            0          87s
kube-controller-manager-minikube           1/1     Running            0          100s
kube-proxy-5jmdk                           1/1     Running            0          2m23s
kube-scheduler-minikube                    1/1     Running            0          84s
kubernetes-dashboard-85b7755d8f-4g5dc      1/1     Running            0          2m19s
metrics-server-6fc4b7bcff-2crq7            1/1     Running            0          22s
nginx-ingress-controller-c69f7cdd7-mp4q8   1/1     Running            0          2m18s
storage-provisioner                        1/1     Running            0          2m16s

$ kubectl logs pods/metrics-server-6fc4b7bcff-2crq7    -n kube-system
I0123 06:25:20.905890       1 heapster.go:71] /metrics-server --source=kubernetes.summary_api:''
I0123 06:25:20.906021       1 heapster.go:72] Metrics Server version v0.2.1
I0123 06:25:20.906276       1 configs.go:61] Using Kubernetes client with master "https://10.96.0.1:443" and version
I0123 06:25:20.906309       1 configs.go:62] Using kubelet port 10255
I0123 06:25:20.907962       1 heapster.go:128] Starting with Metric Sink
I0123 06:25:21.887668       1 serving.go:308] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0123 06:25:22.561804       1 heapster.go:101] Starting Heapster API server...
[restful] 2019/01/23 06:25:22 log.go:33: [restful/swagger] listing is available at https:///swaggerapi
[restful] 2019/01/23 06:25:22 log.go:33: [restful/swagger] https:///swaggerui/ is mapped to folder /swagger-ui/
I0123 06:25:22.564940       1 serve.go:85] Serving securely on 0.0.0.0:443

There is no error log in metrics-server.

And after I disable metrics-server addon, there raise an error when I use kubectl api-resources. 

$ kubectl api-resources
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

What you expected to happen:
It shoudn't throw an error when user disable addons.

How to reproduce it (as minimally and precisely as possible):

  1. create a cluster: sudo minikube start --vm-driver=none --kubernetes-version=v1.13.2
  2. enable 'metrics-server' addon: sudo minikube addons enable metrics-server
  3. get the metrics: sudo kubectl get pods.metrics.k8s.io
  4. disable the addon: sudo minikube addons disable metrics-server
  5. see the resources: sudo kubectl api-resources

Output of minikube logs (if applicable):

$ kubectl logs pods/kube-apiserver-minikube -n kube-system
I0123 06:23:10.557896 1 controller.go:608] quota admission added evaluator for: deployments.apps
I0123 06:23:10.646166 1 controller.go:608] quota admission added evaluator for: daemonsets.apps
I0123 06:23:16.069174 1 controller.go:608] quota admission added evaluator for: controllerrevisions.apps
I0123 06:23:16.142450 1 controller.go:608] quota admission added evaluator for: replicasets.apps
I0123 06:23:21.188746 1 controller.go:608] quota admission added evaluator for: deployments.extensions
I0123 06:25:21.864048 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:25:22.347592 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.368621 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.373397 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.381806 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.459431 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:22.544807 1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Get https://10.97.40.73:443: dial tcp 10.97.40.73:443: connect: connection refused
E0123 06:25:24.039650 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:25:24.039673 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:25:28.225301 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:25:28.229348 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:25:28.229374 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:26:28.229649 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:26:28.237167 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0123 06:26:28.237233 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0123 06:27:20.119495 1 controller.go:608] quota admission added evaluator for: replicasets.extensions
I0123 06:27:22.079911 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:27:22.080127 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:27:22.080156 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:27:27.310802 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:27:57.333088 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:28:01.208934 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:28:01.209130 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:28:01.209162 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:28:27.363134 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:28:57.393069 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:29:01.209377 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:29:01.209532 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:29:01.209551 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:29:27.423372 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:29:57.450537 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:30:27.480751 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:30:57.511436 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:31:01.209873 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:31:01.210132 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:31:01.210148 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:31:27.539124 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:31:57.568052 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:32:27.593809 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:32:57.615577 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:33:01.194325 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:33:01.194779 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:33:01.194866 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:33:27.645871 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:33:57.679229 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:34:01.195242 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:34:01.195429 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:34:01.195469 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:34:27.710700 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:34:57.739270 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:35:27.768806 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:35:57.799318 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:36:01.195847 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:36:01.196262 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:36:01.196340 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:36:27.829623 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:36:57.856002 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:37:27.887110 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:37:57.912339 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:38:01.421432 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:38:01.421771 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:38:01.421826 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:38:27.941176 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:38:57.972746 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:39:01.422093 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:39:01.422267 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:39:01.422296 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:39:28.000507 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:39:58.030544 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:40:28.062406 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:40:58.090693 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:41:01.422587 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:41:01.422810 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:41:01.422835 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:41:28.118852 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:41:58.147632 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:42:28.176996 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:42:58.200169 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:43:01.438219 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:43:01.438607 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:43:01.438632 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:43:28.227598 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:43:58.257716 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:44:01.439085 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:44:01.439341 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:44:01.439368 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:44:28.286862 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:44:58.316254 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:45:28.345326 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:45:58.374960 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:46:01.439720 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:46:01.439912 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:46:01.439937 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:46:28.401303 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:46:58.432436 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:47:28.461957 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:47:58.515028 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:48:01.179057 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:48:01.179478 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:48:01.179505 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:48:28.544684 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:48:58.574627 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:49:01.179946 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:49:01.180426 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:49:01.180455 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:49:28.605946 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:49:58.634988 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:50:28.662338 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:50:58.693018 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:51:01.180981 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:51:01.181470 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:51:01.181493 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:51:28.721102 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:51:58.751810 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:52:28.784909 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:52:58.806793 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:53:01.158125 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:53:01.158545 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 06:53:01.158591 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:53:28.836887 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:53:58.869002 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:54:01.158997 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:54:01.159173 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:54:01.159195 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:54:28.899156 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:54:58.932305 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:55:28.963440 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:55:58.991959 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:56:01.159565 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:56:01.160042 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:56:01.160067 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:56:29.020089 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:56:59.050326 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:57:29.078128 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:57:59.101355 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:58:01.153808 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:58:01.153947 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:58:01.153966 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:58:29.129283 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:58:59.153660 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 06:59:01.154228 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 06:59:01.154350 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 06:59:01.154489 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 06:59:29.184314 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 06:59:59.212270 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:00:29.238526 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:00:59.264794 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:01:01.154957 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:01:01.155168 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:01:01.155194 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:01:29.291611 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:01:59.321175 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:02:29.351060 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:02:59.378184 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:03:01.243614 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:03:01.243967 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:03:01.244024 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:03:29.405791 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:03:59.431544 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:04:01.244431 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:04:01.244576 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:04:01.244590 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:04:29.457090 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:04:59.484954 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:05:29.505677 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:05:59.532492 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:06:01.244888 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:06:01.245533 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0123 07:06:01.245716 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:06:29.559155 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:06:59.583618 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:07:29.609731 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0123 07:07:59.636070 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0123 07:08:01.252289 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0123 07:08:01.252474 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
I0123 07:08:01.252498 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0123 07:08:29.660252 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request

Anything else do we need to know:

addonmetrics-server help wanted kinbug lifecyclrotten prioritbacklog 2019q2
Source
picture dongwenjuan

Most helpful comment

I had the same issue on a "regular" (non-minikube) cluster. I found this comment
https://github.com/kubernetes/kubernetes/issues/72051#issuecomment-506962702

Once I ran kubectl delete apiservice v1beta1.metrics.k8s.io the errors stopped on their own.

picture ryancalhoun on 12 Aug 2019
❤5 👍5 👎2

All 10 comments

I haven't tried the metrics-server addon yet, but the CoreDNS pods in CrashLoopBackoff looks suspicious. Perhaps it is trying to reference metrics.k8s.io internally, but DNS isn't available. Do you mind looking at the logs for the CoreDNS pods?

This might just be due to #3511

tstromberg picture tstromberg on 23 Jan 2019

@tsmetana Hi, thanks for your reply. I followed the guide: delete the loop in coredns configmap and restart the coredns pods, it works! Thanks a lot~

dongwenjuan picture dongwenjuan on 24 Jan 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot picture fejta-bot on 29 Apr 2019

I can confirm that this is still an issue in v1.1:

minikube addons enable metrics-server
kubectl get pods --all-namespaces --show-labels | grep metric
minikube addons disable metrics-server
kubectl get pods --all-namespaces --show-labels | grep metric
kubectl api-resources

output:

error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
tstromberg picture tstromberg on 23 May 2019

I had the same issue on a "regular" (non-minikube) cluster. I found this comment
https://github.com/kubernetes/kubernetes/issues/72051#issuecomment-506962702

Once I ran kubectl delete apiservice v1beta1.metrics.k8s.io the errors stopped on their own.

ryancalhoun picture ryancalhoun on 12 Aug 2019
❤5 👍5 👎2

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot picture fejta-bot on 10 Nov 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot picture fejta-bot on 10 Dec 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot picture fejta-bot on 9 Jan 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 9 Jan 2020

I had the same issue on a "regular" (non-minikube) cluster. I found this comment
kubernetes/kubernetes#72051 (comment)

Once I ran kubectl delete apiservice v1beta1.metrics.k8s.io the errors stopped on their own.

Keep in mind the command will delete the metrics service and doesn't fix the underlying problem, possibly introducing other issues. I don't recommend doing that. In our case one master node could not connect to the metrics server but after deleting this master and waiting for a new one to spin up the issue was gone.

mbaeuerle picture mbaeuerle on 6 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

olalonde picture olalonde  路  3Comments
hank-cp picture hank-cp  路  3Comments
tnine picture tnine  路  3Comments
mdkess picture mdkess  路  3Comments
mrtkp9993 picture mrtkp9993  路  3Comments