Metrics-server: OpenAPI spec does not exists

Created on 23 Nov 2017 · 14Comments · Source: kubernetes-sigs/metrics-server

@DirectXMan12
So, I'm reopening this as a followup to issue #22 :
I've added kubelet & kube-proxy to all the nodes (controller, etc.) and thus the weave overlay network device and adresses are avaiable to the apiserver & al.

Any idea what is now going on by Nov 21 17:24:45?

Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.327639       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (5.12395ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.328952       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.332223       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (2.323296ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.339954       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (3.352474ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.421626       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (4.097894ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.423560       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.432152       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (5.594283ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.471791       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (14.25075ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: E1121 22:21:05.473203       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.478075       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (2.201772ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:05 localhost docker/kube-apiserver[891]: I1121 22:21:05.487308       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (7.405118ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.973669       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (23.105986ms) 409 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: E1121 22:21:07.974848       1 available_controller.go:225] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.990372       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (10.307827ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:07 localhost docker/kube-apiserver[891]: I1121 22:21:07.993020       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (1.674686ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: I1121 22:21:11.361404       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: E1121 22:21:11.390341       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:21:11 localhost docker/kube-apiserver[891]: I1121 22:21:11.390397       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: I1121 22:22:11.391155       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: E1121 22:22:11.395587       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:22:11 localhost docker/kube-apiserver[891]: I1121 22:22:11.395604       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: I1121 22:24:11.395958       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: E1121 22:24:11.398517       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:24:11 localhost docker/kube-apiserver[891]: I1121 22:24:11.398543       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:24:45 localhost docker/kube-apiserver[891]: I1121 22:24:45.689134       1 wrap.go:42] PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io/status: (1.802252ms) 200 [[kube-apiserver/v1.8.4 (linux/amd64) kubernetes/9befc2b] 127.0.0.1:53326]
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: I1121 22:24:49.146330       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: E1121 22:24:49.148977       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:24:49 localhost docker/kube-apiserver[891]: I1121 22:24:49.149034       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: I1121 22:25:49.149473       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: E1121 22:25:49.153288       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:25:49 localhost docker/kube-apiserver[891]: I1121 22:25:49.153311       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: I1121 22:27:49.153602       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: E1121 22:27:49.157182       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
Nov 21 17:27:49 localhost docker/kube-apiserver[891]: I1121 22:27:49.157200       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

It seems to be able to talk to the API but then gets:
OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists

kubectl top pod / node still says:
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)

I seem to be getting closer, but still missing something... :-)

Source

RRAlex

Most helpful comment

I use k8s1.11 and the error report always，what am I supposed to do to make it stop?
{"log":"E0113 15:04:10.857205 1 controller.go:111] loading OpenAPI spec for \"v1beta1.metrics.k8s.io\" failed with: OpenAPI spec does not existsn","stream":"stderr","time":"2019-01-13T15:04:10.857349081Z"}

xuejipeng on 13 Jan 2019

👍9

All 14 comments

You still need heapster for kubectl top to work. it has not been switched over to use metric-server

also the error "OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists" i got told by @DirectXMan12 is just annoying and not a actuall error

roffe on 27 Nov 2017

👍4

Correct, the missing OpenAPI spec is more of a warning. @piosz it's actually possible to serve an OpenAPI spec for the resource metrics API, so we should get that plumbing in.

There's a WIP PR to make kubectl top use the resource metrics API instead of using Heapster directly: https://github.com/kubernetes/kubernetes/pull/56206

DirectXMan12 on 27 Nov 2017

👍2

65 added openapi spec support, and kubectl top support landed in 1.10. closing

DirectXMan12 on 8 Aug 2018

I was in the middle of filing a fresh issue about this problem this morning when I discovered this issue. #65 is merged now. Will we have a publicly available container image that we can use to take advantage of this fix?

seh on 9 Aug 2018

I need to figure out who needs to push the buttons on the release. I'll try and sync with them and hopefully get a new image out by next week/

DirectXMan12 on 9 Aug 2018

Thank you. We're eager to try all of your recent improvements!

seh on 9 Aug 2018

I have encountered the same problem here.

kubernetes version

Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

this is kube-apiserver log

$ docker ps | grep api
4521e9e67ed3 harbor.enncloud.cn/enncloud/hyperkube-amd64@sha256:2466e6219aba9d88dcef04c0f5578a6df5e9cc0d9f1913357585bc71436b4237 "/apiserver --enab..." 16 hours ago Up 15 hours k8s_kube-apiserver_kube-apiserver-master-47-35_kube-system_77ab11667a564948640e969f1cf0f291_0
50d8168fd885 harbor.enncloud.cn/paas/pause-amd64:3.1 "/pause" 16 hours ago Up 16 hours k8s_POD_kube-apiserver-master-47-35_kube-system_77ab11667a564948640e969f1cf0f291_0

$ docker logs 4521e9e67ed3
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
[restful] 2018/09/12 08:51:22 log.go:33: [restful/swagger] listing is available at https://10.39.47.35:6443/swaggerapi
[restful] 2018/09/12 08:51:22 log.go:33: [restful/swagger] https://master_ip:6443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/09/12 08:51:23 log.go:33: [restful/swagger] listing is available at https://master_ip:6443/swaggerapi
[restful] 2018/09/12 08:51:23 log.go:33: [restful/swagger] https://master_ip:6443/swaggerui/ is mapped to folder /swagger-ui/
E0912 08:51:27.393041 1 memcache.go:147] couldn't get resource list for batch/v2alpha1: the server could not find the requested resource
E0912 08:51:27.404958 1 memcache.go:147] couldn't get resource list for settings.k8s.io/v1alpha1: the server could not find the requested resource
E0912 08:51:30.865043 1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:52:30.865424 1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:54:30.865872 1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:56:31.168069 1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:57:08.690662 1 available_controller.go:311] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0912 08:57:13.943069 1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]
E0912 08:57:26.264538 1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:57:31.169414 1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:58:21.522594 1 available_controller.go:311] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0912 08:58:21.541018 1 available_controller.go:311] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0912 08:58:26.675389 1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 08:58:43.855735 1 available_controller.go:311] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0912 08:58:43.865437 1 available_controller.go:311] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E0912 08:58:48.657936 1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

k8s-prometheus-adapter version is v0.2.1

But does not affect the use

How can I solve this problem?
@DirectXMan12

qinzhao168 on 12 Sep 2018

simply splatting log output without context isn't useful. Please provide context for your comment.

DirectXMan12 on 12 Sep 2018

Somewhat related to the above, using the metrics server at version 0.3.1, I see this sequence repeatedly in my API server logs:

I1208 18:00:48.917023       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1208 18:00:48.918572       1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
E1208 18:00:56.769158       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E1208 18:01:06.782579       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E1208 18:01:16.782063       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E1208 18:01:26.790570       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
E1208 18:01:36.796293       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I1208 18:01:48.918747       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1208 18:01:48.941642       1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
E1208 18:01:56.811659       1 available_controller.go:316] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again

Note that this modification complaint repeats approximately every ten seconds, with the requeue notification once per minute.

Does this warrant a fresh issue?

seh on 8 Dec 2018

@seh hmm... that's weird. Yeah, file that against the main kube repo. Looks like an aggregator bug.

DirectXMan12 on 21 Dec 2018

xuejipeng on 13 Jan 2019

👍9

@qinzhao168 check if apiservice/v1beta1.metrics.k8s.io is pointing to the right service:

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.metrics.k8s.io","namespace":""},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"prometheus-adapter","namespace":"monitoring"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: 2018-08-05T09:20:15Z
  name: v1beta1.metrics.k8s.io
  resourceVersion: "61054942"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: c39747f6-9890-11e8-9574-06596da1cff8
spec:
  caBundle: null
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server                <------
    namespace: kube-system              <------
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2019-02-21T02:47:42Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

I've installed kube-prometheus manifests and it replaced the original metric-server service for its own prometheus-adpater. After removing kube-prometheus I got an non-functional apiservice adapter. Took me a while to figure it out.