Kube-prometheus: v1beta1.metrics.k8s.io failed with: OpenAPI spec does not exists

Created on 12 Nov 2019  ยท  18Comments  ยท  Source: prometheus-operator/kube-prometheus 
kube-apiserver version :1.12.4

error log in kube-apiserver it repeats about every 60 seconds

11ๆœˆ 12 18:25:38 wlwk8s83 kube-apiserver[106573]: I1112 18:25:38.763724  106573 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
11ๆœˆ 12 18:25:38 wlwk8s83 kube-apiserver[106573]: E1112 18:25:38.763713  106573 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
11ๆœˆ 12 18:25:38 wlwk8s83 kube-apiserver[106573]: I1112 18:25:38.758608  106573 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
11ๆœˆ 12 18:23:38 wlwk8s83 kube-apiserver[106573]: I1112 18:23:38.758427  106573 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
11ๆœˆ 12 18:23:38 wlwk8s83 kube-apiserver[106573]: E1112 18:23:38.758410  106573 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
11ๆœˆ 12 18:23:38 wlwk8s83 kube-apiserver[106573]: I1112 18:23:38.754208  106573 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
11ๆœˆ 12 18:22:38 wlwk8s83 kube-apiserver[106573]: I1112 18:22:38.754039  106573 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
11ๆœˆ 12 18:22:38 wlwk8s83 kube-apiserver[106573]: E1112 18:22:38.754021  106573 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
11ๆœˆ 12 18:22:38 wlwk8s83 kube-apiserver[106573]: I1112 18:22:38.749891  106573 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
11ๆœˆ 12 18:20:38 wlwk8s83 kube-apiserver[106573]: I1112 18:20:38.739905  106573 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
11ๆœˆ 12 18:20:38 wlwk8s83 kube-apiserver[106573]: E1112 18:20:38.739884  106573 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
11ๆœˆ 12 18:20:38 wlwk8s83 kube-apiserver[106573]: I1112 18:20:38.735557  106573 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
11ๆœˆ 12 18:18:38 wlwk8s83 kube-apiserver[106573]: I1112 18:18:38.735386  106573 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
11ๆœˆ 12 18:18:38 wlwk8s83 kube-apiserver[106573]: E1112 18:18:38.735366  106573 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists

kubectl top node get right answer

[root@wlwk8s83 ~]# kubectl top node
NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
192.168.104.123   386m         12%    4945Mi          7%        
192.168.104.124   383m         12%    6688Mi          10%       
192.168.104.125   390m         13%    4450Mi          7%        
192.168.104.126   381m         12%    4643Mi          7%        
192.168.104.127   395m         13%    6456Mi          10%       
192.168.104.83    361m         5%     3600Mi          5%        
192.168.104.84    183m         2%     3456Mi          6%        
192.168.104.85    192m         2%     3367Mi          5%        
192.168.104.86    429m         6%     8848Mi          14%       
192.168.104.87    451m         6%     17753Mi         28%       
192.168.104.88    951m         13%    19748Mi         31%       
192.168.104.89    217m         3%     19432Mi         30%       
192.168.104.90    173m         2%     10697Mi         16%
picture zhaofengcumt
👍2

Most helpful comment

Fixed by https://github.com/prometheus-operator/kube-prometheus/pull/777

prometheus-adapter:v0.8.2 implements the OpenAPI spec for custom metrics apiserver.

/close

picture ravilr on 2 Dec 2020
👍3 🎉2

All 18 comments

cc @s-serguisz I think the Prometheus adapter should probably return this.

brancz picture brancz on 12 Nov 2019

cc @s-urbaniak is what you meant :)

lilic picture lilic on 12 Nov 2019
😄3 👍2

hmm ... i have to look into this indeed :thinking:

s-urbaniak picture s-urbaniak on 14 Nov 2019

kubernetes 1.16 install kube-prometheus,I have this question too.

duiniwukenaihe picture duiniwukenaihe on 26 Nov 2019

and kubernetes 1.17,if i hava this kube-prometheus my kubect return need many second; i see the manager has many erro as v1beta1.metrics.so i have uninstall prometheus and my kubectl resume

duiniwukenaihe picture duiniwukenaihe on 17 Dec 2019

@s-urbaniak Is there any news on this? I'm having the same issue (kubernetes 1.16, prometheus-operator 0.36.0).

kubectl delete -n kube-system apiservice v1beta1.metrics.k8s.io can be used to stop the log spam, but logically breaks kubectl top nodes|pods and might have other effects as well.

lanmarti picture lanmarti on 6 Mar 2020

we are seeing same behaviour (kube 1.17.3)

E0330 10:52:01.277714       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 10:52:01.277732       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0330 10:53:01.277907       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0330 10:53:01.281828       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 10:53:01.281847       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0330 10:55:01.282076       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0330 10:55:01.287701       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 10:55:01.287738       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0330 10:57:01.274225       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0330 10:57:01.278429       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 10:57:01.278448       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0330 10:58:01.278671       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0330 10:58:01.283495       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 10:58:01.283520       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0330 11:00:01.283800       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0330 11:00:01.289616       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0330 11:00:01.289644       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
zetaab picture zetaab on 30 Mar 2020

cc @s-urbaniak

lilic picture lilic on 30 Mar 2020

Is there any news to this? We're also seeing the same behavior in multiple clusters running different Kubernetes versions.

Hayajiro picture Hayajiro on 4 May 2020

There is some movement on the custom-metrics-apiserver, which is the dependency the k8s-prometheus-adapter component uses which ultimately produces this: https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/65

brancz picture brancz on 11 May 2020

Any news on this? We're also seeing the same behavior (Kubernetes v1.15.9, prometheus-adapter v0.4.1)

I0629 12:13:52.677829       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0629 12:14:52.677953       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0629 12:14:52.681079       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0629 12:14:52.681088       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0629 12:16:52.681324       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0629 12:16:52.684265       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0629 12:16:52.684275       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0629 12:18:52.678609       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0629 12:18:52.681384       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0629 12:18:52.681429       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
^C

# kubectl top nodes
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
controller-54   882m         1%     114726Mi        22%
controller-56   921m         1%     75667Mi         14%
engine-61       847m         1%     119892Mi        46%
engine-64       1481m        2%     57906Mi         22%
engine-67       <unknown>                           <unknown>               <unknown>               <unknown>
controller-55   <unknown>                           <unknown>               <unknown>               <unknown>
therealqxing picture therealqxing on 29 Jun 2020

cc @paulfantom was working on this last IIRC

lilic picture lilic on 29 Jun 2020

Seeing this too. It's resulting in a KubeAPIErrorBudgetBurn alert "The API server is burning too much error budget". Any update @paulfantom ?

Kolossi picture Kolossi on 22 Jul 2020

@Kolossi are you sure that alert was the direct result of these errors? Very interesting! The PR is still in progress https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/65 do you mind having a test to see how it helps you?

lilic picture lilic on 21 Sep 2020

and on kubernetes 1.18.3 . this cause KubeAPIErrorBudgetBurn alert on prometheus?

E0921 16:11:23.518494       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0921 16:11:23.518512       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0921 16:12:23.521884       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0921 16:12:23.521914       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0921 16:14:23.526134       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0921 16:14:23.526165       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0921 16:16:23.533440       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0921 16:16:23.533508       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0921 16:17:23.536931       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0921 16:17:23.536957       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
oldthreefeng picture oldthreefeng on 22 Sep 2020

@oldthreefeng @Kolossi just to verify, the alerts are firing but is there any degraded functionality to the Prometheus adapter? Thanks!

lilic picture lilic on 22 Sep 2020

@lilic In this situation. The cluster is running normally.
. only the alert turns to red~

oldthreefeng picture oldthreefeng on 22 Sep 2020

Fixed by https://github.com/prometheus-operator/kube-prometheus/pull/777

prometheus-adapter:v0.8.2 implements the OpenAPI spec for custom metrics apiserver.

/close

ravilr picture ravilr on 2 Dec 2020
👍3 🎉2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

johscheuer picture johscheuer  ยท  6Comments
RiRa12621 picture RiRa12621  ยท  6Comments
alam0rt picture alam0rt  ยท  5Comments
KlavsKlavsen picture KlavsKlavsen  ยท  3Comments
chiefy picture chiefy  ยท  6Comments