Kube-prometheus: custom metrics not work(http2: stream closed)

[tallongsun]

I'm running the adapter with the following command .

git clone https://github.com/coreos/kube-prometheus.git
cd kube-prometheus/
kubectl create -f manifests/setup
kubectl create -f manifests/ 
cd experimental/custom-metrics-api
sh deploy.sh
kubectl apply -f sample-app.yaml

when i try to run kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1", i got the error Error from server (ServiceUnavailable): the server is currently unable to handle the request

and i check the adapter log, i found the following error.

apiserver was unable to write a JSON response: http2: stream closed
apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

what should i do to make the metrics api start up?
My k8s version is

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

[tallongsun]

btw, i can retrieve http_requests_total metric from prometheus directly , but can't retrieve from prometheus-adapter(custom metric server)

[gitbeyond]

please update image to quay.io/coreos/k8s-prometheus-adapter-amd64:v0.6.0.

[dyakhnov]

Same issue here using release-0.3 as we are on k8s v1.15.3, log full of identical errors:

E0422 13:54:40.295011       1 writers.go:163] apiserver was unable to write a JSON response: http2: stream closed
E0422 13:54:40.295041       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0422 13:54:40.307223       1 writers.go:163] apiserver was unable to write a JSON response: http2: stream closed
E0422 13:54:40.307255       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0422 13:54:40.310829       1 writers.go:163] apiserver was unable to write a JSON response: http2: stream closed
E0422 13:54:40.310845       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}

Everything else seems to be working fine... we can see metrics, HPA is scaling.

[stafot]

Same problem here after upgrade to AWS EKS 1.15. All seems to working fine but there is a flood on this kind of logs.

Update:

Problem remains with adapter 0.6.0
/cc @gitbeyond

 

 

[brancz]

cc @s-urbaniak @lilic

[s-urbaniak]

So far I cannot reproduce this, maybe @sttts can give a hint why api server could potentially close the http2 connection. Is there anything special in that regard in your setup?

[stafot]

@s-urbaniak Nope. And as I described above problem started after rolled out my eks master to be 1.15 from 1.14. nothing else changed on my kube-prometheus setup. Tomorrow I will rollout a kube Prometheus update on my testing environment to see if something will change. Also since Friday 1.16 became available for eks so i am keen to also try this.

[serathius]

I think this is a bug in https://github.com/kubernetes-sigs/custom-metrics-apiserver as it was reported in different adapters.

• https://github.com/GoogleCloudPlatform/k8s-stackdriver/issues/318
• https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/276

[stafot]

@s-urbaniak @brancz As I mentioned in https://github.com/coreos/kube-prometheus/issues/383#issuecomment-623381835 I rolled out kube-prometheus update and problem persists. I agree with @serathius that seems to be the same in multiple adapters. But if as mentioned in the links shared the problem from logs flood on 1.15 becomes not functioning in 1.16 it is somehow blocking on using Custom metrics autoscaling with prometheus and 1.16

[outinspace]

I am seeing this issue as well. I pulled the main branch, uncommented the custom metrics feature in example.jsonnet and built/deployed to my cluster.