Kube-prometheus: Implement configmap-reloader for prometheus-adapter

[middlestone]

What happened?

After deploying kube-prometheus, I want to try HPA, and use deploy.sh to setup custom-metrics-api, but apiservice v1beta1.custom.metrics.k8s.io fails to contact prometheus-adapter service.

kubectl describe apiservice v1beta1.custom.metrics.k8s.io

Name: v1beta1.custom.metrics.k8s.io
Namespace:
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.custom.metrics.k8s.io"},"s...
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2019-11-09T07:33:48Z
Resource Version: 1743138
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
UID: d8985b81-4316-4717-a38d-669ebf9e08c8
Spec:
Group: custom.metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: prometheus-adapter
Namespace: monitoring
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2019-11-09T07:33:48Z
Message: failing or missing response from https://10.111.2.213:443/apis/custom.metrics.k8s.io/v1beta1: bad status from https://10.111.2.213:443/apis/custom.metrics.k8s.io/v1beta1: 404
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:

Did you expect to see some different?
It should show message "all checks passed".

How to reproduce it (as minimally and precisely as possible):
After deploying the kube-prometheus, run the deploy.sh under experimental/custom-metrics-api

Environment

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm64"}

• Prometheus Operator version:

  Insert image tag or Git SHA here
  
  Name: prometheus-operator
  Namespace: monitoring
  CreationTimestamp: Mon, 28 Oct 2019 11:04:03 +0800
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/name=prometheus-operator
  app.kubernetes.io/version=v0.33.0
  Annotations: deployment.kubernetes.io/revision: 2
  kubectl.kubernetes.io/last-applied-configuration:
  {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kuberne...
  Selector: app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator
  Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
  StrategyType: RollingUpdate
  MinReadySeconds: 0
  RollingUpdateStrategy: 25% max unavailable, 25% max surge
  Pod Template:
  Labels: app.kubernetes.io/component=controller
  app.kubernetes.io/name=prometheus-operator
  app.kubernetes.io/version=v0.33.0
  Service Account: prometheus-operator
  Containers:
  prometheus-operator:
  Image: quay.io/coreos/prometheus-operator:v0.33.0
  Port: 8080/TCP
  Host Port: 0/TCP
  Args:
  --kubelet-service=kube-system/kubelet
  --logtostderr=true
  --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
  --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.33.0
  Limits:
  cpu: 200m
  memory: 500Mi
  Requests:
  cpu: 100m
  memory: 300Mi
  Environment:
  Mounts:
  Volumes:
  Conditions:
  Type Status Reason
  ---- ------ ------
  Progressing True NewReplicaSetAvailable
  Available True MinimumReplicasAvailable
  OldReplicaSets:
  NewReplicaSet: prometheus-operator-79df446bd8 (1/1 replicas created)
  Events:

• Kubernetes version information:

  kubectl version
  

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.
  kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.15.3

• Manifests:
  just the same as the ones on master branch.

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:
kubectl get apiservice
....
v1beta1.custom.metrics.k8s.io monitoring/prometheus-adapter False (FailedDiscoveryCheck) 24m
v1beta1.events.k8s.io Local True 14d
v1beta1.extensions Local True 14d
v1beta1.metrics.k8s.io monitoring/prometheus-adapter True 12d
....

kubectl get --raw /apis/custom-metrics.metrics.k8s.io/v1beta1
Error from server (NotFound): the server could not find the requested resource

kubectl get -v=8 --raw /api/v1/namespaces/monitoring/services/https:prometheus-adapter:443/proxy/
I1109 16:00:57.797371 29808 loader.go:359] Config loaded from file: /etc/kubernetes/admin.conf
I1109 16:00:57.798212 29808 round_trippers.go:416] GET https://10.4.5.117:6443/api/v1/namespaces/monitoring/services/https:prometheus-adapter:443/proxy/
I1109 16:00:57.798233 29808 round_trippers.go:423] Request Headers:
I1109 16:00:57.798245 29808 round_trippers.go:426] Accept: application/json, /
I1109 16:00:57.798257 29808 round_trippers.go:426] User-Agent: kubectl/v1.15.3 (linux/arm64) kubernetes/2d3c76f
I1109 16:00:57.821251 29808 round_trippers.go:441] Response Status: 200 OK in 22 milliseconds
I1109 16:00:57.821293 29808 round_trippers.go:444] Response Headers:
I1109 16:00:57.821307 29808 round_trippers.go:447] Date: Sat, 09 Nov 2019 08:00:57 GMT
I1109 16:00:57.821319 29808 round_trippers.go:447] Content-Length: 229
I1109 16:00:57.821331 29808 round_trippers.go:447] Content-Type: application/json
{
"paths": [
"/apis",
"/apis/metrics.k8s.io",
"/apis/metrics.k8s.io/v1beta1",
"/healthz",
"/healthz/ping",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/metrics",
"/version"
]

[middlestone]

why there is no /apis/custom.metrics.k8s.io/v1beta1 in prometheus-adapter? I am using the image of https://hub.docker.com/r/directxman12/k8s-prometheus-adapter, if anyone know this problem, please tell me, thanks in advance.

[brancz]

cc @s-urbaniak

[middlestone]

thanks Frederic for your help, just update antoher found here. if using "helm install --name my-release stable/prometheus-adapter" to install prometheuse-adapter, it has /apis/custom.metrics.k8s.io/v1beta1, I still don't know what is the difference between kube-promethus and the one installed by helm.

output when using helm:
kubectl get -v=8 --raw /api/v1/namespaces/default/services/https:prometheus-adapter:443/proxy/
I1111 20:39:33.287204 17139 loader.go:359] Config loaded from file: /etc/kubernetes/admin.conf
I1111 20:39:33.288039 17139 round_trippers.go:416] GET https://10.4.5.117:6443/api/v1/namespaces/default/services/https:prometheus-adapter:443/proxy/
I1111 20:39:33.288063 17139 round_trippers.go:423] Request Headers:
I1111 20:39:33.288075 17139 round_trippers.go:426] User-Agent: kubectl/v1.15.3 (linux/arm64) kubernetes/2d3c76f
I1111 20:39:33.288088 17139 round_trippers.go:426] Accept: application/json, /
I1111 20:39:33.355544 17139 round_trippers.go:441] Response Status: 200 OK in 67 milliseconds
I1111 20:39:33.355571 17139 round_trippers.go:444] Response Headers:
I1111 20:39:33.355583 17139 round_trippers.go:447] Content-Type: application/json
I1111 20:39:33.355594 17139 round_trippers.go:447] Date: Mon, 11 Nov 2019 12:39:33 GMT
I1111 20:39:33.355605 17139 round_trippers.go:447] Content-Length: 179
{
"paths": [
"/apis",
"/apis/custom.metrics.k8s.io",
"/apis/custom.metrics.k8s.io/v1beta1",
"/healthz",
"/healthz/ping",
"/metrics",
"/version"
]
}

[middlestone]

I think I find the root cause, after running deploy.sh, the pod prometheus-adapter should be deleted manually in order to be recreated so that new configmap can take effect.

[paulfantom]

I changed the title to better reflect what needs to be implemented.