Kubebuilder: Unable to export prometheus metrics be following the steps provided

[camilamacedo86]

, I followed instructions in #1030 but I saw the following error in Prometheus for the metrics service endpoint:

"Get http://172.17.0.15:8443/metrics: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02""

Note that Prometheus was using "http" to access the end point. So I added the following

scheme: https

to "endpoints" of ServiceMonitor but then got this (expected) error:

"Get https://172.17.0.15:8443/metrics: x509: cannot validate certificate for 172.17.0.15 because it doesn't contain any IP SANs"

This is because the certificate used by the metrics service is not issued for the IP. I think we need to pass proper certificate info in "tlsConfig" (see https://coreos.com/operators/prometheus/docs/latest/api.html#tlsconfig). Either that or we need to enable no auth access to metrics end point.
Just to be clear, I am using Prometheus operator and not kube-prometheus but I doubt if that makes any difference. I will see if I can install the later and give it one more try but I would still expect similar errors as metrics end point is served only with "https".

Motivation: https://github.com/kubernetes-sigs/kubebuilder/pull/1245#discussion_r356119790

[camilamacedo86]

Hi @draghuram,

As we spoke in the PR I am unable to reproduce your scenario. Following the steps used to check it.

1. Install the Prometheus with Minikube. I am following the steps of https://github.com/coreos/kube-prometheus#quickstart. Following them:
2. run minikube delete && minikube start --kubernetes-version=v1.16.0 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0
3. run minikube addons disable metrics-server
4. Cloned the project locally git clone [email protected]:coreos/kube-prometheus.git
5. Run kubectl create -f manifests/setup
6. After all be create run kubectl create -f manifests/
7. Check that all is running kubectl get all --namespace monitoring
8. Run kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Now, we can access the Prometheus here: http://localhost:9090/

1. Create a project and export the operator metrics by following the quick start.

Creating the project

mkdir $GOPATH/src/example
cd $GOPATH/src/example
kubebuilder init --domain my.domain
mkdir $GOPATH/src/example
cd $GOPATH/src/example
kubebuilder init --domain my.domain 

PS.: Select y and y options to create the CRD and controller

Enabling the Prometheus

• Go to `config/default/kustomization.yaml``
• Uncomment the line - ../prometheus as follows.

# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'. 
- ../prometheus

Installing the Project in the cluster

make install
kubectl apply -f config/samples/
make docker-build docker-push IMG=<some-registry>/<project-name>:tag
make deploy IMG=<some-registry>/<project-name>:tag

1. Checking the Operator Metrics
2. Go to the Prometheus dashboard http://localhost:9090/
3. Search for all from the namespace where the operator is running {namespace="example-system"}

All worked fine as expected.

[camilamacedo86]

Hi @draghuram,

As you can see all worked as expected. So, could you provide the following information?

1. Are you checking it with the QuickStart or another project?
2. Could you please provide the link for the project used by you?
3. Could you please provide the kubebuilder version/ go version and minikube version used to test it?
4. Did you perform the same above steps to check it?

c/c @mengqiy

[camilamacedo86]

/assign @camilamacedo86

[draghuram]

Hi @draghuram,
As we spoke in the PR I am unable to reproduce your scenario. Following the steps used to check it.

Ok. May be there is some problem with the Prometheus operator as you suggested before. I will switch to kube-prometheus at some point but please proceed with the changes as they seem to work.

[draghuram]

Hi @draghuram,

As you can see all worked as expected. So, could you provide the following information?

1. Are you checking it with the QuickStart or another project?

I was following steps in #1030 except that I used Prometheus operator (https://github.com/coreos/prometheus-operator) instead of kube-prometheus.

1. Could you please provide the kubebuilder version/ go version and minikube version used to test it?

I just realized that I used kubebuilder that I built locally few days ago from master branch. However, I have been using 2.1.0 for my other project. I will retest with this version and kube-prometheus. Note that I already had minikube v1.4.0 running prior to this test.

[camilamacedo86]

Hi @draghuram,

Just one point of attention. See that if you use minikube to setup the Prometheus as described in the doc is required to:

$ minikube delete && minikube start --kubernetes-version=v1.16.0 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0

and

$ minikube addons disable metrics-server

[DustinChaloupka]

Same problem here, using Prometheus Operator instead of kube-prometheus.

https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint are the updated docs for Endpoint which, as @draghuram has mentioned, includes the tlsConfig field for various configuration to make the request over https through the proxy.

I couldn't find anything from a quick look at the proxy that would allow prometheus to scrape over https using the field though, but it wasn't as big of a deal to use https for us at this point, so just switched back to http and fixed the resources generated by kubebuilder.

I had only checked this usage on an AKS cluster that already had the Prometheus Operator running, not on minikube though. Unsure if the extra options supplied to the startup of minikube allow prometheus to work with it somehow.

Would also like to point out that the check for the operator metrics given in the comments does not show any of the actual operator metrics, just other metrics from that namespace. No idea if the actual operator metrics do show up in that check, but I had just been checking the prometheus targets that the operator target was up and did not have any errors.

[mengqiy]

@camilamacedo86 I played a bit with the new ServiceMonitoring stuff.
I suspect we miss something regarding granting permissions to scrape the /metrics endpoints.
We probably should have some manifests or document about how to do it.
IMO we should scaffold the follow cluster role.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metrics
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

And asked the user to grant permissions to Prometheus by creating ClusterRoleBinding like the following.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics
subjects:
- kind: ServiceAccount
  name: default
  namespace: <your-desired-namespace> # e.g. the namespace where you run the Prometheus.

[camilamacedo86]

HI @mengqiy,

I follow all steps of the documentation to install Prometheus operator and how to use it with minikube then, I am unable to face any issue related to the Prometheus do not find the endpoint. Could you reproduce the scenario where the endpoint is not found in Prometheus? Was it solved with the ClusterRole added above? If yes, could you share it?

Also, If it is required in some scenario I agree with you that scaffold is the best option.

PS.: Since is a ClusterRole I think that the namespace can be system in order to do as it is done in for the others ClusterRoleBinding's in the project.

Since I am unable to reproduce the scenario I did a PR with your suggestion @draghuram and @DustinChaloupka could you help us to verify it as well?

[mengqiy]

- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs:
  - get
  - create

Who will be using this permissions?

[mengqiy]

Prometheus Operator seems to be the only controller needs permissions to do CRUD on ServiceMonitor and it should already have the proper permission if using kube-prometheus.
Thus, IMO this piece is not very useful.

[camilamacedo86]

The above roles are just useful if we would create the ServiceMonitors programmatically, which is not the case in the kubebuilder. Really tks for your help with @mengqiy

[camilamacedo86]

The PR: https://github.com/kubernetes-sigs/kubebuilder/pull/1317 closes it.

[jgustie]

I'm not sure if I have a different version of the Prometheus operator, but for me the endpoint scheme does not default to https which mismatches the port (i.e. out of the box, Prometheus ends up sending an HTTP request to the HTTPS port). I also had the problem with certificate verification. In the end I was able to get Prometheus to scrape metrics through the RBAC proxy using the following endpoint definition in my ServiceMonitor (without additional role configuration):

    - path: /metrics
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true

[camilamacedo86]

Hi @jgustie.

You have another scenario. Could you please raise a new issue and also add all info required for we reproduce your scenario?

PS. This issue should be closed with #1317.

[caneGuy]

Would also like to point out that the check for the operator metrics given in the comments does not show any of the actual operator metrics, just other metrics from that namespace. No idea if the actual operator metrics do show up in that check, but I had just been checking the prometheus targets that the operator target was up and did not have any errors.

this is also my question