Prometheus-operator: ServiceMonitor makes job appears in Configuration but missing in Targets

[Vince-Cercury]

What did you do?
I have a service exposing Cassandra created in a namespace called 'vince1':

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  ports:
  - name: promjmx
    port: 7070
  - name: "9042"
    port: 9042
    targetPort: 9042
  selector:
    app: cassandra

This service works for its purpose.

I then created a ServiceMonitor in 'monitor' namespace:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
  name: cassandra-k8s
  labels:
    k8s-app: cassandra-k8s
spec:
  jobLabel: cassandra-k8s
  endpoints: 
  - port: promjmx
    interval: 30s
  selector:
    matchLabels:
      app: cassandra
  namespaceSelector:
    any: true

The job appears in the Configuration page of Prometheus if I create it in 'monitoring'. But does not if I create in 'vince1'. I wasn't sure if it needs to be next to the actual service.

What did you expect to see?
I expected to see a new target called 'cassandra' on Targets page.

What did you see instead? Under which circumstances?
I only see the job configuration:

- job_name: monitoring/cassandra-k8s/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names: []
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: cassandra
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: promjmx
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_cassandra_k8s]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: []
    separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: promjmx
    action: replace

Environment
Prometheus Operator v0.11.1
Installed via the hack deploy tool.

• Kubernetes version information:

kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-12T22:24:26Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3+coreos.0", GitCommit:"70bc0937018d89b0d529d428eed8c444d8e7729a", GitTreeState:"clean", BuildDate:"2017-05-11T18:15:57Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  cluster created via kube-aws

• Manifests:
  see above

• Prometheus Operator Logs:
  I didn't find anything in operator log that might help

[Vince-Cercury]

I found an error in logs:
User "system:serviceaccount:monitoring:prometheus-k8s" cannot list services in the namespace vince1

I would have thought the monitoring account created by Prometheus Operator could by default read in other namespaces.

[Vince-Cercury]

the solution was to create cluster role with read access to all namespaces:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
- apiGroups: [""]
  resources:
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]

If that is the way I should do this, let me know and we can close this ticket

[brancz]

The kube-prometheus manifests are merely an example of how cluster monitoring can be done, as in "how can I easily monitor my Kubernetes cluster and specifically the Kubernetes components with Prometheus?". The RBAC roles for this are highly detailed to fit exactly this need, following the principle of least privilege.

That said, the kube-prometheus manifests are totally meant to be "take and adapt", so you did everything just right!

In your specific case I would suggest to have a separate Prometheus object/instance though, as then you can do finely grained RBAC roles and also make the single deployments of Prometheus only care about your application or stack. A very typical use case of Prometheus is to have one set of Prometheus instances that monitor the Kubernetes cluster itself and workload on a generic basis, and then per team or stack (or whatever reflects ownership within your organization) a set of Prometheus servers that are specific to those cases.

As your initial issue is resolved, I'll close this here, but feel free to ask any further questions, should you have any. :slightly_smiling_face: