Prometheus-operator: ServiceMonitors doesn't work for custom service

[xuchenhao001]

What did you do?
Created a custom deployment & service (ceph)
What did you expect to see?
prometheus automatically add a target to ceph
What did you see instead? Under which circumstances?
There isn't any error, nor new prometheus target. But new service monitor is created successfully.

$ kubectl get servicemonitors -n monitoring
NAME                      AGE
alertmanager              6d
ceph-exporter             1m
kube-apiserver            6d
kube-controller-manager   6d
kube-scheduler            6d
kube-state-metrics        6d
kubelet                   6d
node-exporter             6d
prometheus                6d
prometheus-operator       6d

Environment
Ubuntu 16.04 x86, Kubernetes v1.8.4, prometheus-operator v0.17.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

kubeadm, with calico network.

• Manifests:

# ceph-exporter.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ceph-exporter
    k8s-app: ceph-exporter
  name: ceph-exporter
  namespace: monitoring
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: http-metrics
    port: 9128
    protocol: TCP
  selector:
    app: ceph-exporter
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ceph-exporter
  namespace: monitoring
spec:
  template:
    metadata:
      labels:
        app: ceph-exporter
      name: ceph-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - image: digitalocean/ceph_exporter:latest
        imagePullPolicy: Never
        name: ceph-exporter
        ports:
        - containerPort: 9128
          hostPort: 9128
          name: scrape
        resources:
          requests:
            memory: 30Mi
            cpu: 100m
          limits:
            memory: 50Mi
            cpu: 200m
        volumeMounts:
        - name: proc
          readOnly:  true
          mountPath: /etc/ceph
      volumes:
      - name: proc
        hostPath:
          path: /etc/ceph
      nodeSelector:
        kubernetes.io/hostname: ubuntu

# prometheus-k8s-service-monitor-ceph-exporter.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ceph-exporter
  namespace: monitoring
  labels:
    k8s-apps: ceph-exporter
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      k8s-app: ceph-exporter
  namespaceSelector:
    matchNames:
    - monitoring
  endpoints:
  - port: http-metrics
    interval: 30s

• Prometheus Operator Logs:
  (I have tried to restart the prometheus-operator, but there still no new target for prometheus)

ts=2018-03-07T09:38:49.914578137Z caller=operator.go:258 component=prometheusoperator msg="connection established" cluster-version=v1.8.4
ts=2018-03-07T09:38:49.91471812Z caller=operator.go:158 component=alertmanageroperator msg="connection established" cluster-version=v1.8.4
ts=2018-03-07T09:38:49.929148142Z caller=operator.go:172 component=alertmanageroperator msg="CRD API endpoints ready"
ts=2018-03-07T09:38:49.932390898Z caller=operator.go:272 component=prometheusoperator msg="CRD API endpoints ready"
ts=2018-03-07T09:38:50.082318312Z caller=operator.go:308 component=prometheusoperator msg="Prometheus added" key=monitoring/k8s
ts=2018-03-07T09:38:50.082505859Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:38:50.082280509Z caller=operator.go:305 component=alertmanageroperator msg="Alertmanager added" key=monitoring/main
ts=2018-03-07T09:38:50.08531966Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:38:50.0870151Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:38:50.087808578Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:38:50.590523482Z caller=operator.go:984 component=prometheusoperator msg="current config or current configmaps has changed"
ts=2018-03-07T09:38:50.590613238Z caller=operator.go:990 component=prometheusoperator msg="updating configuration"
E0307 09:38:50.682473       1 operator.go:257] Sync "monitoring/main" failed: creating statefulset failed: statefulsets.apps "alertmanager-main" already exists
ts=2018-03-07T09:38:50.687159579Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:38:50.793062951Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:38:50.802748431Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:38:50.815838844Z caller=operator.go:984 component=prometheusoperator msg="current config or current configmaps has changed"
ts=2018-03-07T09:38:50.815960764Z caller=operator.go:990 component=prometheusoperator msg="updating configuration"
ts=2018-03-07T09:38:50.892116231Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:38:50.908350319Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:40:20.09589673Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:40:20.096148755Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:40:23.532671398Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:40:23.53288914Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:40:23.606864101Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:40:45.714071388Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:40:45.791757272Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:40:45.812284316Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:40:45.906687283Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:41:43.302576215Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:41:43.302873762Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:41:43.505626819Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:41:48.614219072Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:41:48.614373448Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:42:01.462352909Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:42:01.495549672Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:42:01.528046371Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:42:01.591183182Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:42:45.982758242Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:42:45.983066722Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:42:46.00880287Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:42:57.786321632Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:42:57.789204875Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:43:03.54080239Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:03.59190788Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:43:03.616215846Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:03.6952368Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:43:50.082808469Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:43:50.083308072Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:50.082829057Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:43:50.083524675Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:43:50.106185156Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:43:50.211661498Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:50.287570687Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:43:50.38321012Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2432996 cur=2432996
ts=2018-03-07T09:43:50.383366562Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2433179 cur=2433179
ts=2018-03-07T09:43:50.383450429Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2433289 cur=2433289
ts=2018-03-07T09:43:50.492188336Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:50.492487474Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2433289 cur=2433289
ts=2018-03-07T09:43:50.492552754Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2432996 cur=2432996
ts=2018-03-07T09:43:50.492584004Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2433179 cur=2433179
ts=2018-03-07T09:43:50.585532246Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:43:50.686175101Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:43:50.699363385Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:44:29.987295942Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:44:29.987474765Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:44:30.083334828Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:44:45.382655125Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:44:45.481259425Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:44:45.501898931Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:44:45.515812446Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:44:52.241064271Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:44:52.241211841Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:45:53.581799055Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:45:53.582072262Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:45:58.236557998Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:45:58.289857988Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:45:58.306873656Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:45:58.495712437Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:46:18.590691304Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:46:18.590998375Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:46:18.606960956Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:47:22.060017811Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:47:22.126282Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:47:22.177378736Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:47:22.196732043Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:47:32.870600741Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:47:32.870764587Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:48:17.202378079Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:48:17.202584381Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:17.30373709Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:38.45526901Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:38.589414351Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:38.607260584Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:38.68697762Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.083156132Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:48:50.083452783Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2018-03-07T09:48:50.083274107Z caller=operator.go:328 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2018-03-07T09:48:50.084007682Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.188893357Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.209259877Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.383626135Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2433179 cur=2433179
ts=2018-03-07T09:48:50.384013262Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2433289 cur=2433289
ts=2018-03-07T09:48:50.384051207Z caller=operator.go:637 component=prometheusoperator msg="update handler" old=2432996 cur=2432996
ts=2018-03-07T09:48:50.392390305Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.492864136Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.492837959Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2433289 cur=2433289
ts=2018-03-07T09:48:50.493264161Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2432996 cur=2432996
ts=2018-03-07T09:48:50.493305075Z caller=operator.go:345 component=alertmanageroperator msg="update handler" old=2433179 cur=2433179
ts=2018-03-07T09:48:50.510396874Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.542957974Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.555750143Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.686482105Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.711854187Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:48:50.788433623Z caller=operator.go:672 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2018-03-07T09:48:50.804304208Z caller=operator.go:981 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2018-03-07T09:49:19.730651984Z caller=operator.go:325 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2018-03-07T09:49:19.730756825Z caller=operator.go:381 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main

[brancz]

Could you share the generated Prometheus config? (you can find it in /config of the Prometheus UI)

[xuchenhao001]

Sure. Here is the /config:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
alerting:
  alertmanagers:
  - kubernetes_sd_configs:
    - api_server: null
      role: endpoints
      namespaces:
        names:
        - monitoring
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager-main
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: web
      replacement: $1
      action: keep
rule_files:
- /etc/prometheus/rules/rules-0/*
scrape_configs:
- job_name: monitoring/alertmanager/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_alertmanager]
    separator: ;
    regex: main
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace
- job_name: monitoring/kube-apiserver/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - default
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
    insecure_skip_verify: false
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: apiserver
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_provider]
    separator: ;
    regex: kubernetes
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https
    action: replace
- job_name: monitoring/kube-controller-manager/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kube-controller-manager
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
- job_name: monitoring/kube-scheduler/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kube-scheduler
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
- job_name: monitoring/kube-state-metrics/0
  honor_labels: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kube-state-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-main
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-main
    action: replace
- job_name: monitoring/kube-state-metrics/1
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kube-state-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-self
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-self
    action: replace
- job_name: monitoring/kubelet/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
- job_name: monitoring/kubelet/1
  honor_labels: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
- job_name: monitoring/node-exporter/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: node-exporter
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https
    action: replace
- job_name: monitoring/prometheus/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_prometheus]
    separator: ;
    regex: k8s
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web
    action: replace
- job_name: monitoring/prometheus-operator/0
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - monitoring
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: prometheus-operator
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace

[xuchenhao001]

Problem solved due to issue890. I record this for those who suffering with the same promblem:
First checkout the prometheus object:

$ kubectl get prometheus -o yaml --all-namespaces
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"replicas":2,"resources":{"requests":{"memory":"400Mi"}},"ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"prometheus-rulefiles"}},"serviceAccountName":"prometheus-k8s","serviceMonitorSelector":{"matchExpressions":[{"key":"k8s-app","operator":"Exists"}]},"version":"v2.2.0-rc.0"}}
    clusterName: ""
    creationTimestamp: 2018-03-07T13:14:05Z
    deletionGracePeriodSeconds: null
    deletionTimestamp: null
    labels:
      prometheus: k8s
    name: k8s
    namespace: monitoring
    resourceVersion: "1487"
    selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/k8s
    uid: 697c2f2c-2209-11e8-9296-90fba61fb1ee
  spec:
    alerting:
      alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
    replicas: 2
    resources:
      requests:
        memory: 400Mi
    ruleSelector:
      matchLabels:
        prometheus: k8s
        role: prometheus-rulefiles
    serviceAccountName: prometheus-k8s
    serviceMonitorSelector:
      matchExpressions:
      - key: k8s-app
        operator: Exists
    version: v2.2.0-rc.0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Notice that:

    serviceMonitorSelector:
      matchExpressions:
      - key: k8s-app
        operator: Exists

So my ServiceMonitor object must contain a label with key k8s-app.
However, I used to have a label k8s-apps (that works before I upgrade prometheus-operator).