Prometheus-operator: Adding new Service Monitor to Prometheus, cannot validate certificate

[ohmanandam]

What did you do?
I have downloaded the prometheus operator repo locally and deployed the promethes operator and kube prometheus via helm

$ helm install prometheus-operator --name prometheus-operator --namespace monitoring
$ helm install kube-prometheus --name kube-prometheus --set global.rbacEnable=true --namespace monitoring

I need to add few endpoints in the same/different namespaces, however to get started i moved my Scrape target to monitoring namespace following the earlier issues

https://github.com/coreos/prometheus-operator/issues/1487
https://github.com/coreos/prometheus-operator/issues/1532

Now my ServiceMonitor with basic Auth appears as follows
`---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app: logiq-ash
name: basic-auth-ash
namespace: monitoring
spec:
endpoints:

• basicAuth:
  password:
  name: auth
  key: password
  username:
  name: auth
  key: user
  targetPort: 9998
  #port: "https"
  interval: 30s
  path: /metrics
  scheme: https
  namespaceSelector:
  matchNames:
  
  
  
  • monitoring
    
    selector:
    
    matchLabels:
  
  

app: logiq-ash

apiVersion: v1
kind: Secret
metadata:
name: auth
namespace: monitoring
data:
password: Zmxhc2hfcjcmV0
user: AZmxhc2hfYWXNz
type: Opaque

As seen below, my prometheus scrape target app is running in monitoring namespace

kubectl get pods -n monitoring

NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-0 2/2 Running 0 13h
kube-prometheus-exporter-kube-state-76f498d465-flwp2 2/2 Running 0 13h
kube-prometheus-exporter-node-79nrp 1/1 Running 0 13h
kube-prometheus-grafana-57d5b4d79f-c6b2b 2/2 Running 0 13h
logiq-ash-0 1/1 Running 0 13h
logiq-ash-1 1/1 Running 0 13h
prometheus-kube-prometheus-0 3/3 Running 1 13h
prometheus-operator-d75587d6-l4rn9 1/1 Running 0 14h

Now i create the ServiceMonitor to add my new prometheus scrape end point

kubectl create -f prometheus-operator/basic-auth.yaml

servicemonitor.monitoring.coreos.com "basic-auth-ash" created
secret "auth" created

As seen below, the Service Monitor seems to get created in monitoring namespace

kubectl get ServiceMonitor -n monitoring

NAME AGE
basic-auth-ash 9s
kube-prometheus 14h
kube-prometheus-alertmanager 14h
kube-prometheus-exporter-kube-controller-manager 14h
kube-prometheus-exporter-kube-dns 14h
kube-prometheus-exporter-kube-etcd 14h
kube-prometheus-exporter-kube-scheduler 14h
kube-prometheus-exporter-kube-state 14h
kube-prometheus-exporter-kubelets 14h
kube-prometheus-exporter-kubernetes 14h
kube-prometheus-exporter-node 14h
kube-prometheus-grafana 14h
prometheus-operator 14h

`
What did you expect to see?
I expect to see basic-auth-ash added to prometheus config, how i do not see it being added at all, here is the config from prometheus console.

global:
scrape_interval: 30s
scrape_timeout: 10s
evaluation_interval: 30s
external_labels:
prometheus: monitoring/kube-prometheus
prometheus_replica: prometheus-kube-prometheus-0
alerting:
alert_relabel_configs:

• separator: ;
  regex: prometheus_replica
  replacement: $1
  action: labeldrop
  alertmanagers:
• kubernetes_sd_configs:
  
  
  
  • api_server: null
    
    role: endpoints
    
    namespaces:
    
    names:
    
    
    
    
    
    • monitoring
      
      
      scheme: http
      
      
      path_prefix: /
      
      
      timeout: 10s
      
      
      relabel_configs:
    
    
    
  
  
  • source_labels: [__meta_kubernetes_service_name]
    
    separator: ;
    
    regex: kube-prometheus-alertmanager
    
    replacement: $1
    
    action: keep
  
  
  • source_labels: [__meta_kubernetes_endpoint_port_name]
    
    separator: ;
    
    regex: http
    
    replacement: $1
    
    action: keep
    
    rule_files:
  
  
  • /etc/prometheus/rules/*.yaml
    
    scrape_configs:
  
  
  • job_name: monitoring/kube-prometheus/0
    
    scrape_interval: 30s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • monitoring
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: prometheus
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_chart]
  separator: ;
  regex: prometheus-0.0.49
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_prometheus]
  separator: ;
  regex: kube-prometheus
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-alertmanager/0
    
    scrape_interval: 30s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • monitoring
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_alertmanager]
  separator: ;
  regex: kube-prometheus
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: alertmanager
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_chart]
  separator: ;
  regex: alertmanager-0.1.3
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-controller-manager/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    tls_config:
    
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    
    insecure_skip_verify: true
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-controller-manager
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-controller-manager
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-dns/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-dns
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-dns
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics-dnsmasq
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics-dnsmasq
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-dns/1
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-dns
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-dns
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics-skydns
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics-skydns
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-etcd/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-etcd
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-etcd
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-scheduler/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-scheduler
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-scheduler
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kube-state/0
    
    honor_labels: true
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • monitoring
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-kube-state
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: kube-state
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: kube-state-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: kube-state-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kubelets/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: https
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    tls_config:
    
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    
    insecure_skip_verify: true
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_k8s_app]
  separator: ;
  regex: kubelet
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: https-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: https-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kubelets/1
    
    honor_labels: true
    
    scrape_interval: 30s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics/cadvisor
    
    scheme: https
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • kube-system
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    tls_config:
    
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    
    insecure_skip_verify: true
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_k8s_app]
  separator: ;
  regex: kubelet
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: https-metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: https-metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-kubernetes/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: https
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • default
    
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    
    tls_config:
    
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    
    insecure_skip_verify: true
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: apiserver
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_provider]
  separator: ;
  regex: kubernetes
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: https
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: https
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-exporter-node/0
    
    scrape_interval: 15s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • monitoring
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: exporter-node
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: node-exporter
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: metrics
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_component]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: metrics
  action: replace
  
  
  
  • job_name: monitoring/kube-prometheus-grafana/0
    
    scrape_interval: 30s
    
    scrape_timeout: 10s
    
    metrics_path: /metrics
    
    scheme: http
    
    kubernetes_sd_configs:
  
  
• api_server: null
  role: endpoints
  namespaces:
  names:
  
  
  
  • monitoring
    
    relabel_configs:
  
  
• source_labels: [__meta_kubernetes_service_label_app]
  separator: ;
  regex: kube-prometheus-grafana
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_service_label_chart]
  separator: ;
  regex: grafana-0.0.36
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: http
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_service_label_kube_prometheus_grafana]
  separator: ;
  regex: (.+)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: http
  action: replace

What did you see instead? Under which circumstances?
Athough the new ServiceMonitor (basic-auth-ash) is deployed, but it fails to get added to Prometheus config. The prometheus operator logs do not see any updates as well

Environment
Prometheus Operator downloaded from master on 7/9/2018

helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

• Kubernetes version information:

  insert output of kubectl version here
  kubectl version
  Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.
  minikube

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:
  level=info ts=2018-07-15T20:07:36.325968639Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:07:43.685723496Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:08:27.406024521Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:08:27.523566626Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:08:54.682947844Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:09:07.471167322Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:09:07.539352773Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:09:16.073641595Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:09:58.822346685Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:10:21.995862016Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
  level=info ts=2018-07-15T20:10:22.105502504Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-pro

insert Prometheus Operator logs relevant to the issue here
no updates

[ohmanandam]

An earlier issue is helping to get closer to resolving this
https://github.com/coreos/prometheus-operator/issues/1470

[ohmanandam]

After following issue #1470 and adding the label to contain prometheus: kube-prometheus, my scrape end point in picked up as Prometheus target. However I am running into the below issue

Get https://172.17.0.7:9998/metrics: x509: cannot validate certificate for 172.17.0.7 because it doesn't contain any IP SANs

I may need to add tls_config insecure_skip_verify as true to bypass the above issue, i tried adding it to my ServerMonitor spec, however this is not picked up in the Prometheus configuration. This attribute shows up fine via Kubectl describe ServiceMonitor.
`
spec:
endpoints:

• basicAuth:
  password:
  name: auth
  key: password
  username:
  name: auth
  key: user
  targetPort: 9998
  interval: 40s
  path: /metrics
  scheme: https
  tls_config:
  insecure_skip_verify: true
  namespaceSelector:
  matchNames:
  
  
  
  • monitoring
    
    selector:
    
    matchLabels:
    
    app: logiq-ash
    
    `
  
  

[ohmanandam]

https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/monitoring-external-etcd.md

I needed to use in my ServiceMonitor
tlsConfig:
insecureSkipVerify: true

Thanks Prometheus Operator team, amazing work with the prometheus and k8s config!

[mxinden]

@ohmanandam Thanks for documenting your solution path here!