Prometheus-operator: Unable to view default prometheus consoles visualization

[xinau]

What did you do?
Try to view the default prometheus consoles in my browser. I did try:

• <prometheus.url>/consoles/index.html.example
• <prometheus.url>/consoles/node.html

What did you expect to see?
I expected to see dashboards as mentioned at https://resin.io/blog

What did you see instead? Under which circumstances?
I saw a single line of text containing open consoles/index.html.example: no such file or directory
Then i tried to open <prometheus.url>/consoles, where i got this response open consoles: no such file or directory. So i assumed prometheus at least knows about the consoles directory. A quick debugging session in the pod showed me the following files and directories.

$ kubectl exec -it -n monitoring prometheus-k8s-0 sh
Defaulting container name to prometheus.
Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod.
/prometheus $ find / -name 'consoles'
/etc/prometheus/consoles
/usr/share/prometheus/consoles
find: /proc/tty/driver: Permission denied
/proc/consoles
/prometheus $ find / -name 'console_libraries'
/etc/prometheus/console_libraries
/usr/share/prometheus/console_libraries
find: /proc/tty/driver: Permission denied
/prometheus $ ls /etc/prometheus/consoles
index.html.example        node-cpu.html             node-disk.html            node-overview.html        node.html                 prometheus-overview.html  prometheus.html
/prometheus $ ls /usr/share/prometheus/consoles
index.html.example        node-cpu.html             node-disk.html            node-overview.html        node.html                 prometheus-overview.html  prometheus.html
/prometheus $ ls /etc/prometheus/console_libraries/
menu.lib  prom.lib
/prometheus $ ls /usr/share/prometheus/console_libraries/
menu.lib  prom.lib

Environment

Manifests:

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app: prometheus
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    beta.kubernetes.io/os: linux
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector:
    matchExpressions:
    - key: k8s-app
      operator: Exists
  version: v2.2.1

---
apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  ports:
  - name: web
    port: 9090
    targetPort: web
  selector:
    app: prometheus
    prometheus: k8s

[brancz]

Yes, you're absolutely right, we are not setting the flags for the console templates and libraries correctly. We can just start out with a pre-initialized array of flags here: https://github.com/coreos/prometheus-operator/blob/14d647a5c6b42a58dd172e4314baf8f6ca4b87c2/pkg/prometheus/statefulset.go#L289

The two flags we would want to add here are:

• -web.console.templates=/etc/prometheus/consoles
• -web.console.libraries=/etc/prometheus/console-libraries

[allisonking]

I think the flag should be:
-web.console.libraries=/etc/prometheus/console_libraries, not -web.console.libraries=/etc/prometheus/console-libraries (underscore vs. hyphen).

I get the error:
error executing template __console_/index.html.example: html/template:__console_/index.html.example:28:12: no such template "tail"
when I navigate to /consoles/index.example.html, which seems to tell me that it can find the templates but not the libraries.

And looking in the pod:

allisonking$ kubectl exec -it prometheus-main-0 -c prometheus ls /etc/prometheus
config_out         consoles           rules
console_libraries  prometheus.yml

the directory is console_libraries

[squat]

As @allisonking notes, this issue still exists. The fix is to change line https://github.com/coreos/prometheus-operator/blob/master/pkg/prometheus/statefulset.go#L299
from

"-web.console.libraries=/etc/prometheus/console-libraries",

to

"-web.console.libraries=/etc/prometheus/console_libraries",

This is a good first-time contribution to the project.