Charts: [stable/prometheus-operator] Expose grafana, prometheus, and alertmanager via nginx-ingress

[amilkov3]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Version of Helm and Kubernetes: v2.13.1 and v1.11.8-eks-7c34c0

Which chart: prometheus-operator

What happened:

What you expected to happen: All I want is to expose prometheus server at /prometheus,
grafana at /grafana and alertmanager at /alertmanager

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:
I'm running the nginx-ingress chart and have my.host.com (not actual domain obviously) pointed at its ELB. I don't see any docs on how to achieve this with the prometheus-operator chart specifically Here is my prometheus-operator config:

kubeScheduler:
    enabled: false
kubeControllerManager:
  enabled: false

prometheus:
  additionalServiceMonitors:
    - name: user-prometheus
      namespaceSelector:
        any: true
      selector:
        matchLabels:
          app: prometheus
      endpoints:
        - port: http
          path: /federate
          honorLabels: true
          params:
            match[]:
              - '{__name__=~".+"}'
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /$1
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    hosts:
      - my.host.com
    paths:
      - /prometheus
    prometheusSpec:
      externalUrl: https://my.host.com/prometheus/
    serviceMonitorSelector:
      matchLabels:
        prometheus: kube-prometheus

commonLabels:
  prometheus: kube-prometheus


grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    hosts:
      - my.host.com
    path: /grafana
  grafana.ini:
    server:
      root_url: https://my.host.com/grafana

[amilkov3]

Got it working. Mostly. Here's my config:

prometheus:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: basic-auth
      nginx.ingress.kubernetes.io/rewrite-target: /$1
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    hosts:
      - my.host.com
    paths:
      - /prometheus/?(.*)
  prometheusSpec:
    externalUrl: https://my.host.com/prometheus/

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
      nginx.ingress.kubernetes.io/rewrite-target: /$1
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    hosts:
      - my.host.com
    path: /grafana/?(.*)
  grafana.ini:
    server:
      root_url: https://my.host.com/grafana/

Unfortunately setting root_url does propogate to your grafana ConfigMap but when checking /etc/grafana/grafana.ini in the container, it is not set so i had to add:

- name: GF_ROOT_URL
  value: https://my.host.com/grafana/

to the grafana Deployment. I also added basic auth for prometheus server UI via the method described here: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx/examples/auth

have yet to expose alertmanager but anticipate the process is similar to prometheus.

I would like to know if there is a way to set root_url for grafana via prometheus-operator and where I can update the docs for all of this, since it is completely unnecessary for someone to have fiddle with something so basic for 10 hours before finally stumbling on a solution on their own

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[phumberdroz]

Hey I am currently facing the same issues.

Can you maybe explain to me what you did ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[shibumi]

@amilkov3 Hi, do you remember how you got the basic-Auth working? I tried this for my alertmanager and it runs into an error 500:

alertmanager:
  ingress:
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: alertmanager-auth
    enabled: true
    ...

EDIT: Solved it via looking into the nginx ingress logs. The ingress didn't support bcrypt so I changed to another algorithm.