Charts: Prometheus chart healthcheck does not honor web.external-url

Created on 8 Aug 2017 · 7Comments · Source: helm/charts

Bug Report:

When defining a custom ingress rule based on paths[1], you'll need to specify web.external-url[2] as additional args. When this parameter is set, the readiness probe fails because :9090/status responds with a 404[3]. In this case, the readiness probe should honor the path defined byweb.external-url or web.route-prefix so that Prometheus stays running.

Kubernetes 1.7.2
Helm 2.5.1
Helm command run to reproduce:

helm install stable/prometheus  --set alertmanager.image.tag=v0.8.0 --set kubeStateMetrics.image.tag=v1.0.0-rc.1  --set nodeExporter.image.tag=v0.14.0 --set server.image.tag=v1.7.1 -f vars.yaml

[1]

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus
  namespace: default
  annotations:
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /prometheus
        backend:
          serviceName: sappy-chinchilla-prometheus-server
          servicePort: 80

[2]
vars.yaml

server:
  extraArgs:
    web.external-url: http://mydomain.com/prometheus

[3]

ubuntu@control-01:~$ curl 10.233.86.29:9090/status
404 page not found
ubuntu@control-01:~$ curl 10.233.86.29:9090
<a href="/prometheus">Found</a>.

ubuntu@control-01:~$ curl 10.233.86.29:9090/prometheus
<a href="/prometheus/">Moved Permanently</a>.

ubuntu@control-01:~$ curl 10.233.86.29:9090/prometheus/
<a href="/prometheus/graph">Found</a>.

ubuntu@control-01:~$ curl 10.233.86.29:9090/prometheus/status
<!DOCTYPE html>
<html lang="en">
...

lifecyclrotten

Source

bradbeam

👍2

Most helpful comment

@bradbeam i figured this out after digging around the prometheus source code.

You also need to add this cli option

server:
  extraArgs:
    web.route-prefix: "/"

jwhitcraft on 11 Jan 2018

🎉3

All 7 comments

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

fejta-bot on 2 Jan 2018

I just ran into this problem as well and I don't believe that you can use a readiness probe with the chart when using the external-url param. I came this conclusion because the pod never becomes ready, which means it's never actually allowed to be available in the service.

I'll do some more digging, but this does seem to be an issue.

jwhitcraft on 11 Jan 2018

@bradbeam i figured this out after digging around the prometheus source code.

You also need to add this cli option

server:
  extraArgs:
    web.route-prefix: "/"

jwhitcraft on 11 Jan 2018

🎉3

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

fejta-bot on 11 Feb 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 13 Mar 2018

@bradbeam i figured this out after digging around the prometheus source code.

You also need to add this cli option
server:
  extraArgs:
    web.route-prefix: "/"

This should be documented somewhere. Saved me so much time.

nvenky on 16 May 2019

👍1

I know this ticket is already closed, but I'd like to add that the current stable/prometheus helm chart has the following value available that you can set to define the external-url:

By default:
baseURL: ""

Change it to:
baseURL: "http://<insert-your-url-here/prometheus"

When you launch the helm chart with baseURL defined, it will create the pod/container with the extraArg --external-url=baseURL

You still need to set the web.route-prefix: "/" extraArg for the readiness/liveness probes to work (which is need for the container to come up):