Alertmanager: external location: alertmanager uses external locations for internal listener

[kayrus]

For example prometheus configured with the -alertmanager.url=http://alertmanager:9093 option.

And alertmanager has -web.external-url=https://my-external-ingress-controller.example.com/alertmanager option.

For some reason alertmanager uses /alertmanager location even for internal listeners and as a result you receive time="2016-10-06T11:20:37Z" level=error msg="Error sending alerts: bad response status 404 Not Found" alertmanager="http://alertmanager:9093" count=3 source="notifier.go:308" error message in prometheus.

External and internal configuration options should be independent.

[brian-brazil]

I believe this behaviour is correct. You told the alertmanager it'd only be accessed via a prefix, and then did not access it via that prefix.

[kayrus]

@brian-brazil that's not correct. my prometheus ingress location has my-external-ingress-controller.example.com domain name. And this location serves several services including alertmanager. And when you define external-url you define _external_ url, not internal.

[brian-brazil]

The URL provided is what all external access to the AM is via. There is no internal AM http.

[groob]

I ran into this issue earlier today and wanted to document my usecase:

I have prometheus and alertmanager deployed on a k8s cluster which is publicly accessible(not behind VPM/firewall). We use an ingress controller which
a) terminates TLS and
b) optionally forces OAuth authentication for users to access various dashboards, internal pages etc.
So far it hasn't been a problem as most applications either have some form of robust auth on their own, or allow direct access to the backend unsecured(in which case we proxy it through oauth).

For the setup to work, I need the -web.external-url flag because I want the users to get the correct URL in their alerts, but I do not want to expose the alertmanager API directly to the public. Unfortunately, there's no way of pointing prometheus to a external URL which will force the request to go through a custom authentication proxy.

What would be a good approach to resolve this issue? I'd be glad to submit a PR for separating the API and external URL if needed.

[brian-brazil]

Unfortunately, there's no way of pointing prometheus to a external URL which will force the request to go through a custom authentication proxy.

That's configurable on the Prometheus side.

[mrwacky42]

Related...
If I'm using -web.external-url=https://example.com/alertmanager to reverse proxy to Alertmanager, the /metrics endpoint is also moved to /alertmanager/metrics, which is unfortunate from my perspective. A 302 to /alertmanager/metrics would be nice.

[alsm]

This is still an issue. I have a prometheus that is sending alerts to an alertmanager running on the same host, but an external display of the alerts comes in via a reverse proxy, the hostname of the reverse proxy is not visible to alertmanager or prometheus but I need this url in the alerts that are generated.
ie: I need to say alertmanager is here, but alert urls need to be generated to point to here instead

[Kellel]

As mentioned in https://github.com/prometheus/alertmanager/pull/706

Actually starting with the 0.7.0 release the Alertmanager has the -web.route-prefix flag in addition to
the -web.external-url just like Prometheus, to be able to control the prefix of the internal http handlers
and how Alertmanager is accessed.

You should be able to use the -web.route-prefix for this requirement.

[brian-brazil]

Sounds like this was incorrect usage.

[nelsonfassis]

I have a similar issue I believe.
If I use "--web.external-url" with an actual external url on my deployment, the external URL works for accessing, but Prometheus fails to communicate with alertmanager (alertmanager-0.alertmanager and alertmanager-1.alertmanager).

If I remove the "--web.external-url", Prometheus works properly, but the link on slack alert will point to either alertmanager-0.alertmanager or alertmanager-1.alertmanager, which are not accessible outside of kubernetes cluster.

  containers:
  - name: alertmanager
    image: prom/alertmanager:v0.12.0
    args:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'
      # - '--web.external-url=http://myaddress.com:31080/alertmanager'
      - '--mesh.listen-address=:6783'
      - '--mesh.peer=alertmanager-mesh'

[stuartnelson3]

Is your prometheus communicating within the kubernetes cluster with the alert manager?

[nelsonfassis]

Yes @stuartnelson3 , I have a prometheus Deployment and an Alertmanager statefulset. The communication on network level is fine, I can ping services, name resolution works. The problem is if I set the '--web.external-url=http://myaddress.com:31080/alertmanager' parameter, being myaddress.com my cluster address, 31080 my ingress controller service port and /alertmanager my ingress path. If I set those values, seems like alertmanager will only listen in this path.

In this case, If I set my targets on Prometheus to - alertmanager-1.alertmanager:9093 (alertmanager-0 is the pod name, alertmanager is the service name), the connection will be refused by alertmanager.

If I set my targets on Prometheus to - myaddress.com:31080, it will build a path myaddress.com:31080/api/v1/alerts, it doesn't let me change to myaddress.com:31080/alertmanager/api/v1/alerts (if I could do it, I think it would work, but then I'd be sending alerts to the service and not to the pods as recommended on some articles.

If I remove this option '--web.external-url=http://myaddress.com:31080/alertmanager' from my alertmanager, Prometheus and Alertmanager work as they should, but then my URLs on Slack alerts will be directing to http://alertmanager-0.alertmanager:9093 which isn't available outside of Kubernetes (users for example).

[brian-brazil]

myaddress.com:31080, it will build a path myaddress.com:31080/api/v1/alerts, it doesn't let me change to myaddress.com:31080/alertmanager/api/v1/alert

You want the path_prefix option.

[nelsonfassis]

@brian-brazil I used path_prefix like this on my Prometheus configmap and it didn't work (nothing happened actually). I even deleted the deployment and redeployed thinking that maybe the curl -XPOST myaddress.com:31080/prometheus/-/reload wasn't working properly (and it would be a different problem), but no... still doesn't work. My prometheus target keep being myaddress.com:31080/api/v1/alerts.

Am I setting it in the wrong place? Thank you so much

alerting:
        alertmanagers:
        - path_prefix: alertmanager
        - scheme: http
          static_configs:
          - targets:
             - myaddress.com:31080
             - alertmanager-0.alertmanager.monitoring.svc.cluster.local:9093
             - alertmanager-1.alertmanager.monitoring.svc.cluster.local:9093

[brian-brazil]

The path prefix goes with the existing definition, not as a new alertmanager.

[nelsonfassis]

@brian-brazil Sorry, but I don't know where do you mean. Tried "randomly" putting this "path_prefix: alertmanager" on my documents and they didn't work. Also I can't find documentation or samples of path_prefix anywhere. Could you please be a bit more specific?

[muniyalr]

Should the path_prefix be prefixed with a /

- path_prefix: "/alertmanager"

[nelsonfassis]

@muniyalr made no difference. with or without "/", the configuration is shown the same on my mydomain:31080/prometheus/config

`

global:
  scrape_interval: 15s
  scrape_timeout: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: my-monitor
alerting:
  alertmanagers:
  - scheme: http
    path_prefix: /alertmanager
    timeout: 10s
  - static_configs:
    - targets:
      - mydomain:31080
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus-rules/prom.rules

`

[nelsonfassis]

@muniyalr @brian-brazil It seems to me that most adequate way would be to have Alertmanager accessible by its host address (alertmanager-0.alertmanager) AND from its external-url.

After all, if it is external-url, it should only matter to whoever is accessing it from a different network (like users). And other services on the same network (as Prometheus) trying to access Alertmanager on :9093 should just work. As I see, the external-url option shouldn't bind the service to this address.

[brian-brazil]

As discussed above, that doesn't make sense. Syntax questions are best handled on the mailing list.

[nelsonfassis]

@brian-brazil How does it not make sense? Questions about the same thing on mailing list are open without answers since November last year. Clearly it is not just me having this issue and documentation about the path_prefix you mentioned is nearly inexistent.

[nelsonfassis]

@muniyalr Fixed my problem as you suggested with "/", but at the end.
`

alerting:
      alertmanagers:
      - scheme: http
        path_prefix: "alertmanager/"
        static_configs:
        - targets:
            - alertmanager-0.alertmanager:9093
            - alertmanager-1.alertmanager:9093

`

Now it works

[Sean-Bradley]

changing --web.external-url also affects the internal url if it contains a path. I find, this creates more problems than it fixes.

But anyway, for other people reading this,

If you find that using a reverse proxy, with a custom path, eg /alertmanager/ breaks your internal alertmanager urls, since all internal references for other processes now also need to use /alertmanager/ in the localhost:9093 url,
then you can also add --web.route-prefix=/ to the alertmanager execute command

eg,
./alertmanager --config.file=alertmanager.yml --web.external-url=https://[your domain]/alertmanager --web.route-prefix=/

See my docs here for an example,
https://prometheustutorials.seanwasere.com/alert-manager/
See In the section where I create the service.