Alertmanager: <http_config> does not support setting of no_proxy

[msglueck]

What did you do?
configure corporate web_proxy with authentication via proxy_url

What did you expect to see?
a configuration option like no_proxy to exclude certain domain patterns like localhost, internal.domain

What did you see instead? Under which circumstances?
" " <-- this is void. na actually I saw an error, when I tried setting undocumented no_proy :(

Environment
Ubuntu 16.04 LTS + 0.18.0 on OpenStack

• System information:

Linux 4.4.0-154-generic x86_64

• Alertmanager version:

alertmanager, version 0.18.0 (branch: HEAD, revision: 1ace0f76b7101cccc149d7298022df36039858ca)
  build user:       root@868685ed3ed0
  build date:       20190708-14:31:49
  go version:       go1.12.6

• Prometheus version:

  insert output of prometheus -version here (if relevant to the issue)

• Alertmanager configuration file:

global:
  http_config:
    proxy_url: http://user:[email protected]:8080
  resolve_timeout: 5m
  slack_api_url: https://hooks.slack.com/services/<token>

• Prometheus configuration file:

not relevant

• Logs:

Jan 28 16:10:22 prometheus-prod alertmanager[26873]: level=error ts=2020-01-28T15:10:22.722Z caller=notify.go:367 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 404 from http://localhost:8765/"
Jan 28 16:10:22 prometheus-prod alertmanager[26873]: level=error ts=2020-01-28T15:10:22.722Z caller=dispatch.go:264 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 404 from http://localhost:8765/"

[pietervincken]

👍

[roidelapluie]

http_proxy in global config is only there if the proxy is global.

All the receivers have their own http_config, so you want to add your corporate proxies to the receivers that need it.

[roidelapluie]

This is however a feature we could consider in prometheus/common, as this is not specific to alertmanager.

[brian-brazil]

I don't think it makes sense generally. If you don't want to use a proxy, don't specify a proxy.

[msglueck]

thanks for the quick response. We somehow failed to configure the proxy_url on the receivers level. maybe did a mistake there. good info. It would work for us, but then we would need to do some routing magic, since we have to support slack.com for our project and an internal system for the corporation. we had the 0.14.0 version before and specified the http_proxy & no_proxy via environment. That didn't work anymore with 0.18.0.

[msglueck]

@brian-brazil we're forced to use the web proxy for internet access, that's the only way to reach e.g. slack.com We now have a local tinyproxy that does the upstream routing for external urls, but having no_proxy for certain domain patterns is a rather common thing in *nix environments.

[roidelapluie]

it is common when you setup the proxy from env variable, which is not the case here as the proxy can be defined explicitly for each receiver.

Should you have more comments or not agree with this, I recommend you to ask the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[simonpasquier]

Closing since it's not specific to Alertmanager. As @roidelapluie already said, https://github.com/prometheus/common is the relevant project here and you'd rather use the mailing list to discuss this topic. Thanks!

[msglueck]

true, it's not a question, more like feature request. that's why I opened it here. there are workarounds and the way you proposed. would have looked into the change myself and opened a PR directly, but a colleague did a similar thing on the blackbox exporter and the PR got declined (we still run the patched version though). So I opted in starting the discussion first. If this ends up in a PR the this would be the right place to have the commits linked to the discussion. :)