Alertmanager: Needs RFC1918 Private Address or --cluster.advertise-address flag to start alertmanager

[skorpy2009]

What did you do?
Try to start the alertmanager. I used /opt/alertmanager-0.15.0.linux-amd64/alertmanager --config.file /etc/prometheus/alertmanager.yml
What did you expect to see?
See the alertmanager start.
What did you see instead? Under which circumstances?
Alertmanager did not start. It started when I added 192.168.2.2/24 to any of my interfaces or when I added the --cluster.advertise-address flag.
It seems like it has something to do with: https://github.com/hashicorp/memberlist/blob/022f081/net_transport.go#L144

Environment

• System information:

  Linux 4.16.0-0.bpo.2-amd64 x86_64

• Alertmanager version:

alertmanager, version 0.15.0 (branch: HEAD, revision: 462c969d85cf1a473587754d55e4a3c4a2abc63c)
  build user:       root@bec9939eb862
  build date:       20180622-11:58:41
  go version:       go1.10.3

• Logs:

level=info ts=2018-06-23T06:35:35.484534947Z caller=main.go:174 msg="Starting Alertmanager" version="(version=0.15.0, branch=HEAD, revision=462c969d85cf1a473587754d55e4a3c4a2abc63c)"
level=info ts=2018-06-23T06:35:35.484603566Z caller=main.go:175 build_context="(go=go1.10.3, user=root@bec9939eb862, date=20180622-11:58:41)"
level=info ts=2018-06-23T06:35:35.494030639Z caller=cluster.go:155 component=cluster msg="setting advertise address explicitly" addr=<nil> port=9094
level=error ts=2018-06-23T06:35:35.494578525Z caller=main.go:201 msg="Unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: Failed to parse advertise address \"<nil>\""

[mxinden]

@skorpy2009 Thanks for reporting this issue.

Would you mind adding more details to "I added 192.168.2.2/24 to any of my interfaces"? It is surprising to me, that your system can not find its IP address or determine one.

[roidelapluie]

I guess OP's server just has public IP addresses

[simonpasquier]

@skorpy2009 can you share the IP addresses (eg ip addr output)? Looking at the code, I don't understand how it ended up with a <nil> address for the advertise address.

It is indeed undocumented that you need to specify the --cluster.advertise-address when your box only has public addresses.

[skorpy2009]

prometheus1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether aa:00:00:fb:e1:30 brd ff:ff:ff:ff:ff:ff
    inet 169.254.42.2/24 scope link ens13
       valid_lft forever preferred_lft forever
    inet 185.206.209.*/32 brd 185.206.209.* scope global ens13
       valid_lft forever preferred_lft forever
    inet6 2a06:8187:fb**::*128 scope global nodad
       valid_lft forever preferred_lft forever
    inet6 fe80::*/64 scope link
       valid_lft forever preferred_lft forever

[simonpasquier]

Thanks! This is helpful. I'll fix the logic of calculateAdvertiseAddress so it returns a meaningful error message (instead of "Failed to parse advertise address \"<nil>\""). That being said, you will still have to pass the --cluster.advertise-address argument to be able to start AlertManager: this is a "security" from the memberlist library to avoid exposing cluster ports over the Internet.