Consul: advertise_addrs in the config doesn't work

[preetapan]

When investigating https://github.com/hashicorp/consul/issues/3436, I found that in 0.9.3 we expect the advertise_addrs config option to include host and port, and it also incorrectly wants the RPC port to be an address. If the config sets all this, it uses the rpc port from inside advertise_addrs, but ignores the serf_lan/serf_wan ports.

In the 1.0 beta branch, it throws an error when any sub elements in advertise_addrs contain both an IP and port, and only accepts IPs. So people using advertise_addrs in 0.9.3's configs will break during an upgrade (albeit a partially working thing)

Discussed with @slackpad and it definitely looks like advertise_addrs never worked properly. We should remove support for it (and make sure one can override both the advertise address and the serf lan/wan ports via other config options), and update the documentation to remove references to it.

Here's the config file I used for testing

 {
"bind_addr": "127.0.0.1",
"advertise_addr":"127.0.0.1",
"node_id":"ac5ec7af-5238-4c18-988a-0385d0a0f477",
"data_dir":"/tmp/consultest",
"ports":        {
                "https":-1,
                "dns":-1,
                "serf_lan":9301 //if this line is removed, it uses the default port and not 9302 though its configured according to what the documentation says
        },
"advertise_addrs": {"serf_lan": "127.0.0.1:9302", "serf_wan": "127.0.0.1:9302", "rpc": "127.0.0.1:9303"},
"datacenter":"mydc",
"ui":true,
"server":true
}

[magiconair]

I think this is the wrong approach altogether since the whole port/address configuration is way too complex and you have to understand the heuristics on how addresses are derived anyway. If we are going to properly fix this then we should do the following:

• deprecate bind, client, ports, advertise_addr, advertise_addr_wan, advertise_addrs altogether in one fell swoop
• provide a single addrs structure with the following default below
• I'm ok with making the advertise address the first bind address unless something else is configured.

To retain the ability to have multiple addresses and use go-sockaddr I'd stick with the space separated list. The only issue there is that you can't have unix socket names which contain spaces but I think that's a minor issue. If you think that this is an issue we can use either an encoding scheme or switch to arrays (e.g. dns { bind ["0.0.0.0:8600", "unix:///var/run/consul with a space.sock"] })

The new -hcl flag also immediately supports the new flags, e.g. consul agent -server -hcl 'addrs { dns { bind "127.0.0.1:53" } }'

addrs {
  dns {
    bind "0.0.0.0:8600"
  }
  http {
    bind "0.0.0.0:8500"
  }
  https {
    bind "" # disabled
  }
  serf_lan {
    bind "{{GetPrivateIP}}:8301"
    # advertise "{{GetPrivateIP}}:8301" # default is first bind addr
  }
  serf_wan {
    bind "{{GetPrivateIP}}:8302"
    # advertise "{{GetPrivateIP}}:8302" # default is first bind addr
  }
  rpc {
    bind "{{GetPrivateIP}}:8300"
    # advertise "{{GetPrivateIP}}:8300" # default is first bind addr
  }
}

You could then configure your setup as follows:

addrs {
  http {
    bind "{{GetPrivateIP}}:8500"
  }
  serf_lan {
    bind "{{GetPrivateIP}}:8301"
    advertise "127.0.0.1:8301"
  }
  serf_wan {
    bind "{{GetPrivateIP}}:8302"
    advertise "127.0.0.1:8302"
  }
}

What I need to check is whether this would allow you to configure just a different value for the http bind address for example, e.g.

addrs {
  http {
    bind "1.2.3.4:8999" # overwrite only the http bind address
  }
}

[magiconair]

I've pushed #3517 which drops advertise_addrs as described by @preetapan and @slackpad so that we can resolve this for 1.0.

[azagirov]

@magiconair , now I can change advertise port by any means?
advertise_addrs now not used,

addrs {
  http {
    bind "{{GetPrivateIP}}:8500"
  }
  serf_lan {
    bind "{{GetPrivateIP}}:8301"
    advertise "127.0.0.1:8341"
  }
  serf_wan {
    bind "{{GetPrivateIP}}:8302"
    advertise "127.0.0.1:8402"
  }
}

this config not working too.

/opt/consul/consul agent -server -data-dir=/opt/consul -bootstrap -config-dir=/opt/consul/ -ui -syslog
==> Error parsing /opt/consul/cluster.hcl: key 'bind "{{GetPrivateIP}}:8500"' expected start of object ('{') or assignment ('=')

I try start consul with this command? and receive error too:

consul agent -server -hcl 'addrs { dns { bind "127.0.0.1:53" } }'
==> Error parsing flags-0.hcl: key 'bind "127.0.0.1:53"' expected start of object ('{') or assignment ('=')

[slackpad]

Hi @azagirov you should still be able to set the port using https://www.consul.io/docs/agent/options.html#ports

[utdrmac]

@magiconair Please update documentation https://www.consul.io/docs/agent/options.html to indicate which parameters are gone.

Dec 13 20:26:46 orc consul[20572]: ==> Error parsing /etc/consul.d/consul.hcl: 1 error(s) occurred:
Dec 13 20:26:46 orc consul[20572]: * invalid config key advertise