Consul: Allow binding multiple client addresses

[mfischer-zd]

It should be possible to bind to multiple interface addresses (or interface names) for client communication.

A typical use case for this is Docker hosts, where we want to bind both to 127.0.0.1 (interface lo) and 169.254.1.1 (link-local address assigned to dummy0 interface). Clients on the host can connect to 127.0.0.1 as usual, while Docker containers can communicate with the link-local address (since 127.0.0.1 is namespaced, and we don't want to share the host network).

As a workaround, we're using iptables to restrict remote client communication with Consul, but being able to better control the bindings would be a simpler solution.

[sean-]

@mfischer-zd , while this doesn't address the binding to multiple IP addresses portion of your issue, there is code in master now that will let you bind to an IP on a named interface:

-bind='{{ GetInterfaceIP "eth0" }}'

There is now a configurable template language for examples and docs) behind this that you can use to create a customizable heuristic that should allow you to get whatever it is that you need from your environment when using an immutable image (see hashicorp/go-sockaddr/template and cmd/sockaddr.

[mfischer-zd]

That's great news, @sean- . Let me know how I can help finish the work.

[magiconair]

I'm going to pick this one up. Expect this to become a multi-step change.

[shilov]

Any chance we can also get support for binding the http server to both IP and unix socket?

We chose unix sockets to take advantage of the reduced overhead, but it came at the expense of not being able to access the http API as easily.

[rhyas]

A first pass I did at this a while back was closed in favor of #3037. Just curious, how does this relate to that one? We've been using a patched build with the PR I did for a few things that needed those specific binds, so I'm interested in seeing either #3037, or this one, get main-lined. Also, as an aside to @shilov the patch I did, supported multiple interfaces in general, they could be sockets or addresses. I would assume whatever new thing should do the same thing, it was just a binding for the listener.

[magiconair]

@shilov yes, that will be possible. I'll add that on top of #3037

@rhyas I took the opportunity to refactor the agent startup/shutdown code to allow us to have faster and more reliable tests and cleanup that code in general. The HTTP and DNS servers for example were started and managed independent of the agent itself. The HTTP startup code now already accepts multiple bind addresses and types just the command line and configuration code does not allow it yet. I need to do this also for the DNS server to be consistent. I should be able to finish this next week.

[magiconair]

The server code in #3037 can now accept multiple listening addresses and starts/stops them correctly. However, getting the configuration propagated through the code requires a bit more work since the code uses the user provided configuration directly and does the transformation on the fly. The assumption that there is a single address is in multiple places and patching it there doesn't make much sense.

I'd like to separate the user provided configuration from what the server actually uses properly so that we can have a translation once at the beginning. I had planned to do this in a separate PR but if I want to support this feature I think I have to add this now. This is going to become one giant PR ...

[lievendp]

It's not very clear to me but am I right to understand that "bind_addr": "0.0.0.0" is no longer functional in consul 0.8.4 as it was in 0.7.2 and 0.6.4 ?

In the 7 & 6 versions that I've used I didn't have to specify the bind_addr (def to 0.0.0.0) and the consul was functional in the docker container with multiple ethernet interfaces (overlay, bridge, local)
However in the 8.4 version I'm trying to use now this fails miserably.

Is it expected to have a fix for this in 0.8.x versions? Or should I stick to 0.7.x versions for my case. (I'd rather not fix the ip I get in the overlay network)

ftr: I've tried setting "eth0" or "10.0.." or "10.0.0.0/16" for bind_addr as well to no avail, service just fails which corresponds to how I understand things here?

[magiconair]

is that in 0.8.4 or also earlier?

[lievendp]

I've only used 0.8.4 just now. which I got like this:
wget https://releases.hashicorp.com/consul/0.8.4/consul_0.8.4_linux_amd64.zip

my goal was to have the most recent available version 0.8.4 instead of the olders I've been lagging around.

[slackpad]

Hi @lievendp can you describe a little more about what you are seeing when you say it fails? There have been some changes since 0.7.x, but setting 0.0.0.0 should work the same. What are you seeing?

[lievendp]

Sorry I'll have to reproduce the setup to give the exact error but basically:

• the consul 0.8.4 is inside a docker container based on alpine 3.6
• the container starts consul but exits because consul fails to start
• the "bind_addr": "0.0.0.0" or leave out or any of above described values
  The error just tells me that there is more than 1 ip and I should specify one in the config and just exits.

will add exact error when I reproduce it.

[slackpad]

Hi @lievendp that "more than 1 IP" check has been in there since Consul 0.6, so is it possible your network configuration changed and maybe another interface was added inside your containers? In any case, you can now use go-sockaddr templates to bind by things like interface name, which should make it possible to keep your container from having to know its IP. You can see an example here - https://github.com/hashicorp/consul/issues/1110#issuecomment-264465379.

[lievendp]

maybe I'm missing something and I will try the go-sockaddr as well but to clarify on the situation:
I have a docker container image for consul 0.7.2

this is the config:

{
  "datacenter": "dc",
  "data_dir": "/var/consul",
  "log_level": "INFO",
  "ui": true,
  "server": true,
  "bootstrap_expect": 1,

  "recursors" : [
    "8.8.8.8"
  ],
  "retry_join" : [
    "kvm-centos7-docker-a_consul",
    "kvm-centos7-docker-b_consul"
  ],
  "telemetry": {
    "statsd_address": "<IP>:8125",
    "disable_hostname": false
  },
  "addresses": {
    "https": "0.0.0.0"
  },
  "ports": {
    "https": 8443
  },
  "key_file": "/consul/config/consul.key",
  "cert_file": "/consul/config/consul.crt",
  "ca_file": "/consul/config/consul.ca",
  "client_addr": "0.0.0.0"
}

This runs fine.

Then I have a container for consul 0.8.4 which I run with the same config then I get the following error:

==> Starting Consul agent...
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Error starting agent: Failed to get advertise address: Multiple private IPs found. Please configure one.

The bootstrap afaik is to start a 1-server node consul "cluster" (it's just a test) and I add one more agent. So the message is to be expected.

but the "multiple" ip's found is not expected as it works for 0.7.2.

furthermore, I've started the container from the image just fine in a test with docker run. But in that case, I didn't give any config except:
docker run -d -p 8400:8400 -p 8600:8600/udp -p 8500:8500 -t -h consul01 local/consul:0.8.4 agent --data-dir /data -server -bootstrap

EDIT:
but ... there is however a but I didn't see, the environment variable set with

CONSUL_BIND_INTERFACE=eth0

works for the 0.7.2 container but not for the 0.8.4 container
At least it explains how the 0.7.2 version was able to determine the required ip
Inside the running 0.7.2 container I have an interface eth0 with 10.0.0.x for the overlay network and another interface eth1 for the natting outside connectivity.

EDIT.2: got some troubles to get the sockaddr to work but putting the "dhcp ip" I get from the overlay into the bind_addr makes it start.
so:

• environment var no longer accepted
• some problem with sockaddr which I'll have to fix here.

EDIT.3: works fine now, don't rely on env var as before and use the sockaddr wrapper ( which appears to be already in consul - I included it manually at first :-) ) Oh and escape those double-quotes and it works just fine.
my sample case:
"bind_addr": "{{ GetAllInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}",

[magiconair]

Good news: #3480 finally implements this. You can either list a number of ip addresses, specify a go-sockaddr template or both. To support both unix sockets and ip addresses you need to use the address overrides, e.g. addresses.http = '127.0.0.1 unix:///var/sock/consul {{GetPrivateIP}}'