Consul: Failed to start TCP listener with a VM in a consul cluster

[domrod]

Hi,

I try to use consul in a 2 nodes configuration (2 physical servers) plus 1 VM hosted on one of these hosts.

The VM has a local IP (say 10.10.20.30) and a public nated IP (say 12.34.56.78).

The hosts have their public IP (say 12.34.10.10 and 12.34.10.20).

I managed to set the hosts as a consul cluster (one in bootstrap mode and the other in server mode). This works.

I tried to add the VM in this cluster, using its public IP in consul config, but it does not work :

"Error starting agent: Failed to start Consul client: Failed to start lan serf: Failed to start TCP listener. Err: listen tcp 12.34.56.78:8301: bind: cannot assign requested address"

I have declared the IP/hostname in /etc/hosts, in case of. I guess that there is some DNS problem in my config.

May be the VM only uses LAN net, as eth0 is configured with local IP (10.10.20.30)?

Could you give me some insights on how to solve my issue?

Regards,
Dominique

[highlyunavailable]

You actually figured the issue out already:

"May be the VM only uses LAN net, as eth0 is configured with local IP (10.10.20.30)?"

That's exactly the case, Consul is binding to an IP on one of the server's NICs. If the server does not actually have a NIC that has the public IP (which is the case since you're going through a NAT) the you cannot "bind" to the public IP. What you actually want to do is use the advertise_addr config option (or -advertise command line option) to specify the public IP as what is advertised, while letting Consul bind to the private IP and letting the NAT take care of the translation.

[domrod]

Great! You just saved my day.

I have put the following in the config.json file:

"bind_addr": "10.10.20.30",
"advertise_addr": "12.34.56.78",

and it works like a charm!

Thanks a lot :)

Regards,
Dominique

[armon]

Sounds like this is working now! Closing, but please reopen if an issue.

[RongxinZhang]

@armon love your product, but your getting started for consul sucks.

[slackpad]

Hi @RongxinZhang - sorry that you didn't find that guide helpful. Is there anything specific that didn't work well for you?

[bluefeet]

@RongxinZhang, why troll man? If you got something to criticize at least be helpful.

Anyways, I'd like to answer the question raised about the guide. I've found that all the information I need is in there. Sometimes its hard to find. A prime example is IP addresses and ports, which is partly why I ended up here. It would be nice to have a concise document talking about all the networking pieces. It took me a bit of trial and error to figure out how exactly I needed to setup my AWS security groups and VPCs so that everything could communicate properly. And, my fingers are still crossed I got it right.

I think you guys have documented everything, its just there is, I believe, lack of high-level documentation in some places which pull it all together. You've got all the vertical documents "this is how this specific feature works" and seem to be missing the horizontal documents "this is how you get this workflow running, which uses all these features".

Oh, one other documentation point that I've hit and haven't overcome yet. The documentation on how to setup ACLs is very lacking. The documentation about how ACLs changed from one version to the next is great, but there is no guide to set them up in the first place. All I have are samples of the contents of ACL files. But... where do I put them, etc? Maybe I missed a document somewhere. If so, maybe there needs to be more cross-referencing within the documents.

Hope that helps!

As @RongxinZhang said, love you product!