Nomad: Nomad should not set Consul ServiceAddress / setting ServiceAddress should be configurable

Created on 29 Oct 2018 · 2Comments · Source: hashicorp/nomad

Nomad version

Nomad v0.8.4 (dbee1d7d051619e90a809c23cf7e55750900742a)

Operating system and Environment details

Ubuntu 16.04

Consul v0.7.5
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Issue

When Nomad registers a service in Consul, it always sets the ServiceAddress for that service. This means that the address translation that works with the node address for WAN/LAN addresses will not work for these registered services. There are workarounds when using Consul Template (just use the .NodeAddress), but when the services are accessed through DNS, Consul will only provide the ServiceAddress (which is the LAN address), even when the service is being queried from another datacenter.

We have consul clusters in multiple regions, and nodes have different WAN and LAN addresses, and sometimes need to be addressed cross-region through DNS. It would be ideal if we could ask Nomad to not set the service address (perhaps in the job file or in the server or client config), so that the node address will be used and translated between WAN/LAN properly.

Reproduction steps

Create two Consul clusters in separate regions (region1 and region2), create a Nomad cluster in one of those regions (region1).

If you query a service in region1 from region2, via DNS, you will get the LAN IP of that service, rather than the translated WAN node address.

Possibly Related Issues

3629

2770

themconsul themdiscovery typenhancement

Source

mimato

👍8

Most helpful comment

We also need to register Consul services using FQDN by Nomad because we want to use HTTPS with TLS verify for discovered services.
Also a service healthcheck address should be also configurable and allow us to set FQDN to do an internal consul http check using https://hostname address w/o any InsecureVerify True because it's insecure :)

Currently, we patched Nomad like this https://github.com/Quiq/nomad/commit/a081ee28ad22dc1acc0a8c3fde7b063b7809c4aa

roman-vynar on 19 Aug 2020

👍3

All 2 comments

Agreed. This is essentially the same as the "running Nomad behind NAT" scenario. Issue https://github.com/hashicorp/nomad/issues/2770 does seem to be the same. I'd like to be able to run a "microservice behind NAT" scenario. This would enable Fabio, for instance, to correctly route to the public IP of a service when the host of that service is behind NAT.