Nomad: Allow health checks to use exposed container ports on driver-assigned ip.

[far-blue]

It's great to see the new address_mode service parameter that allows a service to be registered with a driver-managed IP instead of the host IP and it would be super-helpful if this capability was extended to the health checks.

My understanding is that it is actually consul that performs the health checks and my Consul agents have access to the same internal network Docker uses to assign IPs to containers. It would be great if the health checks could therefore use this internal IP - the same one used to register the service with Consul when the address_mode is set to 'driver'.

Similarly, my docker containers expose ports but don't map ports to the host - they are on an internal network with a proxy (Fabio) handling access from the outside world. But this means there is no port stanza for most of my containers. In the case, as explained above, where a driver-allocated IP is used for a health-check, it would be extremely useful to also specify an actual port number rather than a port label so health checks could be performed on exposed ports.

Maybe the best approach would be to have some new check types such as container_tcp and container_http that behave in the way described.

[rickardrosen]

This is exactly what I'm dealing with right now as we use MACVLAN and Ubuntu FAN networking, and the workaround we employ is to use a templated health script that performs the required checks.
Allowing a port instead of a port label to be specified for a check would be great.

There's probably a good reason for not allowing this, I probably just to ignorant to see it. =)

[schmichael]

Linked with #3599 as it describes another shortcoming of address_mode. Here's my plan for fixing both:

To properly support driver-determined-networks (SDNs, etc) service advertisements and health checks Nomad lacks proper support for 2 pieces of information:

1. port on service and check can be an int if address_mode=driver
2. address_mode on checks (defaults to host for backward compat, no auto option) to allow choosing the host or driver IP

This should allow advertising and checking SDN IPs and ports.

Let me know if you think this won't fix your issue! Looking to get this into 0.7.1 and release an RC ASAP.

[thetooth]

This solves my use case where nodes are connected via a weave overlay network and each service is given a unique ip address but ports are always 80 and 443. My only concern is under what conditions does nomad bind these ports? In the case of using weave the address space is private to docker(consul and nomad run in containers as well) so if nomad binds those ports, which address is used? because if it's the host it defeats the purpose of making that network private.

[schmichael]

@thetooth Nomad never binds task ports itself, it always leaves binding up to the driver and application. So in your case Docker would provide a Weave IP for your application. Nomad would advertise it along with the port you specify, and then it's up to you to bind to that same port in your application code.

[schmichael]

Still needs more testing and docs, but if anyone wants to test it so far I've uploaded some binaries: https://github.com/hashicorp/nomad/pull/3619#issuecomment-349419210

checks can now have address_mode=driver and if you're using the driver's address the port can be a numeric port.

The default behavior if you don't specify address_mode on a check is still the current behavior: use the host's IP and port. Not only is this backward compatible, but I think a significant number of users run Consul on the host without access to the SDN addresses.

[far-blue]

Sorry in the delay in replying. Yes, sounds like it will fix the issues I'm having first by allowing a check to be performed on the SDN address rather than the host address and second by allowing a port to be specified which hasn't been port mapped so I can use exposed port numbers. I'm sorry I'm not in a position to test things - I'm snowed under getting stuff ready for jan sales before we freeze system changes for xmas - but I'm looking forward to being able to ditch Registrator and go native in the new year :D