Go-micro: When deregistering a service, it does not always work

[yarbelk]

Stack:
Using Nomad, Consul, Docker, dnsmasq

This two out of 3 times, when changing my job definition in Nomad, or when I deploy a new Job, or change my job count to 0 in nomad, the service is not deregesterd from Consul

my nomad setup has:

env {
        MICRO_REGISTRY_ADDRESS = "consul.service.consul:8500"
        MICRO_SERVER_ADDRESS   = "0.0.0.0:8080"
        MICRO_SERVER_ADVERTISE = "${NOMAD_IP_rpc}:${NOMAD_HOST_PORT_rpc}"
        MICRO_SERVER_NAME      = "myService"

}

The same happens when I use the following to register with the local client:

env {
        MICRO_REGISTRY_ADDRESS = "${NOMAD_IP_rpc}:8500"
        MICRO_SERVER_ADDRESS   = "0.0.0.0:8080"
        MICRO_SERVER_ADVERTISE = "${NOMAD_IP_rpc}:${NOMAD_HOST_PORT_rpc}"
        MICRO_SERVER_NAME      = "myService"
}

in the consul server I am still registered on one of the 'servers', 10.1.5.10, instead of the expected ip of 10.1.2.10 of the worker.

Which lines up nicely with the fact I have 3 Consul servers, and calls to deregister consul servers only work if you call the server it is registered with

For example - if the service is registered with consul server 10.1.5.10, but the deregister command is sent by dns to 10.1.5.20, the service will not be deregisterd. which breaks things.

Is there a way to set a health check to automatically deregister?

Is there a way to ensure that the service is deregisterd with the correct consul server without hard coding the same server?

[asim]

With a cluster deployment of consul you may likely need to run a local consul agent client which connects to the main cluster. This handles membership, registration, etc in a more cleaner manner. The issue you're highlighting is with consul itself in that you must always maintain registration/communication through one agent. Using DNS in the way you're doing will result in transient failures. The go-micro consul registry does not attempt to interpret the address, this is handed directly to the consul go client library.

To setup the healthcheck which automatically deregisters a service use the following RegisterTTL and RegisterInterval options when creating your service.

    service := micro.NewService(
        micro.Name("go.micro.srv.greeter"),
        micro.RegisterTTL(time.Second*30),
        micro.RegisterInterval(time.Second*10),
    )

https://github.com/micro/examples/blob/master/greeter/srv/main.go#L24L25

[yarbelk]

Fantastic.