Nomad: [feature] readiness health checks

[ygersie]

Some situations require a one time (deployment) health check to pass. K8s has a separation between liveness and readiness health checks. Ideally Nomad would support the same thing. In some cases you might want to warm-up caches before inserting the service into the pool, or you want to wait for a new MySQL slave to be caught up with the Master. Is this on the roadmap somewhere?

I looked around trying to find a similar open issue but haven't found anything. If this is a duplicate please close ;)

[dadgar]

@dropje86 Is essentially what you are asking for is to be able to ignore certain health checks when doing a deployment?

[ygersie]

Hey Alex, yes I think you’d achieve the same thing. Basically a check which is specifically for deployments. If possible keeping the service registration of the others (for discovery purposes during and after deployment). So maybe using maintenance mode or smth?

[jippi]

I could use this as well - my case is like this

We run a lot of Redis clusters in Nomad, using Resec - some of these clusters are rather big (12GB+ data). We run 1 master + $n replicas

During a rolling replacement, we need to health-check the following (using redis-health)

1) Redis is running
2) Redis is not loading data from disk
3) Redis is done getting initial data dump from the master
4) Redis slave<->master link is up

once these 4 steps are complete / healthy, we can deem the allocation healthy - after that point, Resec handle the normal (differently named) consul service management.

today we do something like below - the thing is, we don't care about these service checks after the allocation has been marked healthy - at that point, its up to Resec to maintain the consul catalog

      service {
        name = "${NOMAD_JOB_NAME}-health"
        port = "redis"

        check {
          name     = "tcp-health"
          type     = "tcp"
          port     = "redis"
          interval = "10s"
          timeout  = "2s"
        }

        check {
          type     = "script"
          name     = "redis-health"
          command  = "/usr/local/bin/redis-health"
          interval = "10s"
          timeout  = "5s"
        }
      }

for us, it would be nice to mark this service / checks as "only run these until the alloc is healthy, then deregister and/or stop checking them" - as post the healthy alloc, they aren't used for anything anyway, and just wasting cycles in the cluster

[ygersie]

@jippi yeah so I think to cover most use cases you’d have smth like:

check {
    name = “whatever”
    type = “http”
    path = “/health/deploy”
    deploy_only = true
}

Then hold off on other check registrations until the deploy_only check passes. That way the deployed service can’t end up in your active pool. This would cover a lot of scenarios where I might need to wait for a new MySQL slave to recover from a snapshot, or warm up the InnoDB pool in the entry point or load data from disk.. Agree?

[Dirrk]

I just proposed this last week with no feedback but I feel like it follows the pattern of the existing configs https://github.com/hashicorp/nomad/issues/4496

[lorenzo]

I think it is also important to consider readiness checks not tied to the concept of a service. For instance parameterised jobs and batch jobs should not do any work before the checks have passed, but currently it is not possible without coding the checks in the job itself.

[josegonzalez]

To expand on @lorenzo's point, here is a real use-case we have at work:

We run a few different services as sidecar tasks - telegraf (metric collection), filebeat (log shipping), maxscale sql proxy (for connection pooling), and soon envoy (for request routing). What we'd like to do is ensure that all of those things are running and healthy before having any leader tasks start. This way we can ensure that we'll be properly proxying SQL requests, or ensuring we're not going to drop any metrics.

We can currently configure min_healthy_time at a job _or_ group level, but not for a given task, so that wouldn't work here. It also doesn't guarantee that a sidecar is necessarily healthy. Ideally each one could have a readiness-probe configured - separate from a consul service registration - that would happen only on task start, similar to how kubernetes handles it (I'm not sure of what the semantics should be when a task restarts, but I'll leave that up for debate).

As an aside, we'd also love to do this for non-sidecar processes when there _isn't_ a service running. As @lorenzo said, you could imagine running a periodic or parameterized job that has a binary invoked in it's context to ensure it's running properly, or so that we can restart it as necessary via the regular nomad setup. We've had issues in the past where - due to a bug in our deploy process - database credentials are incorrectly provided but the tasks don't fail, resulting in a high number of errors.

[shantanugadgil]

Hi, I too feel the need for something like this (liveness + readiness) often.
The use case would be:
don't start executing healthcheck2 until healthchech1 is passing.

Could Consul's "alias" type healthcheck help here?
https://www.consul.io/docs/agent/checks.html#alias

BTW, Nomad's documentation doesn't mention that the alias type is supported:
https://www.nomadproject.io/docs/job-specification/service.html#type

[HINT-SJ]

Hi, I and my team also have a need for seperating readiness and liveness probes.
Liveness: the service is up and running (when service can't signal it's alive nomad should restart it)
Readiness: The service is marked as healthy and Consul (or others) can mark the service to be able to receive traffic.

A good example that always comes up is:
When a service detects that a database, messaging service, etc. is currently not available, the service should signal that it's not ready to receive any traffic.
Currently nomad restarts the service due to a failed check but this just wastes resources since starting the service up again takes a lot of CPU just to be restarted soon again when the database is still not available.
When nomad/consul would just flag the service as unavailable it could remain alive and signal readiness again once the database comes back up again, without having to waste CPU resources to start up again.

[cgthayer]

+1 Need this too.