Nomad: Allocation eviction/migration/preempt for higher priority jobs

[jippi]

I would love to have Nomad be able to evict / migrate allocations away from a client if a job with higher priority need to be scheduled on the client.

A very concrete, and frequent issue we see at SeatGeek is our routing mesh (system job with priority 75) not being put on all instances since some clients do not have sufficient cpu/memory resources for it.

It happens some times when new instances come up, and a lot of blocked evaluations release at the same time and is put on the node, that the priority job 50 allocs get placed before priority 75.

Other times during an upgrade of the job, where allocs get replaced, that Nomad put a different allocation on the "spot" where the routing mesh alloc used to "fit", leaving us with a node that do not have routing mesh running, so can't handle traffic, nor talk out to other services.

Our preference would be that Nomad would see a priority 75 (system) job should be put on the client, see there isn't enough resources, so find a service / batch job and migrate an allocation to another node, and then puts the priority 75 job on the client once the migration has completed.

I think having this feature could be a system-job only thing initially if it makes it easier.

[luckymike]

This is the behavior I would expect for system jobs, so big +1.

[preetapan]

@jippi We've been talking about this internally (preemption) and is a road map item for Nomad 0.10

[skyrocknroll]

If we are able to move not only system jobs but other jobs for bin packing so that the less utilized nodes can be shut down. It is nice to have the feature 😄 . If it is complex to implement forget it.

[dukeland9]

Priority based preemption is very much needed.