Nomad: start task balancing deploy over the cluster

[zsluedem]

I have read about the docs.

It says one task of one specific group name would always start a job in the specific node.

What if now i want to start multiple similar jobs across the cluster (let's say 3 nodes).

Is there a way for nomad to do sth like that. I want to job balancing across the 3 nodes.

[angrycub]

There are two possible ways to distribute your job on multiple nodes in your Nomad cluster: increasing group count and system jobs.

Increasing the count parameter in the group stanza of your job, more instances of your job will be run in your cluster. There is an anti-affinity value included in the scheduling calculation to instances of the same job which makes them unlikely to be scheduled on the same node, assuming sufficient node count available.

If you need the job to be running on all of the nodes in your cluster, then you can use a system service to schedule a job on all running nodes.

From the sounds of your question, you just need to increase the count on your job.

[skyrocknroll]

There is an anti-affinity value included in the scheduling calculation to instances of the same job which makes them unlikely to be scheduled on the same node, assuming sufficient node count available. Is this anywhere mentioned in the documentation ? I was looking for this for very long time.

[angrycub]

It is mentioned in the specific on the Nomad website and mentioned more generally as "anti-affinity rules" in the Nomad scheduler documentation itself. I think that this is an opportunity to improve the documentation and I will work on submitting a docs PR to make this more clear in the upcoming week.

[zsluedem]

@angrycub Thanks a lot!!!!

[zsluedem]

@angrycub I can't really understand the concept of anti-affinity value.
I try to increase the count number you said ,but it seems not the output i want.

Let's say we have 3 nodes.

I have 4 similar task with different args. Let's say it's a ping task going to ping different host.
When i deploy the first three tasks. The 3 tasks can be deployed on the 3 nodes pinging different host. But the fourth task keep queued unless one of the 3 tasks complete(exit).

What I want to do is the FOURTH TASK can still run with the 3 tasks running and if i want to deploy the fifth task , It can balancing run across the 3 nodes.

Can I do it this way?

[skyrocknroll]

@zsluedem Do you have enough resources to start the 4 resources at the same time ex: cpu , memory ?

[zsluedem]

@skyrocknroll i have enough resources. Are you sure is the resource problem?

[angrycub]

@zsluedem I have prepared a walkthrough that discusses job anti-affinity and resource considerations as part of the scheduling process. In order to keep the comment section smaller, I made it a gist.

https://gist.github.com/angrycub/86dbb92422cb27684678b8066c1ef4f2

If resource exhaustion is impacting the job being scheduled, it will be noted in the output of nomad run and nomad status for your job.

[dadgar]

@zsluedem Can you post your job? Nomad will automatically try to spread the task groups onto different hosts. You can force nomad to do this by adding a distinct_host constraint as well: https://www.nomadproject.io/docs/job-specification/constraint.html#quot-distinct_hosts-quot-

Further you can use nomad plan to see why it isn't placing the 4th instances. There must be some resource that is constrained otherwise it would place them all.

[zsluedem]

@dadgar @angrycub thanks~! I will look into this later . I am going to close this now~!