Nomad: Scheduler not distributing allocations across idle agent nodes

[latchmihay]

Nomad bin packing is great but tends to start all allocations on a few agent nodes, and leaves the rest idle.

Nomad version

Nomad v0.8.4 (dbee1d7d051619e90a809c23cf7e55750900742a)

Operating system and Environment details

Linux  3.10.0-862.9.1.el7.x86_64 #1 SMP Wed Jun 27 04:30:39 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Red Hat Enterprise Linux Server release 7.5 (Maipo)

Issue

Nomad bin packing is too good leaving available cluster agent nodes idle.

# nomad node status -allocs
ID        DC     Name       Class   Drain  Eligibility  Status  Running Allocs
4e26c41f  DC1  server238  <none>  false  eligible     ready   1
7032ecca  DC1  server239  <none>  false  eligible     ready   1
c25a5922  DC1  server240  <none>  false  eligible     ready   34
9719f743  DC1  server236  <none>  false  eligible     ready   7
c5038917  DC1  server235  <none>  false  eligible     ready   30
c836fafd  DC1  server234  <none>  false  eligible     ready   1
1a3a0fbf  DC1  server233  <none>  false  eligible     ready   1
38242d1d  DC1  server232  <none>  false  eligible     ready   1

The non utilized servers do have one allocation running because its running a system job

[nickethier]

Hello!

This is the intended behavior. Nomad will optimize scheduling for bin packing. There are definitely many valid use cases where this is problematic. Look out for our next major release (0.9) as it introduces an new feature to tell the scheduler to spread instead of bin pack.

Closing this issue, but feel free to reopen if needed.

Thanks for using Nomad!