Nomad: [question] Is the "CPU" resource a maximum limit or a guaranteed minimum?

[davecb]

In most discussions of resource controls, the authors indicate if

• it is a maximum limit, as used in the old BSD4.X-derived ulimit, or
• if it is a minimum guarantee, as used in more modern zones or containers.

The documentation reads as if it's a maximum limit of X out of Y MHz, eg

NOMAD_CPU_LIMIT | CPU limit in MHz for the task

or

Task Resources
CPU        Memory          Disk     IOPS  Addresses
75/100 MHz  784 KiB/10 MiB  300 MiB  0     http: 10.1.1.196:5678

Is this actually a hard limit? Our site uses nomad to manage docker, which uses shares (cited by your note in https://www.nomadproject.io/docs/drivers/docker.html) and expects docker minimum guarantees.

If it really is a hard limit, we will have to shut down instances of one program to provide resources for another. If so, we would appreciate advice on how to avoid _that_ kind of manual reconfiguration.

[schmichael]

it is a minimum guarantee, as used in more modern zones or containers.

^ It's this one - a minimum which can burst to use idle capacity - for the Docker, exec, java, and rkt drivers.

raw_exec and Qemu drivers don't limit CPU usage and only use the information for scheduling, not enforcement.

[davecb]

Thanks!

[ashald]

The funny thing is that for rkt (if anybody cares about it) it's not true (at least right now) - it's configured with --cpu flag as seen in https://github.com/hashicorp/nomad/blob/v0.8.6/client/driver/rkt.go#L521-L522 and that in turn enforces a hard limit with CFS. What is even more funny is that the in the master branch which is the future v0.9 as of time of writing this behavior is changed to be a "guarantee" rather than a limit as seen is https://github.com/hashicorp/nomad/blob/master/drivers/rkt/driver.go#L548-L549
In other, as far as rkt is involved it's not the case right now but it's going to be the case in future release. At this point we probably can consider that to be a bug that is going to be fixed but people like us who relied on that behavior (hard limit) are going to be very sad - cc @nickethier and @schmichael who did the change. I wonder what are the chances of "not fixing this" or at least adding an option to enforce hard limit as it's with Docker driver? Just to not make existing users sad.

[schmichael]

@Ashald Thanks for pointing that out! #5197 reverts to the v0.8 behavior as the change was unintentional.

We'd like to add both guarantee and limit options to any driver that supports it, but we don't have the time to add that to rkt in the 0.9.0 release. Feel free to open a new ticket if you want to be notified when that support lands.

[ashald]

Thanks! We're switching to Docker which already has such a capability implemented in Nomad so I rather just wanted to point to [likely] unintentional change in behavior. :)