nomad should not require network bandwidth resource requirement for jobs

[kelseyhightower]

Nomad 0.4.0 does a better job of enforcing network bandwidth resource constraints which is great, but the major draw back is that most cloud instances only have 100 mbits of network bandwidth reported. This means that all jobs must declare a really low number for the network resource requirement or there will be no way to run more than a handful of jobs on a single node.

Ideally you will need to set mbits to 1 to get 100 jobs on a single node. I think this will be so common that Nomad should not require network bandwidth resource requirements for jobs. Network bandwidth is really hard to allocate upfront especially given the imbalance between CPU, memory, and network bandwidth available on a single machine.

[dadgar]

@kelseyhightower Operators can override the network bandwidth reported when the Client does not properly fingerprint it via network_speed https://www.nomadproject.io/docs/agent/config.html

So the operator can set it to what it actually should be. And ideally we can parse the information from metadata provided by the cloud provider do so for AWS and is on the roadmap for other providers.

[kelseyhightower]

Good point. Maybe a call out in the docs is all we need, but I think this one is going to bite people. I'm going to study the configuration options and make adjustments to my deployment.

[dadgar]

Yeah agree on the docs will mark it

[ketzacoatl]

In practice, I find setting resource limits for CPU/memory relatively easy to determine and set, though I find network resource limits more difficult and not always desirable. At times I want to set the network limits, but most of the time I would rather "nomad figure it out". Nearly all of my network limits are very arbitrary, and I keep changing them as I add new jobs (they feel as though they are arbitrary and just getting in my way more than not).

I would greatly prefer network resource limits as optional, or, if we cannot make mbits optional, it would also suffice to set a priority or other value, where nomad does some math to figure out mbits for the jobs on the cluster (that don't have an explicit limit set).

[spicykaiju]

This bit me today big time, happy to hear that there will be some more focus on the docs. Huzzah!

[lpicanco]

I had this problema today. Following the discussion.

[anxiousmodernman]

I got bit by this one. Here's my scenario.

1. running Nomad under systemd on my Arch Linux laptop for dev/testing
2. I don't want to run with -dev. I'd like to persist state, and have access to the real configuration values. No problem: an example combined (client + server), single-node agent configuration can be obtained here. Plus, Nomad's pretty lightweight.
3. I had successfully run a single redis container, and I was about to scale up to count = 3, but when I ran nomad plan example.nomad, I got a strange warning Dimension 'network: bandwidth exceeded":

I googled a bit. Here are a couple of things that made this warning go away:

• setting mbits = 1 in the example job file
• removing the network_speed config param in my combined server/client config

What probably would have helped is if network_speed were documented or called out more prominently here, or perhaps removed/commented out in that example. Alternatively, perhaps a lower mbits value could be printed out when running nomad init. It was all the more confusing because running 3 redis containers isn't really the most resource intensive task.

In general, I think the single-node setup could be given more emphasis. The -dev mode doesn't _really_ let me explore configs.

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[stale[bot]]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem :+1: