Nomad: Emits per-task allocated resources telemetry

[multani]

We would like to measure how much resources a task is using compared to how much it reserved in the first place, with 2 specific goals in mind:

• detect tasks which reserved a lot of resources but are actually using not that much. We could then reduce the resources reservation from these tasks.
• detect tasks which are getting closer to use all the reserved resources. We could then tune up the task and/or the reservations to prevent tasks from starving in the future.

AFAIK, there's no allocated resources metrics which are emitted per-tasks. https://github.com/hashicorp/nomad/pull/2327 provides these allocated metrics at the client level. https://github.com/hashicorp/nomad/pull/2330 was supposed to provide these metrics at the tasks level but I guess @burdandrei deleted his branch :) and the code is not available anymore.

[burdandrei]

@multaniI just moved it to another branch.
In the meantime, we're pretty happy with telemetry per allocation.
FYI https://github.com/jippi/hashi-ui can show per task usage, but only in real time

[multani]

@burdandrei Thanks, I just saw the branch now!

As for jippi/hashi-ui, I guess it probably does it by polling Nomad API directly (it's definitely available there). Would be nice to provide them as part of the telemetry process though :)

[burdandrei]

@multani API calls it is.
I could bring this PR alive, but I still smaller PRs opened for months =( https://github.com/hashicorp/nomad/pull/3882 for example

[mlehner616]

Just a bump on this. Because nomad is so strict on memory reservations, it is really crucial to be able to measure a given allocation's usage against its own reservation over time. Measuring this in real time via API is _almost_ pointless because it doesn't account for so many things that can happen while you're not observing the task (nightly process or high load request for example).

[margueritepd]

@burdandrei please open up a PR with your branch! I see that #3882 has been merged since your comment :)

[burdandrei]

@margueritepd hardly believe that so outdated branch will survive the rebase, but I'll see what's left from it

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[mlehner616]

Bump