Cadvisor: Memory usage definition

[mboussaa]

Hi,
How can you define the memory usage metric provided by cadvisor? is it dynamic stack size, heap requirement, cache misses, code size?
as well for cpu cumulative usage, in which measure unit it is calculated? How can you define it?
Is there any documentation on that ?
@jimmidyson Do you have any idea?
Thank you

[jimmidyson]

See https://github.com/google/cadvisor/blob/master/info/v1/container.go#L304-L319 & https://github.com/google/cadvisor/blob/master/info/v1/container.go#L258-L275

[vishh]

Memory usage is total usage including cold pages which the kernel can reclaim safely under pressure.
Working Set is the bytes of memory that the kernel deems to be necessary for continuing to run the processes in a container. Kernel cannot tolerate overcommitment of the sum total of working set.

[mboussaa]

If I well understood:
memory_usage= RAM (hot+cold)+SWAP
working_set= RAM (hot)
That's right?

I am not sure if SWAP memory is included in "memory_usage"
Can you confirm that?

[vishh]

That mostly correct:
IIUC, cAdvisor is not supporting
memory+swap accounting as of now. You should file an issue separately.

As of now,
memory_usage = RAM usage include pages that have not been accessed in a
long time.
working set = memory usage - inactive memory.

On Mon, Oct 26, 2015 at 3:38 AM, mboussaa [email protected] wrote:

If I well understood:
memory_usage= Virtual memory= RAM+SWAP
working_set= RAM
That's right?

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/913#issuecomment-151096072.

[solomatnikov]

I am wondering if current approach to measuring memory footprint in cAdvisor makes sense. Total memory usage reported by Linux or cgroups is generally misleading because it includes cache, e.g.:

http://www.linuxatemyram.com/

So called "hot" or "active" memory metric is also usually useless because it is a rough approximation of LRU used by the kernel to select pages for swapping. However, it the system starts swapping performance decreases by orders of magnitude, so usually people try hard to avoid swapping.

This is not a new problem and recently a new more meaningful metric was added to the kernel:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773

So, the suggestion is to use MemTotal - MemAvailable as memory utilization metric for the host/top level.

Unfortunately, there is no similar metric for cgroups but RSS makes the most sense.

[vishh]

@solomatnikov: I believe there is an implicit assumption that swap is disabled in the current logic. Under memory pressure, the kernel will use hot memory to identify a victim.

[solomatnikov]

@vishh Not sure I understand your comment: are you agreeing with my point that hot/active memory is not useful metric because in general swapping should not happen under normal conditions?

I am trying to figure out what are the "actionable" metrics for containers, particularly memory related.

[mboussaa]

@solomatnikov You can enable/disable swapping for your container and check the memory consumption using cAdvisor

[dashpole]

This appears to be resolved. cAdvisor doesn't currently support running with swap enabled.