Nomad: Docker containers use swap in 0.9.4

[pznamensky]

Nomad version

Nomad v0.9.4 (a81aa846a45fb8248551b12616287cb57c418cd6)

Operating system and Environment details

CentOS Linux release 7.6.1810 (Core)
Docker version 18.03.1-ce, build 9ee9f40

Issue

After updating nomad from 0.8.4 to 0.9.4 docker containers started to use swap.
Despite this:
https://github.com/hashicorp/nomad/blob/v0.9.4/drivers/docker/driver.go#L730
docker inspect shows that containers use unlimited swap:

docker inspect 2bc3d0fd13b3 | grep Memory
            "Memory": 268435456,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": -1,
            "MemorySwappiness": null,

Reproduction steps

• Create a job that uses docker driver.
• Check MemorySwap parameter of the container

[langmartin]

Thanks for this! We'll need to do some more investigation and document this behavior or change it back. As a workaround, docker inherits MemorySwappiness from the host system's settings: https://docs.docker.com/config/containers/resource_constraints/#--memory-swappiness-details

/proc/sys/vm/swappiness is the global value on linux, and the cgroup parameter /sys/fs/cgroup/memory/*/memory.swappiness can restrict the value for process trees.

[pznamensky]

Indeed, we had to disable swap globally.
Thanks for the workaround.

[langmartin]

@pznamensky did you happen to see this? https://news.ycombinator.com/item?id=20651510 The article references a thread from this week on the linux kernel mailing list, the summary offered in the article is that leaving at least a bit of swap enabled on even the latest linux kernel allows the system to avoid some very ugly behavior under memory pressure. You may want to investigate leaving the global swap enabled and running nomad in a cgroup instead. Cheers

[pznamensky]

@langmartin sorry for the delay and thanks for the link. Indeed, this is an old issue with swap, and we make a decision to use a little swap in our production, but I expected that nomad disables it for containers as described here:
https://www.nomadproject.io/docs/drivers/docker.html#memory

[shantanugadgil]

Just reached here after @honnibal pointed it out. I would love to leave the swappiness behavior as-is.
Thereby letting Nomad provide swapping to docker containers if users need it.

_I wonder how Nomad 0.9.5 and 0.10-beta1 behave._

I am for Nomad allowing "swap". 🦊

[AcaDemIQ]

zram linux module use swap mechanism for compressing memory in desktop/server/etc.

[fho]

1.) Please correct me if I'm wrong. It seems to me that the nomad documentation is wrong. https://www.nomadproject.io/docs/drivers/docker.html#memory states:

Nomad limits containers' memory usage based on total virtual memory. This means that containers scheduled by Nomad cannot use swap.

But docker containers scheduled via nomad are allowed to use swap:
In the cgroups of the docker containers memory.memsw.limit_in_bytes is set to the same value then memory.limit_in_bytes and swappiness is not altered.
memory.memsw.limit_in_bytes limits the amount of Swap+RAM the container can use.
memory.limit_in_bytes limits the amount of RAM the container can use.
If in the parent cgroup swappiness is enabled (the default), it has the effect that the docker container is allowed to use swap.

Nomad should set swappiness to 0 for scheduled docker container. Docker supports it via docker run --memory-swappiness)

2.) It would be great to be able to configure the swappiness per nomad job. The resources or docker driver section in a jobspec could have a swappiness parameter.

[pznamensky]

Is there a plan to fix it?

[shantanugadgil]

I am confused with the various issues around this. What is the latest news with versions 0.10.x and the 0.11.x?
Is swap usage now completely disabled, with no way to make Docker containers use swap. If true, then 😢
Will there be an option and/or documented way of enabling and letting the Docker containers use swap if needed?

[notnoop]

@shantanugadgil The PR #7550 does indeed disable swap, as we never intended to support it.

We can entertain adding a flag or supporting swappiness if there is a demand. Mind if you elaborate on your case? Do you currently rely on current swap behavior and how so? Would you like more swap knobs at the client/job levels.

If I understand correctly, nomad scheduling only factor in available RAM on the host, so swap memory should only be used if non-nomad work is using up memory and/or some of the nomad tasks are unconstrained (e.g. raw_exec) and use more memory than declared. Does that seem accurate to you?