Amazon-ecs-agent: Add support for `--shm-size` `docker run` flag

[aaithal]

Has 7 +1s and the following mentions:

• 502
• 655

[sdeprez]

Hi ,

Any ETA for this feature? Or any workaround for it in the meantime?

Basically, I need to run some version of Chromium headless inside containers, and that requires more than the default 64M of shared memory so at the moment it's simply impossible to run it.

Actually a quick but probably dirty hack is to mount the /dev/shm of the container to the /dev/shm of the container instance in your task definition. The equivalent in docker run would be:
docker run -v /dev/shm:/dev/shm. Use with caution, it may have some side effects I'm not aware of.
Another workaround is to run docker in privileged mode and manually increase the shared memory size from inside, see https://bugs.chromium.org/p/chromium/issues/detail?id=519952#c6.

Thanks,

[csvan]

+1, also running Chromium inside ECS here. Next to impossible with default shm size.

[linuxfreakus]

+1, ditto on headless chromium.

[matthew-cox]

+1, I would really prefer support for this option vs implementing the possible work arounds

[trstephen]

+1, headless chromium is a shrinking violet with default 64mb shm

[AlexMarchini]

+1, also stuck until this is implemented.

[mkleint]

we are also waiting for the Amazon ECS AMi to contain latest docker daemon to at least be able to change the default size of shm to be bigger than 64M

[alinpa]

+1, I am waiting to see this feature implemented.
Did anyone find any workaround for this issue?

[zackarychapple]

Any update on when this will be a reality? We are running into issues on this as well.

[tobypinder]

@alinpa we use

"Volumes": [
    {
      "HostDirectory": "/dev/shm",
      "ContainerDirectory": "/dev/shm"
    }
  ],

in the Dockerrun.aws.json to effectively hand the host's shm to the guest container. I am unclear of any side effects here but it appears to work well for our single container environment. I'm not an expert but you and others here may find this a suitable workaround and we haven't noticed any issues arising from this yet.

EDIT: I should mention to aid understanding that this is the exact equivalent of @sdeprez's approach, for AWS instead of docker run on a local machine.

[aatreya]

Anyone interested in this for use with Chrome/Chromium may also want to star / comment on these bugs:
https://bugs.chromium.org/p/chromium/issues/detail?id=715363
https://bugs.chromium.org/p/chromium/issues/detail?id=736452

[zackarychapple]

This article is amazing https://medium.com/@marco.luethy/running-headless-chrome-on-aws-lambda-fa82ad33a9eb as it shows how to remove the requirement on shm with a custom binary.

[linuxfreakus]

Yes, its pretty cool, but if you're having SHM issues already with chromium, they are likely to continue with lambda. You can force it to run in that environment, but when I tried, it didn't work for our purposes. Very crashy.

[linuxfreakus]

FWIW, we currently hack it by granting elevated privileges and remounting /dev/shm on container boot. Not optimal, especially if you have strong security requirements, but one of the only ways to safely make it happen from a memory allocation standpoint if you use more than one container. Other than that, you have to give up ECS and roll your own.

[paulrutter]

+1
See https://github.com/cyrus-and/chrome-remote-interface#why-my-program-stalls-or-behave-unexpectedly-if-i-run-chrome-in-a-docker-container

[mrzacarias]

I'm trying to use Google Puppeteer on ECS and I'm having this problem. I'm using Fargate as well, so there's no way to use the privileged mode hack...

Do you have an ETA for that? That's keeping me and my team away of using cool stuff like ECS...

[mkleint]

FYI on latest Amazon ECS AMIs (or any AMI with docker 17.06+) it's possible to increase the size of /dev/shm by passing --default-shm-size 256M to the docker daemon on startup.

[mrzacarias]

@mkleint That are great news! I couldn't find the documentation about this option, do you have any source explaining how to use this?

[stephen]

From reading the latest in https://bugs.chromium.org/p/chromium/issues/detail?id=736452#c56, it looks like this issue is fixed in Chrome >= 64.0.3281.0 with a --disable-dev-shm-usage flag.

[mkleint]

@mrzacarias https://github.com/moby/moby/pull/29692 https://docs.docker.com/enterprise/17.06/

[sharanyad]

A proposal for the implementation of shmsize is available in https://github.com/aws/amazon-ecs-agent/pull/1149.

[aaithal]

Hello everyone, we announced support for shm-size parameter in the task definition today! Please refer to our documentation for more details.

Closing this issue for now. Please let us know if you have any follow up comments/questions regarding this.

[robertorocha3]

@aaithal have you added this option to cloudformation? I don't see it in here:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-linuxparameters.html
If not, do you have a recommended workaround for this?