Amazon-ecs-agent: Support `--sysctl ` `docker run` flag

[aaithal]

From 502 (Has 4 +1s):

Support for the sysctl option would be useful, especially to allow setting somaxconn.

[wezzynl]

Yeah, being able to set somaxconn would be very, very welcome.

[mancej]

We definitely need this feature ASAP, the workaround is a security risk.

[jdreaver]

Are there any workarounds to set sysctl parameters until this feature is implemented? In particular, I want to set net.ipv4.ip_local_port_range and net.core.somaxconn. There is a lot of advice on the internet about workarounds, but a lot of it is old and I want to follow best practices here without shooting myself in the foot.

[ayozemr]

Suscribed. We may need to tune somaxconn too for UWSGI python server.

Thanks!

[mancej]

Just a heads up, we instituted a workaround (for now, while we wait for this), but unfortunately it involves running the container as privileged. We also had to set the somaxconn on the host itself to get it working. That being said, it does work.

[shoumitragametime]

@mancej Can you give me a little insight on the workaround? I'm running into a similar issue where I am trying to set the tcp_keepalive_time for one of my Airflow operators. Though it would be great if Amazon could deliver on the feature soon.

[jdreaver]

@shoumitragametime if you set your container as privileged then you can just call sysctl yourself on startup.

AWS also recently implemented adding Linux capabilities to containers (https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-ecs-adds-support-for-adding-or-dropping-linux-capabilities-to-containers/). I haven't tried this yet but maybe you don't have to use privileged and can instead just add a particular capability.

[wezzynl]

Any update on this? It's really gnarly to have to run a lot of containers eating up memory because we can't simply increase somaxconn.

[owengo]

Any update? being unable to configure properly the tcp_keepalive_xxx sysctl settings causes TCP_ELB_Reset_Count errors on NLB.

[johvet]

Any update on this one? Running into the same problem. need to adjust somaxconn

[jnmcross]

We also need to set net.ipv4.tcp_keepalive_* properties.
If this doesn't get fixed we may move off of ECS

[shoumitragametime]

@jnmcross I was able to get around this by running my container in a privileged mode(if you are okay with that) and executing sysctl -w net.ipv4.tcp_keepalive_time=xxx net.ipv4.tcp_keepalive_intvl=xxx net.ipv4.tcp_keepalive_probes=x in my entrypoint.sh script. Just make sure your script is executable and that should work.

[ziggythehamster]

If you're not OK with running a privileged container and exposing read-write procfs and sysfs to your container for its lifetime, this trick works for us on latest Docker/ECS:

nsenter --target $FIND_YOUR_CONTAINER_PID_SOMEHOW --mount --uts --ipc --net --pid \
   /bin/sh -c '/usr/bin/mount /proc/sys -o remount,rw;
               /usr/sbin/sysctl -q net.ipv6.conf.all.forwarding=1;
               /usr/bin/mount /proc/sys -o remount,ro;
               /usr/bin/mount /proc -o remount,rw # restore rw on /proc'

We built this into an internal tool which polls Docker for new containers having a sysctls label. Then we validate the (pipe-delimited) sysctls against a whitelist (net.core.* basically) and plug them into this command and have our tool run it.

Kind of a janky workaround for something ECS should support out of the box, but it works.

Also, I'd add that this has to run, as root, on the host machine since the namespaces of other containers would be inaccessible if you ran this in a container. You also need procps or equivalent installed in the container.

[nehalrp]

+1, any update on this? We need to set keepalives to work with NLBs.

[rifelpet]

We were relying on the <4.13 kernel behavior of containers inheriting sysctl changes from the host in order for our containers to use the following overrides:

net_ipv4_tcp_tw_reuse = 1
net_core_somaxconn = 10000
net_ipv4_tcp_max_tw_buckets = 250000
net_ipv4_tcp_fin_timeout = 10
net_ipv4_ip_local_port_range = 25000 61000

This behavior changed with 4.13 as mentioned on the bottom of this page.

When we upgraded to the latest Amazon Linux AMI we lost those sysctl overrides. While the workarounds mentioned in this issue do work, it'd be great to avoid those altogether and instead specify them in the task definition.

[nehalrp]

@rifelpet Agreed. Especially since Amazon Linux AMI 2018.03 uses a 4.14+ kernel.

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/

[jishi]

I want to point out that allowing setting the net.ipv4.tcp_keepalive_time is a very useful feature to mitigate the silently dropped connections in the AWS NAT gateways (which happens after 5-6 minutes for each connection), since not all applications/driver allows you do configure the desired keepalive delay.

We always have issues with this on any externally hosted service that uses sockets, where Compose being one of the biggest culprits. Adding sysctl options to the taskDefinition seems to be by far the most bang for the buck IMO :)

[shahidash]

Hi
net.ipv4.tcp_keepalive_time is an important variable to set, when using the aws infrastructure. As we are using NLB to connect the multiple docker services. and NLB just has the idle timeout of 350 seconds, in this case we need the net.ipv4.tcp_keepalive_time to increase the NLB idle timeout.

[devops-eatigo]

@shoumitragametime can you share some insights of your entrypoint.sh or how you run the sysctl command? i tried to add the command in dockerfile RUN and command section under ContainerDefinition, but both failed with some no path/no such file or permission denied errors. I did set privileged to true.

[wezzynl]

Not having control over various Docker run options has been reported as early as 20 August 2016... Now more than 2 years later, we're still not able to set kernel parameters with Docker's sysctl flag. This is ridiculous.

[wezzynl]

@aaithal Can you provide us with an update?

[sharanyad]

You can now add sysctl options to your containers in your task definition. You can add these new options in the AWS Console now, and they'll be available in the AWS CLI and SDKs soon.

[PaulLiang1]

Looks like Cloudformation team has not catch up yet..

Encountered unsupported property SystemControls

[jrgwv]

Is there any estimated time when Cloudformation will have this ability in the task definition?