Amazon-ecs-agent: ECS missing many docker run options

[hasimo]

Amazon ECS is Docker Management Serivice.
Amazon ECS Product Details said

Amazon EC2 Container Service supports Docker and enables you to run and manage Docker containers across a cluster of Amazon EC2 instances. Each EC2 instance in a cluster managed by
Amazon ECS runs a Docker daemon, so whatever application you package as a container locally will deploy and run on Amazon ECS without the need for any configuration changes.

But many case configuration changes is needed. because ECS support a few docker run options only.

This is the list. blank is not supported.

| docker run options | ecs | comment |
| --- | --- | --- |
| image | limited | digest is not supported |
| --add-host | supported | |
| --blkio-weight | | |
| --blkio-weight-device | | |
| --cap-add | | |
| --cap-drop | | |
| --cgroup-parent | | |
| ---cidfile | | |
| --cpu-percent | | |
| --cpu-period | | |
| --cpu-quota | | |
| --cpu-shares | | |
| --cpuset-cpus | | |
| --cpuset-mems | | |
| --detach-keys | | |
| --device | | |
| --device-read-bps | | |
| --device-read-iops | | |
| --device-write-bps | | |
| --device-wirte-iops | | |
| --disable-content-trust | | |
| --dns | supported | |
| --dns-opt | | |
| --dns-search | supported | |
| --entrypoint | supported | |
| --env | supported | |
| --env-file | | |
| --expose | limited | no range support |
| --group-add | | |
| --helth-cmd | | |
| --helth-interval | | |
| --helth-retries | | |
| --helth-timeout | | |
| --hostname | supported | |
| --io-maxbandwidth | | |
| --io-maxiops | | |
| --ip | | |
| --ipv6 | | |
| --ipc | | |
| --kernel-memory | | |
| --label | supported | |
| --label-file | | |
| --link | supported | |
| --link-local-ip | | |
| --log-driver | supported | |
| --log-opt | supported | |
| --mac-address | | |
| --memory | supported | |
| --memory-reservation | supported | |
| --memory-swap | | |
| --memory-swappiness | | |
| --name | | |
| --network-alias | | |
| --network | limited | no user defined network |
| --no-helthcheck | | |
| --oom-kill-disable | | |
| --oom-score-adj | | |
| --pid | | |
| --pids-limit | | |
| --privileged | supported | |
| --publish | limited | no range support |
| --pubilsh-all | | |
| --read-only | supported | |
| --restart | | |
| --rm | | |
| --runtime | | |
| --security-opt | supported | |
| --shm-size | | |
| --sig-proxy | | |
| --stop-signal | | |
| --storage-opt | | |
| --sysctl | | |
| --tmpfs | | |
| --user | supported | |
| --userns | | |
| --uts | | |
| --volume-driver | | |
| --volume-from | | |
| --volume | limited. | [rw |ro] options only |

I probably miss something.

Please add those options to run Docker-enabled applications without any configuration changes.

---

Why do we need that options?
That is already discussed in docker repository. Please check it.

[kiranmeduri]

Hi @hasimo,

Thanks for compiling the list. We are constantly adding options supported by Docker to ECS as applicable. Is there any specific options you are more interested in, which can help us prioritize? The reason I say is that some of the options may not be directly applicable in ECS ecosystem. For e.g. 'restart' will be an issue w.r.t the state management on the cluster which can be solved by ECS Service Scheduler.

-kiran

[hasimo]

Hi @kiranmeduri,

for restart option,
I made #493 and I'm still thinking ECS supporting restart options is useful for me.
because ECS Service Scheduler is not same as restart options. docker's restart options has on-failure,always and unles-stopped.
I would like to use docker's restart options.
Also sometime I would like to schedule by myself and pass restart option to StartTask and RunTask.
would be great ECS Scheduler start working after docker give up restarting.

The main thing of this issue is supporting those options quickly and following the docker's innovation.

My idea is adding AdvancedContainerOptions to TaskDefinition.
It is overrides options of CreateContainer.
ecs-agnet unmarshal it and deep merge to CreateContainerOptions.

This is example.

"AdvancedContainerOptions": {
  "Config":{
    "Domainname": "aaa.example.com",
    "StopSignal": "SIGKILL"
  },
  "HostConfig":{
    "MemorySwappiness": 60,
    "PidMode": "host"
  },
  "NetworkConfig":{
    "isolated_nw" : {
      "IPAMConfig": {
        "IPv4Address":"172.20.30.33",
        "IPv6Address":"2001:db8:abcd::3033",
        "LinkLocalIPs":["169.254.34.68", "fe80::3468"]
      },
      "Links":["container_1", "container_2"],
      "Aliases":["server_x", "server_y"]
    }
  }
}

Thank you.

Tohru

[Nuean]

Hello @kiranmeduri

I would like to request that the --stop-signal parameter get added as an ECS option.

There is an issue using .netcore 1.0 in docker containers where SIGTERM is not able to be captured when the container is stopped, preventing a graceful shutdown. My understanding is the .netcore runtime captures it but does not pass it on to any running applications.

However, SIGINT is captured correctly and by passing --stop-signal=SIGQUIT to docker run, docker will send a SIGINT in place of the SITGTERM, and things can be shutdown safely.

I hope this issue can get some attention.

Thanks!
Nuean

[gollawil]

For docker run I would like to see --shm-size added.
Also, being able to specify the timeout used by docker stop would be great too.

[jbergknoff]

It would be nice to support --cpu-period and --cpu-quota so that we can limit a container's CPU usage, rather than just guarantee it a minimum CPU share. I've seen some situations where it seems that greedy tasks consume entire CPU cores and make it impossible for the ECS agent and/or Docker daemon to function.

If there is some more fundamental approach which could guarantee that some slice of CPU will always be reserved for the ECS agent and the Docker daemon, that would be even better.

[fundkis]

@kiranmeduri

We are constantly adding options supported by Docker to ECS as applicable.

This is an endless task as docker add more options. Everyone around can find good reason for a missing option. I think ECS should provide a way to pass options to the underlying docker commands.

The one I miss (for now) is --shm-size

[davidmarquis]

I somewhat agree with @fundkis although I can see why some options might cause issues with ECS.

Support for the --ipc option would be much useful.

In some cases IPC helps services running multiples tasks on the same host that need to work together communicating with each other with much less overhead than network.

[autopulated]

Support for the sysctl option would be useful, especially to allow setting somaxconn.

[xsellier]

@autopulated For the sysctl option, I found this workaround:
http://stackoverflow.com/questions/26177059/refresh-net-core-somaxcomm-or-any-sysctl-property-for-docker-containers/26197875#26197875

Otherwise, start the container with the privileged flag (sysctl should eb available).

Not sure if it is a good idea ...

[johntiror]

In order to support ElasticSearch I'd need --cap-add=IPC_LOCK and sysctl in order to set vm.max_map_count
https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

[herriojr]

cap-add is important for me as well -- I need to use /dev/fuse

[jwkuo]

I need --cap-add=IPC_LOCK as well for Vault server implementation. See Docker Hub's Vault repo: https://hub.docker.com/_/vault/ Seems like ECS should support whatever collection of options for the supported version of Docker. ATM, 1.12.6

[lrvick]

Lack of --env-file makes it imposible to load in dynamically generated env vars (such as those created by a kms decryption container)

[etsuo]

We need --cap-add (specifically, --cap-add=NET_ADMIN)

[notjames]

+1 on cap-add. We also need --cap-add=NET_ADMIN in order to run VPN server containers.

[simplesteph]

sysctl and memory-swapiness for me in order to run Kafka and Zookeeper properly

[cxmcc]

+1 for CAPADD

[darvids0n]

I need cap-add=SYS_ADMIN to handle NFS (EFS) mounts in containers.

[EugeniuZ]

--tmpfs

[aaithal]

Hi All,

Thank you for providing these details regarding your use-cases for various docker run options that are not yet supported in ECS. However, having a single catch-all issue makes it harder for us when prioritizing these on our roadmap. We want to split this up into multiple issues in order to bring in more focus and to be able to sort these feature requests by different parameters. Hence, we are closing this issue, but with pointers to issues that are more focused on each flag/option that was asked here. Please feel free to continue the discussion in the respective issues:

• #787: --shm-size
• #788: --ipc
• #789: --sysctl
• #223: --cap-add, --cap-drop
• #247: --env-file
• #790: --memory-swapiness
• #469: --tmpfs

Thanks,
Anirudh

[EugeniuZ]

Shouldn't https://github.com/aws/amazon-ecs-agent/issues/469 be added to the list above?

[jhaynes]

@EugeniuZ Sure should. I've updated @aaithal's list accordingly.

[ibrahima]

This is an endless task as docker add more options. Everyone around can find good reason for a missing option. I think ECS should provide a way to pass options to the underlying docker commands.

I think this is really worth considering, in addition to all the issues for specific flags. At least some type of unsupported, you're-on-your-own API for passing arbitrary flags would be nice, because as @fundkis said, there's always going to be new flags for Docker run that people want to use. Some of them might conflict with ECS agent but at that point the developer is responsible for anything that goes wrong.

[chicofranchico]

Is the list on the top being constantly updated or is there a better place to follow that?

[UnitasBrooks]

--cpu-rt-runtime is also missing.

[nickm4062]

--pid as well

[AdrianAntunez]

I would like to use the --cpuset-cpus parameter. By that way we could limit in which cpu should be executed each docker container.

Like @hasimo states, I think the easiest and more reasonable way IMHO would be to define a way to add/forward extra parameters directly from task-definition to docker run.

[aaithal]

@AdrianAntunez Thanks for reaching out to us with this use-case. We do not support setting that parameter today via the ECS task definition.

However, we are taking into account all of these use-cases in thinking about making it more flexible to support these flags. We'll let you know when we have an update.

Thanks,
Anirudh

[mvanholsteijn]

In Marathon, the task definition has a array 'parameters' which allows you to set any of the Docker parameters.

      "parameters": [
        {
          "value": "true",
          "key": "rm"
        },
        ...
    ]

[dtoubelis]

--ipv6

[varxy20k]

--restart would be great!

[outime]

Can we get support for IOPS limits? I feel it's essential to prevent single apps from consuming all available I/O.

• --device-read-bps
• --device-write-bps
• --device-read-iops
• --device-write-iops

[mfortin]

Can the --userns option also be added ? I have a use case of a container running in privileged mode and I need to enable user namespace on my EC2 instances. setting --userns=host to the running container seems like my only option of getting it to work.

[sanchitaittech]

For --dns-opt is still unsupported by ECS and therefore cannot be passed to the task definition as a flag currently.
So please add dns-opt in the docker run

[gitfool]

I also need the option to set --userns=host so I can enable user namespacing on the cluster nodes but allow exceptions for specialized "infrastructure" daemon sets: Consul Agent and Registrator, Datadog, Fluent Bit, HA Proxy, etc.