Amazon-ecs-agent: --pid=host support

[marinusd]

Related: #185

Use case:
Resque workers identify themselves with a string made up of ${hostname}:${pid}
This id string allows resque's management utility to present groups of workers-by-hostname, as well as selecting individual workers. The resque worker id is not externally configurable.

When an ECS Service starts >1 resque workers (defined in a Task, obviously), there are two possibilities:

1. The Task Definition specifies a container hostname, or
2. The Task Definition does not specify a container hostname.

If 1., then all the containers have the same hostname, and (by virtue of starting as identical tasks) the same PID. Thus the resque worker id string is identical for all running tasks. This defeats the resque management utility's ability to select an individual worker, plus other undesirable effects.

If 2., then the containers get unique hostnames, and so the resque workers id strings are unique. But this defeats the resque management utility's ability to present groups of workers-by-hostname.

Supporting the _--pid=host_ Docker run option (to use the host's PID namespace vs. the container's PID namespace) will help restore the resque management utility's abilities.

[errordeveloper]

We need this for Weave Scope. Another use-case would be Kubernetes on ECS. _(cc @bchav)_

[veqryn]

Another use case is infrastructure monitoring

[reedflinch]

@veqryn 👍 , would love to use this to implement Sysdig in ECS task definitions

[panga]

+1

[xrwang]

+1, this would be great to use for infrastructure monitoring

[lodotek]

+1

[bjbishop]

👍 @veqryn We really need this for installing Sysdig in ECS as well

[MarvGilb]

+1, for infrastructure monitoring like new relic.
#502 lists the parameter that ECS is missing.

[i-ghost]

This would be very useful to run the Stackify agent with! 👍

[joekiller]

Supporting this would allow datadog to use its psutil features like in https://github.com/DataDog/dd-agent/issues/2923

[PaulLiang1]

Almost two years. Another reason to ditch ECS.

[buybackoff]

Any update/ETA on this? Without this option I cannot use safely LMDB on instance storage from multiple containers from ECS. https://twitter.com/buybackoff/status/1013503236857368576

[nunofernandes]

+1
I would like to have a icinga/nagios running inside a container and lots of monitoring checks do need the proc namespace to be accessible..

[arminc]

A lot of security related containers rely on this, we finally have ECS Daemon Scheduling but we can't use it because this feature is not available.

[MoLow]

+++!

[sharanyad]

Hi everyone,
Thank you all for your feedback. We are currently exploring options to address this feature. In particular, we are assessing whether the pid option should be configured at the task level (for all containers) or separately for each container. Do you have use-cases where you need to set pid=host for only a subset of containers in the Task? For comparison, currently the network mode is configured at the task level for simplicity.

[MoLow]

The same as network mode is fine for my use case

[joekiller]

Task based toggle is likely sufficient but container based may be
necessary.

It depends on the agent's purpose. If the agent needs to talk over network
to processes in a task, ie query nginx stats, it needs networking to link
via task definition and pid=host for the agent container but I wouldn't
want the setting for the my whole task.

Contrarily, when using an agent to monitor tasks in an ubiquitous
environmental where that agent in the task will reconfigure on the fly
based on what else is running on that container host. The agent would
likely be its own task where the toggle on the task would be sufficient.

So if I had to pick one, container based toggle.

[nunofernandes]

The same as network mode is also fine for my use case

[xose]

Another vote for container-level setting, even though a task-level setting would also be "enough" for my current use case:

• All other security/isolation settings are configurable per-container (except networking)
• Least privilege: this is a setting you may not want all containers in a task to share.
• Simplicity: it's an advanced, rarely used setting for very specific use cases, unlike the settings that are now available at the task level (networking, memory, cpu). It would complicate the UI unnecessarily for most people.

[i-ghost]

Per-container would be ideal, but I can live with task level as a compromise.

[sharanyad]

Thanks for your suggestions and feedback. It appears that the use-cases mentioned above will be addressed by having the following process namespace settings in the task definition at the task level:

• pid="host" where all the containers’ process namespace is set to host
• pid="task" where all the containers in the task share the same process namespace

The ability to have all the containers in the task share the same namespace (with pid=task), without any of them having pid="host", will address the use-case mentioned by @joekiller where the "agent" is a side-car container inside the task. It also addresses the security concern of having to set-up pid="host" for all containers. pid="host" will address the use-case where the "agent" has its own task (and is likely deployed as a Daemon Service). Finally, setting the parameter at the task level seems to be the most intuitive and simple behavior since it requires only one parameter to be configured in the whole Task Definition (and not one per container), while achieving both use-cases.

Please let us know if you have any further feedback or comments.

[xose]

That looks like a good middle ground, it works for my use case. Just a couple questions:

• Which container's namespace will be joined when using pid="task"? That is, which container from a task definition will have a process with PID 1?
• Will there be a third option to keep the current behaviour of each container in a task having its own isolated PID namespace? Or will pid="task" become the default?

[arminc]

I like the proposed solution but had the same questions as @xose.
If I understand correctly not setting the "pid=" option would mean the current behaviour while we get the option to set it with the two purposed solutions to change the default behaviour?

Because loosing the isolation when you deploy a task witch has a primary container and a simpel side-car container where now the side-car container can access way to much information from the primary container would not be a good thing.

[sharanyad]

Thanks for your comments.
If the pid field is not set, it defaults to the current behavior where each container gets its own PID namespace.
In the case of pid=task, are there use cases where you want a particular container to have PID=1?
There are two options that we are considering:

• Have an internal empty container and run all the task’s containers in the internal container’s namespace.
• Run any one container in the task initially and start the remaining containers in the first container’s namespace. If there are “links” and “volumesFrom” dependencies, those will be taken into consideration while choosing the base container for the pid namespace.

[joekiller]

If I had to choose, both of course 😄 Run an empty container and launch with consideration to links and volumes. perhaps otherwise just launch in order of the the container definitions.

[wmitsuda]

This is needed to run prometheus node_exporter on ECS, for example: https://github.com/prometheus/node_exporter#using-docker

[charlieoleary]

Also needed to support the new Threat Stack Docker-based agent.

[boynux]

+1

[cnelson]

+1

[ashmastaflash]

+1

[charlieoleary]

Support was added for --pid with the release of 1.22.0! 🎉 It appears that adding "pidMode": "host" to your container definition will enable the functionality.

[ashmastaflash]

Thanks @charlieoleary, that's great news!

[linkar-ec2]

Hi all,

Closing this issue.
Amazon ECS now supports an pidMode (as well as ipcMode) option as a parameter for ECS Task Definitions. You can now add host or task as pidMode options. Please see the latest announcement and the AWS Docs for more information. Please reach out to us for any questions/comments/clarifications.

Thanks,
Kar Shin

[wmitsuda]

Any timeframe for this parameter to be now supported inside CloudFormation templates?