Amazon-ecs-agent: Fine grained control over capabilities (--cap-add, --cap-drop)

[adrienkohlbecker]

Hello,

Support for privileged mode was introduced recently, which is nice. However, there is no support for --cap-add and --cap-drop in the configuration, which allows for a more fine grained control over what's allowed or not.
Moreover, --privileged changes some AppArmor and SELinux settings, which might not be needed in all cases.

My use case is simple, I don't need --privileged, I just need --cap-add=SYS_ADMIN to be able to umount a volume.

Best

[jessesanford]

Would really like to see this. I need to add SYS_PTRACE to a machine that contains tracing instrumentation.

[mattmanning]

+1 for this

We want to support mounting FUSE filesystems which only needs --cap-add SYS_ADMIN --device to acess /dev/fuse, not full-blown --privileged.

[adrienkohlbecker]

@mattmanning FYI, you will probably also need a custom seccomp profile with docker >= 1.10
https://github.com/docker/docker/issues/20092
https://github.com/docker/docker/issues/20082

[aboutvale]

+1 for this, especially --cap-add SYS_ADMIN --device

[noreabu]

+1 because i like to have --cap-add LINUX_IMMUTABLE for security improvements that will help many people sleep better at night :+1:

[adamjkeller]

+1

[aaronsturm]

+1

[eukaryote]

+1 to this. Is it being worked on? I'm surprised there is no way currently to remove unneeded capabilities in ECS.

[voltechs]

Guys, please don't +1 comments unless you have something else to add; it gets noisy.

Use GitHub's comment-emoji feature.

Can anyone comment on if this is in the works?

[ChristopherMotes]

+1 - From a general security perspective, we should have the ability to drop all unused privileges for any container/application. In our current case, we need NET_ADMIN to futz with the container's network namespace. Running a web facing container in privileged mode is a show stopper for ECS.

[herriojr]

Instead of having to make modifications for each option added to docker, why not just allow people to arbitrarily set the list of flags?

[ebekker]

+1 Looking for --cap-add=IPC_LOCK

[liorchen]

+1

[corydodt]

Opened almost 2 years ago at this point. Any status update?

[singhjagmohan1000]

We need --cap-add SYS_PTRACE to view syscall info, as we can not run container in privileged mode. Please do something about this.

[sharanyad]

Thanks for all the comments here! We've been looking into how to support this and think we'll take the following approach:

• We'll add new fields to the task definition to indicate Linux capabilities to add and drop
• We'll add a way for the ECS agent to indicate what capabilities are known by the kernel; this will ensure that a given task definition is actually placeable on your container instances
• The ECS agent may auto-detect capabilities or allow you to specify which ones matter to you. Since the set of capabilities that are possible is pretty large, we'd appreciate feedback as to whether everything should be auto-detected or not. Another approach could be letting the agent auto-detect the most common capabilities and also let you specify the other required capabilities.

Your input is highly appreciated.

[nmeyerhans]

IMO it's not actually worth auto-detecting and registering support of individual capabilities. The list of capabilities that the kernel supports really doesn't change very much over time. The commit history for the capabilities manpages suggests that no new capabilities have been added since at least 2014, and the majority of them are much older than that.. I think we're over-engineering the solution if we try to handle the list dynamically.

It's notable that Docker itself doesn't try to do anything clever around detection of capability support either.

[eukaryote]

The only one of the three bullet points that I would make use of is the first one:

• We'll add new fields to the task definition to indicate Linux capabilities to add and drop

Also, please don't overlook 'ALL' as an option.

I'd prefer a simple implementation that just passes through the capabilities to Docker as provided. I know which kernel versions I run, which capabilities they support, and which I want to add and remove for a given task, and I want exactly the ones I specify to be passed to Docker, with no possibility of ecs-agent doing anything clever that results in something other that what I specified being used.

[lawrencepit]

We want to do --cap-drop=all --cap-add=sys_ptrace

See also: http://rhelblog.redhat.com/2016/10/17/secure-your-containers-with-this-one-weird-trick/

[jhmartin]

While running Vault I'd like to drop several but add IPC_LOCK. I could just enabled privileged but that adds a bunch of capabilities it doesn't need.

[pagameba]

I'm also looking for IPC_LOCK support specifically for Vault support.

[karnauskas]

Any progress on this? +1 for Vault.

[cs-mahmoud-khateeb]

is it really needed for vault? the purpose of --cap-add=IPC_LOCK is to lock memory for this container so it doesn't get swapped to disk which is not even supported in aws (at least not yet) isn't that right??

[sharanyad]

Hi all,

The add and drop Linux Capabilities feature has been released:
https://aws.amazon.com/about-aws/whats-new/2017/09/amazon-ecs-adds-support-for-adding-or-dropping-linux-capabilities-to-containers/

Customers can now add/drop Linux Capabilities to each container via the Task Definition. Please refer to linuxParameters section here:
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_security

These options can now be added via Configure via JSON on the ECS Console or with the AWS CLI or SDKs, which will be available later today.

Thank you.

[lawrencepit]

@sharanyad Could it be it's not quite released yet? I'm getting the following response back when using aws ecs register-task-definition --cli-input-json file:///tmp/example.json in eu-west-1:

Parameter validation failed:
Unknown parameter in containerDefinitions[0]: "linuxParameters", must be one of: name, image, cpu, memory, memoryReservation, links, portMappings, essential, entryPoint, command, environment, mountPoints, volumesFrom, hostname, user, workingDirectory, disableNetworking, privileged, readonlyRootFilesystem, dnsServers, dnsSearchDomains, extraHosts, dockerSecurityOptions, dockerLabels, ulimits, logConfiguration

[sharanyad]

@lawrencepit Sorry for the inconvenience. The CLI team is investigating this and we shall get back as soon as this gets resolved.

[sharanyad]

@lawrencepit It's fixed now. Updating your CLI should work. Please let us know if there are other concerns.
Thanks.

[lawrencepit]

@sharanyad Unfortunately I still get the exact same error response.

[sharanyad]

@lawrencepit Did you try after updating the AWS CLI?

[lawrencepit]

@sharanyad missed that one, sorry. Yes, it works now. Thanks a lot!

[pikeas]

Having some trouble getting this to work. Task definition has:

"containerDefinitions": [
  {
    "linuxParameters": {
      "capabilities": {
        "add": [
          "SYS_ADMIN"
        ]
      }
  }
]

This is a new ECS cluster running Docker 17.06.2-ce and ECS Agent 1.16.0.

The service throws an error: "unable to place a task because no container instance met all of its requirements. The closest matching container-instance <...> is missing an attribute required by your task."

Is anything else needed to get this working?

[sharanyad]

@pikeas A task with linux capabilities field in the task definition does not require any attribute for it to be placed in an instance. I suspect there is/are some other field(s) in the task definition that is causing this issue.
Could you please post here the other fields that you're using in the task definition and output of describe-container-instances for the instance? If you're not comfortable sharing this information here, please feel free to send it directly at sharanyd at amazon.com.

[pikeas]

@sharanyad I just checked describe-container-instances for one of the instances in the ECS cluster, and it appears to have all of the capabilities needed by the task.

The task definition has:

com.amazonaws.ecs.capability.docker-remote-api.1.17
com.amazonaws.ecs.capability.docker-remote-api.1.18
ecs.capability.task-eni
com.amazonaws.ecs.capability.ecr-auth
com.amazonaws.ecs.capability.logging-driver.awslogs
com.amazonaws.ecs.capability.docker-remote-api.1.19

The instance needs:

com.amazonaws.ecs.capability.logging-driver.syslog
com.amazonaws.ecs.capability.logging-driver.none
com.amazonaws.ecs.capability.logging-driver.json-file
ecs.capability.execution-role-awslogs
com.amazonaws.ecs.capability.docker-remote-api.1.17
com.amazonaws.ecs.capability.docker-remote-api.1.18
com.amazonaws.ecs.capability.docker-remote-api.1.19
ecs.capability.task-eni
ecs.capability.task-cpu-mem-limit
com.amazonaws.ecs.capability.docker-remote-api.1.30
ecs.capability.execution-role-ecr-pull
com.amazonaws.ecs.capability.task-iam-role-network-host
com.amazonaws.ecs.capability.logging-driver.awslogs
com.amazonaws.ecs.capability.docker-remote-api.1.24
com.amazonaws.ecs.capability.docker-remote-api.1.25
com.amazonaws.ecs.capability.privileged-container
com.amazonaws.ecs.capability.ecr-auth
com.amazonaws.ecs.capability.docker-remote-api.1.20
com.amazonaws.ecs.capability.docker-remote-api.1.21
com.amazonaws.ecs.capability.docker-remote-api.1.22
com.amazonaws.ecs.capability.task-iam-role
com.amazonaws.ecs.capability.docker-remote-api.1.23

[imran9m]

@sharanyad is there any update after @pikeas feedback?. I have docker 18.03 and ecs agent 1.17.3 on Ubunutu. I am trying to achieve --cap-add SYS_ADMIN --device /dev/fuse with ECS Task def and it's not working with capabilities and devices. It works only with privileged option.

```
"linuxParameters": {
"capabilities": {
"add": [
"SYS_ADMIN"
],
"drop": null
},

"devices": [
{
"containerPath": "/dev/fuse",
"hostPath": "/dev/fuse",
"permissions": null
}
],
```

[amitaymolko]

@imran9m Did you get this working?