Containers-roadmap: Add capability ecs.capability.docker-plugin.rexray in case rexray is installed

[bsingr]

Summary

AWS ECS Agent does not add capability ecs.capability.docker-plugin.rexray altough docker info reports it as volume plugin.

Description

Installed rexray on debian based EC2 using $ curl -sSL https://rexray.io/install | sh (not docker plugin install rexray/ebs or sth because it failed for NVME volumes).

I can create rexray volumes via docker volume and mount them in docker run -v ..., but not using ECS Task Definition because the ecs.capability.docker-plugin.rexray is required for task placement but not exposed by ecs-agent.

Expected Behavior

The ecs agent should list the attribute ecs.capability.docker-plugin.rexray

Observed Behavior

The ecs agent lists only attributes like

• ecs.capability.docker-plugin.local
• or ecs.capability.docker-plugin.rexray/ebs if I install the docker plugin install rexray/ebs (which is not sufficient in my case i guess)

Consequently the task placement fails with service foo was unable to place a task because no container instance met all of its requirements. The closest matching container-instance fc5811db-3634-4710-b6b8-942866b41549 is missing an attribute required by your task. For more information, see the Troubleshooting section.

Environment Details

$ docker volume ls
DRIVER              VOLUME NAME
rexray              foo

$ rexray version
REX-Ray
-------
Binary: /usr/bin/rexray
Flavor: client+agent+controller
SemVer: 0.11.4
OsArch: Linux-x86_64
Commit: e7414eaa971b27977d2283f2882825393493179d
Formed: Tue, 15 Jan 2019 16:03:57 UTC

$ docker info
Containers: 50
 Running: 49
 Paused: 0
 Stopped: 1
Images: 34
Server Version: 18.09.7
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rexray
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-9-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.805GiB
Name: ip-10-129-2-117
ID: GH3E:S6KA:25K3:5ILX:ZK2P:WZBC:WHDT:MLY4:QVAK:XMFE:XYOR:MNLB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

This might be an issue, i don't know:

$ docker plugin ls
ID                  NAME                DESCRIPTION         ENABLED

$ docker run   --rm   --name ecs-agent-2   --volume=/var/run/docker.sock:/var/run/docker.sock   --volume=/var/log/ecs:/log   --volume=/var/lib/ecs/data:/data   --net=host   --env-file=/etc/ecs/ecs.config   --env=ECS_LOGFILE=/log/ecs-agent-2.log   --env=ECS_DATADIR=/data/   --env=ECS_ENABLE_TASK_IAM_ROLE=true   --env=ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true   --label service=ecs-agent-2   --log-driver=fluentd   --log-opt tag=ecs_agent   --log-opt labels=service   amazon/amazon-ecs-agent:v1.29.0 -ecs-attributes
2019-07-10T14:26:34Z [INFO] Loading configuration
2019-07-10T14:26:34Z [INFO] Image excluded from cleanup: amazon/amazon-ecs-agent:latest
2019-07-10T14:26:34Z [INFO] Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0
2019-07-10T14:26:34Z [INFO] Amazon ECS agent Version: 1.29.0, Commit: a190a73f
com.amazonaws.ecs.capability.privileged-container
com.amazonaws.ecs.capability.docker-remote-api.1.17
com.amazonaws.ecs.capability.docker-remote-api.1.18
com.amazonaws.ecs.capability.docker-remote-api.1.19
com.amazonaws.ecs.capability.docker-remote-api.1.20
com.amazonaws.ecs.capability.docker-remote-api.1.21
com.amazonaws.ecs.capability.docker-remote-api.1.22
com.amazonaws.ecs.capability.docker-remote-api.1.23
com.amazonaws.ecs.capability.docker-remote-api.1.24
com.amazonaws.ecs.capability.docker-remote-api.1.25
com.amazonaws.ecs.capability.docker-remote-api.1.26
com.amazonaws.ecs.capability.docker-remote-api.1.27
com.amazonaws.ecs.capability.docker-remote-api.1.28
com.amazonaws.ecs.capability.docker-remote-api.1.29
com.amazonaws.ecs.capability.docker-remote-api.1.30
com.amazonaws.ecs.capability.docker-remote-api.1.31
com.amazonaws.ecs.capability.docker-remote-api.1.32
com.amazonaws.ecs.capability.logging-driver.json-file
com.amazonaws.ecs.capability.logging-driver.fluentd
com.amazonaws.ecs.capability.task-iam-role
com.amazonaws.ecs.capability.task-iam-role-network-host
ecs.capability.task-cpu-mem-limit
com.amazonaws.ecs.capability.ecr-auth
ecs.capability.execution-role-ecr-pull
ecs.capability.container-health-check
ecs.capability.docker-plugin.local
ecs.capability.private-registry-authentication.secretsmanager
ecs.capability.secrets.ssm.environment-variables
ecs.capability.secrets.ssm.bootstrap.log-driver
ecs.capability.pid-ipc-namespace-sharing
ecs.capability.ecr-endpoint
ecs.capability.secrets.asm.environment-variables
ecs.capability.secrets.asm.bootstrap.log-driver
ecs.capability.aws-appmesh
ecs.capability.task-eia
ecs.capability.container-ordering

$ uname -a
Linux ip-10-129-2-117 4.9.0-9-amd64 aws/amazon-ecs-agent#1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 GNU/Linux

[sharanyad]

Hi @bsingr,
Did you restart agent after installing the volume driver? This is necessary for volume plugin to be detected and added as a capability.

[bsingr]

@sharanyad yes, the command above docker run --rm --name ecs-agent-2 --volume=/var/run/docker.sock:/var/run/docker.sock --volume=/var/log/ecs:/log --volume=/var/lib/ecs/data:/data --net=host --env-file=/etc/ecs/ecs.config --env=ECS_LOGFILE=/log/ecs-agent-2.log --env=ECS_DATADIR=/data/ --env=ECS_ENABLE_TASK_IAM_ROLE=true --env=ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true --label service=ecs-agent-2 --log-driver=fluentd --log-opt tag=ecs_agent --log-opt labels=service amazon/amazon-ecs-agent:v1.29.0 -ecs-attributes starts the agent after rexray daemon is already running.

[yumex93]

Hi, agent gets volume driver from two places. Code snippet is here. One is scanning through "/run/docker/plugins" directory, another one is calling docker api to list all plugins. The issue here is raxray does not show at either of them.

So due to this, this will become a feature request that, since it shows in docker info, we can explore about adding another code path that also pull into the plugins showing there.

[fischaz]

Yeah, it will actually work although I had to work around the ecs-agent packaging and system dir to actually make the /run/docker/plugins available to the ecs-agent (inside a container) so it can actuallly see that directory and those .sock unix socket file.

One note though is that it also discover my 'auth' plugin in that directory (which is a bit misleading as it's not a volume plugin).

Indeed, I'd see if we can simply run a docker info API call and list the Plugins.Volume list:

    out, err := m.dockerClient.Info(m.ctx)
    if err != nil {
        log.Printf("Failed to query Local docker daemon: %v", err)
        return false, err
    }

    if len(out.Plugins.Volume) == 0 {
        log.Printf("No Volume plugin detected yet...")
        return false, nil
    }

    for _, vp := range out.Plugins.Volume {
        ...
    }

[angegar]

Hi,
I also have difficulties to mount and EBS Volume on an ECS container and to be honest i don't really understand what you said.

Did you manage to make it works, if yes can you please provide a pointer ?

[Trane9991]

As @bsingr mentioned it works fine in case of installation via docker plugin install.

@angegar you can find usage example in my project here: https://github.com/getsocial-rnd/neo4j-aws-causal-cluster

[ceefour]

The error message needs so much improvement. I got this error message after installing docker plugin rexray and I can't launch tasks. After rebooting the ECS instance, it works again. It should mention which attribute is needed.

@Trane9991 Note that I did install rexray via docker plugin install command and I still get this error, need to reboot the ECS EC2 instance (I don't know how to restart ecs-agent). So, it is _not_ "works fine in case of installation via docker plugin install".

[iamprashantv]

I faced the same issue, it was an issue with the driver name that you configured in the task definition. Please use the driver name: rexray/ebs.