Amazon-ecs-agent: Unable to start task. Fails with error 'ATTRIBUTE'

[anirudhjayakumar]

Hi,

I'm testing two changes I made to our ECS deployments.

1. Logging with Splunk
2. use awsvpc to get task specific IP.

When I run the task (via Java API) I get an error with REASON="ATTRIBUTE" and arn=

After googling a bit, I found the issue is due to missing attribute.

I'm pasting the relevant part of task definition and describe-cluster. Looks like com.amazonaws.ecs.capability.docker-remote-api.1.19 and com.amazonaws.ecs.capability.docker-remote-api.1.18 are missing in my container instance.

How do I remove this from my task definition?

From describe-cluster

"attributes": [
                {
                    "name": "ecs.ami-id",
                    "value": "ami-5253c32d"
                },
                {
                    "name": "ecs.capability.execution-role-awslogs"
                },
                {
                    "name": "ecs.vpc-id",
                    "value": "***"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
                },
                {
                    "name": "ecs.capability.task-eni"
                },
                {
                    "name": "ecs.capability.task-cpu-mem-limit"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
                },
                {
                    "name": "ecs.capability.execution-role-ecr-pull"
                },
                {
                    "name": "ecs.capability.container-health-check"
                },
                {
                    "name": "ecs.subnet-id",
                    "value": "***"
                },
                {
                    "name": "ecs.availability-zone",
                    "value": "us-east-1b"
                },
                {
                    "name": "ecs.instance-type",
                    "value": "t2.xlarge"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role-network-host"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.splunk"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.26"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.27"
                },
                {
                    "name": "com.amazonaws.ecs.capability.privileged-container"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
                },
                {
                    "name": "ecs.capability.cni-plugin-version",
                    "value": "***"
                },
                {
                    "name": "com.amazonaws.ecs.capability.ecr-auth"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
                },
                {
                    "name": "ecs.os-type",
                    "value": "linux"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
                }
            ]

From Task Defintion

requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.splunk"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    }
  ]

Thanks
Anirudh

[anirudhjayakumar]

@aaithal you seem to have answered a bunch of similar questions. Could you help me?

[richardpen]

@anirudhnair Based on the information you provided, the container instance seems have all the attributes required by the task definition:

com.amazonaws.ecs.capability.docker-remote-api.1.18
com.amazonaws.ecs.capability.docker-remote-api.1.19
com.amazonaws.ecs.capability.docker-remote-api.1.21
com.amazonaws.ecs.capability.logging-driver.splunk
com.amazonaws.ecs.capability.task-iam-role
ecs.capability.task-eni

If you are using the awsvpc mode, the possibilitythat you get this error "ATTRIBUTE" is that the vpc/subnet you specified in your start-task command doesn't match the vpc of the container instance. Can you check that first, and if you still have this issue, please send the following information to me at: penyin (at) amazon.com

• Account ID
• Task definition ID
• Command to start task
• Container intance ID and region

Thanks,
Peng

[anirudhjayakumar]

@richardpen That is correct. Your suggestion works. The VPC was same but the subnet I provided in the AwsVpcConfiguration is different from the one the container instance was on.

[nitinyadav745]

@richardpen This solution worked for me also , thanks a lot.

[ceefour]

The error message needs so much improvement. I got this error message after installing docker plugin rexray (https://github.com/aws/containers-roadmap/issues/533) and I can't launch tasks. After rebooting the ECS instance, it works again. It should mention which attribute is needed.

[petderek]

Agreed -- we're planning to improve these error messages. Definitely let us know if any others seem more opaque than necessary.

[vennemp]

I'm getting a similar error. I have just enabled App Mesh and converted my tasks to use awsvpctrunking. The tasks work fine when launched with AWS provided AMI, but we have to use an ami hardened with CIS. I have a custom ami with the CIS benchmarks and the ecs agent properly configured. The tasks run with default bridge networking but not work awsvpc mode on this image. Please note: I also manually opted into vpctrunking for ecs.

My hunch says to enable task networking on the container instance - I added ECS_ENABLE_TASK_ENI=true to the ecs.config file. But when I view the attribute on the container instance in the ECS console it shows the attribute as unassigned. I have tried manually adding the line, and adding it via user data but nothing updates the value. I also restarted the ecs service everytime and still nothing. If this is the correct path forward to enable, please let me know. How can i update the attribute?? Has anyone has either of these problems before?