Amazon-ecs-agent: Containers configured with awslogs never get placed on an instance

Created on 23 May 2018  路  17Comments  路  Source: aws/amazon-ecs-agent

Summary

When trying to set up logs on Windows containers, setting them to auto-configure for awslogs causes the containers to never be placed. The CloudWatch log group does get created.

Description

Two Windows EC2 instances were set up as part of a cluster using the Windows_Server-2016-English-Full-ECS_Optimized-2018.05.01 (ami-46c77939) image. A service set up with containers that require awslogs can't place containers because "no container instance met all of its requirements." Trying to run a taks via the API returns a failure of "ATTRIBUTE".

Expected Behavior

Tasks are placed on instances and log streams are created and content pushed to it.

Observed Behavior

Tasks never get placed.

Environment Details

Docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.06.2-ee-10
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.2214.amd64fre.rs1_release_1.180402-1758)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 2
Total Memory: 4GiB
Name: EC2AMAZ-Q72RKI3
ID: HYRW:725W:EX6L:YSQB:CN5Z:OXVV:4FTS:KXNN:2CK2:VZKY:IBJF:HLDG
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Metadata:

PS C:\Users\Administrator> curl http://localhost:51678/v1/metadata


StatusCode        : 200
StatusDescription : OK
Content           : {"Cluster":"mobiledev-cluster","ContainerInstanceArn":"arn:aws:ecs:us-east-1:169164411397:container-instance/910a2bc4-4972-4dea-8b8b-b4d5e36ca7ed","Version":"Amazon ECS Agent -
                    v1.17.3 (159ae5c3)"}
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 197
                    Content-Type: text/plain; charset=utf-8
                    Date: Wed, 23 May 2018 13:48:44 GMT

                    {"Cluster":"mobiledev-cluster","ContainerInstanceArn":"arn:aws:ecs:us-east-1:16916...
Forms             : {}
Headers           : {[Content-Length, 197], [Content-Type, text/plain; charset=utf-8], [Date, Wed, 23 May 2018 13:48:44 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 197

One thing I noticed is the task definition requires an attribute of ecs.capability.execution-role-awslogs but when I describe the instance that attribute is not found:

{
    "failures": [],
    "containerInstances": [
        {
            "status": "ACTIVE",
            "registeredAt": 1526920787.49,
            "registeredResources": [
                {
                    "integerValue": 2048,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "CPU",
                    "doubleValue": 0.0
                },
                {
                    "integerValue": 4095,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "MEMORY",
                    "doubleValue": 0.0
                },
                {
                    "name": "PORTS",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [
                        "135",
                        "445",
                        "3389",
                        "2376",
                        "139",
                        "2375",
                        "80",
                        "5985",
                        "51678",
                        "51679",
                        "53"
                    ],
                    "type": "STRINGSET",
                    "integerValue": 0
                },
                {
                    "name": "PORTS_UDP",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [],
                    "type": "STRINGSET",
                    "integerValue": 0
                }
            ],
            "ec2InstanceId": "[omit]",
            "agentConnected": true,
            "containerInstanceArn": "[omit]",
            "pendingTasksCount": 0,
            "remainingResources": [
                {
                    "integerValue": 1024,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "CPU",
                    "doubleValue": 0.0
                },
                {
                    "integerValue": 3071,
                    "longValue": 0,
                    "type": "INTEGER",
                    "name": "MEMORY",
                    "doubleValue": 0.0
                },
                {
                    "name": "PORTS",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [
                        "2375",
                        "8080",
                        "135",
                        "445",
                        "3389",
                        "2376",
                        "139",
                        "80",
                        "5985",
                        "51678",
                        "51679",
                        "19800",
                        "53"
                    ],
                    "type": "STRINGSET",
                    "integerValue": 0
                },
                {
                    "name": "PORTS_UDP",
                    "longValue": 0,
                    "doubleValue": 0.0,
                    "stringSetValue": [],
                    "type": "STRINGSET",
                    "integerValue": 0
                }
            ],
            "version": 342,
            "attributes": [
                {
                    "name": "ecs.ami-id",
                    "value": "ami-46c77939"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.json-file"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.30"
                },
                {
                    "name": "ecs.capability.execution-role-ecr-pull"
                },
                {
                    "name": "ecs.capability.container-health-check"
                },
                {
                    "name": "ecs.availability-zone",
                    "value": "us-east-1c"
                },
                {
                    "name": "ecs.instance-type",
                    "value": "t2.medium"
                },
                {
                    "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.24"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.26"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.27"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.28"
                },
                {
                    "name": "com.amazonaws.ecs.capability.privileged-container"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
                },
                {
                    "name": "com.amazonaws.ecs.capability.ecr-auth"
                },
                {
                    "name": "ecs.os-type",
                    "value": "windows"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.20"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.22"
                },
                {
                    "name": "com.amazonaws.ecs.capability.docker-remote-api.1.23"
                },
                {
                    "name": "com.amazonaws.ecs.capability.task-iam-role"
                }
            ],
            "versionInfo": {
                "agentVersion": "1.17.3",
                "agentHash": "159ae5c3",
                "dockerVersion": "DockerVersion: 17.06.2-ee-10"
            },
            "runningTasksCount": 1,
            "attachments": []
        }
    ]
}

Supporting Log Snippets

ECS agent logs: ecs-agent-logs.zip
Docker events: docker-events.zip

kinquestion
Source
picture parkrrr

Most helpful comment

Hi @parkrrr,

I saw you have specified the Task execution role, actually it's designed for Fargate task, you don't need to set it if you are using EC2. If you want to use it on EC2, you must set ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE to true when starting the instance, as Agent only register itself once when it initializes. There are two ways to solve your problem:

  1. Remove the Task execution role.

  2. Keep Task execution role, create another two Windows instances, add one more line PowerShell in user data when creating them:

[Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", 
$TRUE, "Machine")

Thanks,
Haikuo

picture haikuoliu on 25 May 2018
👍21 🎉2

All 17 comments

Hi @parkrrr,

I saw you have specified the Task execution role, actually it's designed for Fargate task, you don't need to set it if you are using EC2. If you want to use it on EC2, you must set ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE to true when starting the instance, as Agent only register itself once when it initializes. There are two ways to solve your problem:

  1. Remove the Task execution role.

  2. Keep Task execution role, create another two Windows instances, add one more line PowerShell in user data when creating them:

[Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", 
$TRUE, "Machine")

Thanks,
Haikuo

haikuoliu picture haikuoliu on 25 May 2018
👍21 🎉2

Oh wow, yes removing the task execution role fixed it. Thank you very much!

parkrrr picture parkrrr on 25 May 2018

if one is using "Installing the Amazon ECS Container Agent on a non-Amazon Linux EC2 Instance " from https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html on Step 9. append /etc/ecs/ecs.config with following

ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true

task execution role is required if using "AWS Systems Manager Parameter Store" for storing sensitive data so it can not be removed in these cases.

kortemik picture kortemik on 27 Feb 2019
👍11

I had this error in my ECS service when using an Ubuntu container host and this solved my issue:

service VoiceEdgeECS-kamailio was unable to place a task because no container instance met all of its requirements. The closest matching container-instance 9f6c639f-2418-4407-bec6-d6f37753263a is missing an attribute required by your task. For more information, see the Troubleshooting section.

rayjanoka picture rayjanoka on 5 Mar 2019

@parkrrr did you actually get your ECS windows nodes to auto-register with the cluster when they were spun up? I have no success in that area. Works fine with linux ecs instances.

Drewster727 picture Drewster727 on 5 Mar 2019

@Drewster727 Yes, I don't recall having issues with that.

3xlogicParker picture 3xlogicParker on 5 Mar 2019

@3xlogicParker yeah, I don't even really know how to troubleshoot it either, I cannot get the windows admin password from the nodes, and they never register with the cluster. Any tips?

Drewster727 picture Drewster727 on 5 Mar 2019

@Drewster727 There's some user-data scripts that run that are supposed to register with the cluster (at least, I think they do), I suspect your problem is something related to that script. Are you using a custom AMI?

3xlogicParker picture 3xlogicParker on 5 Mar 2019

Yeah, I suspected the same thing, but I'm not using a custom AMI. Using the standard ECS optimized 2016 ami -- Windows_Server-2016-English-Full-ECS_Optimized-2018.10.23 (ami-0aa2bc91e0ae61f20)

Drewster727 picture Drewster727 on 5 Mar 2019

I figured it out, sorry to hijack the thread. Note to others on windows optimized ECS nodes... do not allow ipv6 to get attached, it seems to cripple the ECS instance.

Drewster727 picture Drewster727 on 6 Mar 2019

Option 2 from @haikuoliu worked like a charm. Thanks! 馃憤

alexandreabeh picture alexandreabeh on 20 Mar 2019

Hi, having a similar issue with Windows 2019 ECS Optimized Amazon AMI. My task runs only without a Task Role because the EC2 instance has not enabled/activated the

com.amazonaws.ecs.capability.task-iam-role

attribute required when the task has a Task Role. I realize that after describe the ecs container instance and compare with the required attributes of the task.

Following thread comments I already put the following lines for user metadata on EC2 instance:

<powershell>
  Import-Module ECSTools
  Initialize-ECSAgent 鈥揅luster DemoCluster -EnableIAMTaskRole
  [Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", $TRUE, "Machine")
</powershell>

But tasks with Task Role stills fails with message: Run Tasks Failed Reasons : [鈥淎TTRIBUTE鈥漖

Hope someone can give an advice. Thanks in advance!

czarbl picture czarbl on 11 Mar 2020

Hello,
It is EnableTaskIAMRole , not EnableIAMTaskRole

sharanyad picture sharanyad on 11 Mar 2020
👍2

Hello,
It is EnableTaskIAMRole , not EnableIAMTaskRole

Hi Sharanyad, following your suggestion I made the change for user data to this:

<powershell>
  Import-Module ECSTools
  Initialize-ECSAgent 鈥揅luster DemoCluster -EnableTaskIAMRole
  [Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", $TRUE, "Machine")
</powershell>

But after EC2 restart and describing again the ecs container instance still no contains the attribute:

com.amazonaws.ecs.capability.task-iam-role

Do I lacking of something else?

Thanks in advance!

czarbl picture czarbl on 11 Mar 2020

Hi @parkrrr,

I saw you have specified the Task execution role, actually it's designed for Fargate task, you don't need to set it if you are using EC2. If you want to use it on EC2, you must set ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE to true when starting the instance, as Agent only register itself once when it initializes. There are two ways to solve your problem:

  1. Remove the Task execution role.
  2. Keep Task execution role, create another two Windows instances, add one more line PowerShell in user data when creating them:
[Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", 
$TRUE, "Machine")

Thanks,
Haikuo

saved my day as tried to change a fargate template to be used in bridge mode thanks 馃挴

gauravbrills picture gauravbrills on 25 Apr 2020

Hello,
It is EnableTaskIAMRole , not EnableIAMTaskRole

Hi Sharanyad, following your suggestion I made the change for user data to this:

<powershell>
  Import-Module ECSTools
  Initialize-ECSAgent 鈥揅luster DemoCluster -EnableTaskIAMRole
  [Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", $TRUE, "Machine")
</powershell>

But after EC2 restart and describing again the ecs container instance still no contains the attribute:

com.amazonaws.ecs.capability.task-iam-role

Do I lacking of something else?

Thanks in advance!

To save others the time when referencing the above, this should work if the environment is configured prior to the Initialize-ECSAgent call. I also had to update $TRUE to "true" in the current version before the container instance would run.

Hope it helps.

CharlesMichaelReed picture CharlesMichaelReed on 21 Sep 2020

@CharlesMichaelReed 's comments helped me the most here. To clarify, set the environment variable before the Initialize-ECSAgent comment and use the string "true". See below.

<powershell>
  Import-Module ECSTools
  [Environment]::SetEnvironmentVariable("ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE", "true", "Machine")
  Initialize-ECSAgent 鈥揅luster DemoCluster -EnableTaskIAMRole
</powershell>

To save others the time when referencing the above, this should work if the environment is configured prior to the Initialize-ECSAgent call. I also had to update $TRUE to "true" in the current version before the container instance would run.

Hope it helps.

peterjgrainger picture peterjgrainger on 10 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

flowirtz picture flowirtz  路  5Comments
devotox picture devotox  路  3Comments
soumyasmruti picture soumyasmruti  路  5Comments
aaithal picture aaithal  路  3Comments
hayajo picture hayajo  路  3Comments