Amazon-ecs-agent: aws ecs Agent exited with code 1

[Vikash082]

I have create a ECS cluster, but Registered container instances - 0 . There is an instance launched on the process of create cluster.

I did an ssh into instance and tailed log:

 $ tail -f /var/log/ecs/ecs-init.log
    2018-03-21T13:01:15Z [INFO] Removing existing agent container ID: 101574ba6691b1877e2fec3b4473c770dbb85dc9354ef01f98924abf35bb6ab6
    2018-03-21T13:01:15Z [INFO] Starting Amazon Elastic Container Service Agent
    2018-03-21T13:01:15Z [INFO] Agent exited with code 1
    2018-03-21T13:01:15Z [INFO] Container name: /ecs-agent
    2018-03-21T13:01:15Z [INFO] Removing existing agent container ID: 687520b03c5dc64c82a04285768fb2a59718c5cc590c65117b065684fa5b6756
    2018-03-21T13:01:15Z [INFO] Starting Amazon Elastic Container Service Agent
    2018-03-21T13:01:16Z [INFO] Agent exited with code 1
    2018-03-21T13:01:16Z [INFO] Container name: /ecs-agent
    2018-03-21T13:01:16Z [INFO] Removing existing agent container ID: 61f1a878db90d593b42f417f809956a2e11f557b12185030a747df4e8a860b33

There is internet gw attached and also the instance has public IP. Inbound security rule allow traffic on port 80.

[apricote]

I believe we had the same problem when setting up our cluster.
Did you define the cluster that the instance is supposed to join?

We have following script set as user data of our Launch Configuration:

#!/bin/bash
echo ECS_CLUSTER=prod >> /etc/ecs/ecs.config

So you might want to check the file /etc/ecs/ecs.config on the instance.

[aaithal]

Hi @Vikash082, can you please check what the ECS agent logs say? They should be in /var/log/ecs/ directory as well, with the ecs-agent.log prefix. If you don't want to disclose it here, you can send them to aithal at amazon dot com. You could also use ECS logs collector to collect these logs.

Thanks,
Anirudh

[Vikash082]

@aaithal Here is the log from /var/log/ecs

$ tail -f /var/log/ecs/ecs-agent.log.2018-03-22-04
2018-03-22T04:15:28Z [INFO] Loading configuration
2018-03-22T04:15:28Z [INFO] Event stream ContainerChange start listening...
2018-03-22T04:15:28Z [INFO] Loading state! module="statemanager"
2018-03-22T04:15:29Z [WARN] Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:29Z [INFO] Registering Instance with ECS
2018-03-22T04:15:29Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:29Z [ERROR] Error registering: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:34Z [INFO] Loading configuration
2018-03-22T04:15:34Z [INFO] Event stream ContainerChange start listening...
2018-03-22T04:15:34Z [INFO] Loading state! module="statemanager"
2018-03-22T04:15:35Z [WARN] Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:35Z [INFO] Registering Instance with ECS
2018-03-22T04:15:35Z [ERROR] Could not register: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:35Z [ERROR] Error registering: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-03-22T04:15:40Z [INFO] Loading configuration
2018-03-22T04:15:40Z [INFO] Event stream ContainerChange start listening...
2018-03-22T04:15:40Z [INFO] Loading state! module="statemanager"
2018-03-22T04:15:41Z [WARN] Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Any other info needed ?

[Vikash082]

@apricote I have checked the config file.

$ cat /etc/ecs/ecs.config 
ECS_CLUSTER=my-cluster

[aaithal]

Hi @Vikash082,

It seems like you're missing an IAM role for the instance. Can you please make sure that you're specified the appropriate IAM role for the instance? Here's some instructions for the same: Amazon ECS Container Instance IAM Role.

Thanks,
Anirudh

[Vikash082]

Hi @aaithal , the EC2 instance was created implicitly from ECS and has "ecsInstanceRole" attached for IAM role. It has 2 policies attached - 1 . AmazonEC2ContainerServiceforEC2Role 2. AmazonEC2ContainerServiceRole . I also have checked checked Trust Relationship and it is same as specified in link.

[aaithal]

In that case, can you please use ECS logs collector and send the tar ball to aithal at amazon dot com? I'd also suggest creating an AWS support case as our support organization is better equipped to handle cases such as these where you're running into setup issues.

Thanks,
Anirudh

[Vikash082]

Done

[aaithal]

Done

Hi @Vikash082, can you please let me know what were you attributing the Done for? Was it an ack that you had sent logs or that you had engaged AWS support? Just a fyi that I haven't received any logs in my inbox.

Thanks,
Anirudh

[Vikash082]

Hi Anirudh,

I have resend it just now. Earlier, I have sent on Mar 22 at - [email protected]

[aaithal]

Hi @Vikash082,

Thanks for sending those logs. I'd still recommend that you reach out to AWS support as you seem to be running into some setup issue.

From taking a look at the logs, it seems like there's some issue with your IAM role setup. Can you manually ssh into the instance and run these commands?

sudo yum -y install aws-cli
aws ecs create-cluster --region us-east-1 --cluster-name ocx-cluster

That should hopefully provide information as to if IAM role is set up properly on that instance. Also, can you please share the EC2 instance id in an email with me as well (same email id as before)?

Please note that our recommendation is for you to open up a support case with AWS support as they are the best people to help you with any of the setup issues.

Thanks,
Anirudh

[aaithal]

Hi @Vikash082, closing this issue as we haven't heard from you in a while. Feel free to let us know if you have follow up questions/comments. Thanks!

[Vikash082]

Hi @aaithal , The issue was the trust relationship for ecsInstanceRole.

{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}

AWS ECS was creating "Service" with value "ecs.amazonaws.com", it needed to be "ec2.amazonaws.com". AWS support folks pointed that.