Amazon-ecs-agent: issues using ecs-optimized ami without ECS service or with false configuration

[mwiora]

hi @all,

I wanted to keep open the possibility to easily join my dockerhosts to Amazon ECS and therefore I've decided to use the amzn-ami-2015.09.g-amazon-ecs-optimized (ami-341efb5b) AMI.

After a week I had 330 Millions IOPS, which was causing my bill getting higher and higher. I had to investigate for a few minutes and finding the issue....

Directly after booting something is causing a very high IOPS load:

$ dstat -c --top-cpu -d --top-bio --top-latency
----total-cpu-usage---- -most-expensive- -dsk/total- ----most-expensive---- highest-total
usr sys idl wai hiq siq| cpu process | read writ| block i/o process | latency process
17 11 44 28 0 0|docker 5.6| 47M 31M|bash 2966B 12k|rcu_sched 22
20 18 48 14 0 0|docker 9.0| 60M 36M| |kworker/0:3 32
18 16 52 15 0 0|docker 7.0| 60M 39M| |kworker/0:3 34
21 14 51 14 0 0|docker 8.0| 62M 40M| |rcu_sched 34
15 17 55 14 0 0|docker 8.0| 62M 32M| |ntpd 26
13 13 60 13 0 0|docker 7.0| 58M 47M| |native 36^C

Looking for the logfiles of the ecs-agent dockercontianer I found tons of this entry:

==> /var/log/ecs/ecs-agent.log.2016-03-18-00 <==
2016-03-18T00:53:12Z INFO Detected Docker versions http://1.17 1.18 1.19 1.20
2016-03-18T00:53:12Z WARN Error getting valid credentials (AKID ): NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2016-03-18T00:53:12Z INFO Registering Instance with ECS
2016-03-18T00:53:12Z ERROR Could not register module="api client" err="NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
2016-03-18T00:53:12Z CRITICAL Could not register module="api client" err="NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors"
2016-03-18T00:53:12Z ERROR Error registering: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Maybe it would be useful to stop retrying bringing up the docker container again and again after a specified amount of retries and just warn the owner.
I know that this is semi-related to the source of amazon-ecs-agent, but this issue-tracker seems to be the next-possible issue tracker of the usage of the amazon-ecs-agent.

Cheers,
Matthias

[samuelkarp]

Hi @mwiora,

2016-03-18T00:53:12Z [CRITICAL] Could not register module="api client" err="NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors"

This indicates that no credentials are available to the ECS Agent. When launching an EC2 instance for use with ECS, you'll want to attach an instance profile with a role allowing access to the ECS API.

The disk activity you're seeing is a secondary effect of this; the nanny process that is responsible for ensuring the Agent stays up keeps restarting the Agent container when it exits with this error. The reason it keeps restarting is that the ECS-optimized AMI is intended for use with ECS and that the error can be corrected post-launch (the policy applied to the role can be changed); if the nanny were to stop restarting the Agent, fixing an error here would require both correcting the policy and manual intervention on the instance to restart the Agent.

After a week I had 330 Millions IOPS, which was causing my bill getting higher and higher.

The AMI, by default, is configured to use Amazon EBS General Purpose (SSD) volumes which do not charge for IOPS. The only IOPS usage-based pricing for EBS volumes are Amazon EBS Magnetic volumes. I would recommed using either of the SSD volume types (either General Purpose or Provisionied IOPS) as the disk latency has a direct effect on the responsiveness of the Docker daemon. More information on volume types is available in the EBS documentation.

Please let us know if you have any further questions.

Thanks,
Sam

[mwiora]

hi @samuelkarp,

yeah - that was a qualified answer - thank you man!

Cheers,
Matthias