Amazon-ecs-agent: Upgrading from 16.2: Error retrieving credentials

[flowirtz]

Summary

Upgrading from 16.2, as prompted by AWS and done like this leaves me with a Timeout on the Credential retrieving part. Downgrading back to 16.2 (same Task, same everything) fixes it again. This just cost me a whole workday.

Description

I have a ECS Task that works perfectly fine on ecs-agent v16.2.
Now today I wanted to deploy a new version of my app to ECS. Greeted by the alarming

"🚨UPGRADE NOW 🚨"

warning I did just that.
This left me with nine hours of debugging in total, as I got above mentioned error after the upgrade.
Now, the setup is exactly the same: I didn't change the task, containers or anything. The only difference is the version of the container agent.
I could not find any breaking changes in the changelog, so I am guessing this is a bug.

Expected Behavior

Shouldn't throw a timeout.

Observed Behavior

Throws a timeout.

Environment Details

sent via email to @aaithal

Supporting Log Snippets

Error when retrieving credentials from container-role:
Error retrieving metadata: Received error when attempting to retrieve ECS metadata:
HTTPConnectionPool(host='XXX.XXX.XXX.X', port=80):
Max retries exceeded with url: /v2/credentials/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX 
(Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPConnection object at 
0xXXXXXXXX>, 'Connection to XXX.XXX.XXX.X timed out. (connect timeout=2)'))

Might be related to #447

[aaithal]

Hi @FWirtz, thanks for creating this issue. We're sorry that you're running into this. It's probably a coincidence that you saw this issue in 1.16.2. We suspect it's been there since 1.16.0 and you're most likely running into #1230. We're working on a fix for this in https://github.com/aws/amazon-ecs-cni-plugins/pull/73. This will be included in the next ECS agent release.

Thanks,
Anirudh

[flowirtz]

@aaithal Just to clarify: Everything works well on 16.2 - things start breaking once I upgrade to 17.1.
Looking forward to the PR thanks!

[richardpen]

@FWirtz This has been fixed in the aws/amazon-ecs-cni-plugins 2018.02.0 which is included in the agent v1.17.2. Please upgrade to the latest ECS Agent, I'm closing this issue now, feel free to reopen if you still experience this problem.

[JChanceHud]

Hi @aaithal, I seem to be experiencing this issue on the current latest version of the docker amazon/amazon-ecs-agent image (which I believe is v1.17.3).

Our docker images rely on iam permissions for retrieving configuration files from S3 and are throwing the same error message as in the original issue:
Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: HTTPConnectionPool(host='XXX.XXX.XXX.XXX', port=80): Max retries exceeded with url: /v2/credentials/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPConnection object at 0xXXXXXXXXXXXX>, 'Connection to XXX.XXX.XXX.XXX timed out. (connect timeout=2)'))

I've been debugging the container and it successfully made the request once or twice, but the vast majority of the time the s3 transfer fails.

Things I've tried:

• Restarted containers
• Restarted host machine
• curl http://169.254.169.254/latest/meta-data/ from host machine
• curl http://169.254.169.254/latest/meta-data/iam/info from container
• Running aws s3 cp command from host machine

Any advice would be appreciated.

[aaithal]

Hi @JChanceHud, you seem to be running into https://github.com/aws/amazon-ecs-agent/issues/1231. Can you try setting the value of ECS_TASK_METADATA_RPS_LIMIT to meet your higher request rate? Feel free to create a new issue if you keep running into this.

Thanks,
Anirudh