Amazon-ecs-agent: Agent 1.44.2 doesn't allow introspection endpoint to be accessed off-host

[leshik]

Summary

We rely on introspection being accessible inside the internal VPC network. #2588 breaks this.

Expected Behavior

ecs-agent should either bind to 0.0.0.0, or at least allow this to be configurable.

[ellenthsu]

Thanks for reporting this, my team is looking into this issue with priority

[yhlee-aws]

working on impact mitigation: https://github.com/aws/amazon-ecs-agent/pull/2605

[yhlee-aws]

Latest ECS Optimized AMI with agent v1.44.3 has been released, which reverts the change.
You can update to the latest ECS Agent via update-container-agent api or via AWS ECS Console.

Resolving.

[fenxiong]

Note: we are still going to roll forward a change to block off-host access to agent's introspection endpoint by default. The change is not made in agent itself, but is made in the ecs-init RPM (which is used on the ECS Optimized AMI) with an iptables rule that blocks introspection port access coming from eth0 interface. Details of the change can be found in the description of this pull request https://github.com/aws/amazon-ecs-init/pull/354. Access to the introspection endpoint from container running in a task that uses awsvpc network mode (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-networking.html) will be blocked as well as a result, but access from containers (to the introspection endpoint on the same host) running in other network modes or directly via Docker should not be affected.

As mentioned in the pull request, this change is made configurable via a flag, so after we release the change, if you still want to enable off-host access to agent's introspection endpoint and are using the ecs-init RPM, you can still do so, just that you need to enable it explicitly by setting the flag. The change will be released in our next release (1.45.0).

If you have concern over the change, let us know. Thanks.

[mmmeff]

Can you please provide more guidance on the correct way to set this flag?

[ellenthsu]

Yes, here's the pull request: https://github.com/aws/amazon-ecs-init/pull/354

This behavior will be configurable via env ECS_ALLOW_OFFHOST_INTROSPECTION_ACCESS (when set to true, it allows off host/awsvpc access). To set this, for AL1, add env ECS_ALLOW_OFFHOST_INTROSPECTION_ACCESS=true to /etc/init/ecs.conf; for AL2, add ECS_ALLOW_OFFHOST_INTROSPECTION_ACCESS=true to /etc/ecs/ecs.config.