Amazon-ecs-agent: Metadata Endpoint Resetting Connection

[thiagoscherrer]

Summary

Metadata is not accessible or is declined

Description

After updating the ecs-agent, the endpoin metadata is declining the connections.
I'm using CoreOS as a host, and to reproduce the problem, I'm using the official version of ubuntu

Observed Behavior

curl 169.254.170.2/v2/metadata
curl: (56) Recv failure: Connection reset by peer

# nmap 169.254.170.2

Starting Nmap 7.60 ( https://nmap.org ) at 2019-03-08 14:40 UTC
Nmap scan report for 169.254.170.2
Host is up (0.000049s latency).
Not shown: 999 filtered ports
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 9.26 seconds

Environment Details

Amazon ECS Agent - v1.26.0 - 2019-02-28 

```bash
docker info
Containers: 27
Running: 8
Paused: 0
Stopped: 19
Images: 10
Server Version: 18.06.3-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.19.25-coreos
Operating System: Container Linux by CoreOS 2065.0.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.66GiB
Name: ip-10-0-13-24
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

[lsutic]

I believe this is caused by the task metadata server binding to localhost inside the container since this commit: https://github.com/aws/amazon-ecs-agent/commit/a3d87cb45ad6ae62191e563336ea70ee1edb7091

Any server socket bound to localhost is not exposed on the bridge network. See for example https://stackoverflow.com/questions/40383804/how-to-expose-a-service-running-inside-a-docker-container-bound-to-localhost

It seems to work if you add --net=host to the startup script, but I'm still checking that solution.

[lsutic]

The docs say to use --net=host.

[lkslawek]

And this arrived while I was writing my longish https://github.com/aws/amazon-ecs-agent/issues/1930 on the same topic.

[adelsjnr]

Great! Adding --net=host in the startup script worked here in v1.26.0 version!
Thanks @lkslawek and @lsutic .

[adnxn]

@thiagoscherrer: are you starting the agent in bridge network mode?

[thiagoscherrer]

@thiagoscherrer: are you starting the agent in bridge network mode?

Not anymore, I'm now using host.

[adnxn]

@thiagoscherrer: gotcha - closing this issue out for now then.

[petderek]

Anyone landing here should check out #1930 -- we will continue discussion there to see if folks have use cases for bridge mode.

[bsoares]

This issue saved me! Tks @adelsjnr for bringing me here! :)