Amazon-ecs-agent: --net=host support

[ChrisRut]

Dockers default behavior is to give a container a private network stack and to build a bridge with the host's network interface. This has considerable performance drawbacks (see netperf in here ) and is an absolute nogo for some types of applications (including mine). Is host-native interface (--net=host) supported or will it ever be supported?

Copied from: https://forums.aws.amazon.com/thread.jspa?threadID=177850&tstart=25

There are a couple of forks out there that attempt to provide support, for example: https://github.com/chaos-generator/amazon-ecs-agent

But it would be great if there were official support for this in the AWS repo. Not having this really precludes ECS from performance sensitive applications.

[oli-g]

+1 here. It seems that I cannot run an ElasticSearch cluster without that option.

[rthrfrd]

@oli-g In the case of ES a workaround is to use the --network.publish_host option when running elasticsearch and set it to the host IP/hostname. To get the host IP I mount a directory into every container which has the instance's IP written to a file (this is done on the host on instance launch).

If you're running a single ES node per host this works well - ES will use the host IP instead of container IP to discover nodes. That restriction does mean it may not be sufficient in more complex scenarios though.

[CpuID]

+1, would save me messing around with -ip when running registrator - http://gliderlabs.com/registrator/latest/user/run/

[mattcallanan]

@rthrfrd could you please elaborate on how you do this?

To get the host IP I mount a directory into every container which has the instance's IP written to a file (this is done on the host on instance launch).

Sounds like it could also be a good workaround for issue https://github.com/aws/amazon-ecs-agent/issues/3

[rthrfrd]

@mattcallanan In a script in User Data of EC2 instance launch configuration have something like:

#!/bin/bash
mkdir /usr/share/docker
echo "${HOSTNAME}" > /usr/share/docker/hostname
curl http://169.254.169.254/latest/meta-data/local-ipv4 > /usr/share/docker/ip
echo "${CLUSTER_NAME}" > /usr/share/docker/ecs-cluster

Configure your ECS task definition to always mount (read only) that dir as a volume and you can read the relevant info from each file in the container.

[mattcallanan]

Thanks @rthrfrd - that worked well.

Also realised we could actually use DOCKER_HOSTNAME=$(curl http://169.254.169.254/latest/meta-data/local-hostname) from within the container itself. Thanks.

[pditommaso]

+1. Same problem here.

[schuylr]

+1 this would make it easy to set up RDS access on my containers

[Lakret]

+1 will be much easier to deploy Akka Clusters

[ChrisRut]

DataStax released a white paper on running DSE Cassandra in a container;

Given DSE is a distributed database; it is highly recommended that only one DSE node be
configured to run per Docker host to achieve optimal networking configuration. To do so,
use the --net=host option.
...
Docker’s default networking (via Linux bridge) is not recommended for the production use as it slows
down networking considerably. Instead, use the host networking (docker run --net=host) or a plugin
that can manage IP ranges across clusters of hosts. The host networking limits the number of DSE nodes
per a Docker host to one, but this is the recommended configuration to use in production.

• http://www.datastax.com/wp-content/uploads/resources/DataStax-WP-Best_Practices_Running_DSE_Within_Docker.pdf

[luhkevin]

Hi, I need this as well!
+1 for --net=host support, would be really useful!

[egorpushkin]

+1 Pretty much any custom API application will leverage from that in addition to making ECS more attractive for hosting ES/Mongo/Cassandra clusters.

[flrngel]

:+1: really need this

[errordeveloper]

:+1: as well as --pid=host

[amanzyuk]

+1

[rhardih]

+1

[bryantidd]

This needs to be a priority!

[nhuray]

+1 I really need this !

[hiloboy0119]

+1 for ease of deploying an akka cluster as well as performance for network heavy apps

[raimille1]

+1

[RodolfoARodrigues]

+1

[valter-lisboa-credibilit]

+1

[ebirukov]

+1

[bobby]

+1, needed for Zookeeper 3.5.1-alpha with dynamic reconfiguration.

[renanvicente]

+1

[bryceray1121]

+1

[wieczorek1990]

+1

[pedrorjbr]

+1

[pedrorjbr]

Does anynone try to use a ambassador container?

[burdandrei]

+1

[mixja]

+1

[cokline]

+1

[sofam]

+1 from here.

[danbeaulieu]

For those waiting around for this feature, it might be helpful to keep in mind you have full write access to the AMI, which is where you'd set the --net host parameter (not sure where it is on this AMI exactly but my guess is it is in the init system script for docker, in my experience most people bake their own AMI and so control how docker is started. My impression is the AMI is a POC).

The agent has nothing to do with the docker daemon configuration, so I wouldn't be too surprised if this issue is closed eventually.

[hatt]

@danbeaulieu I may be misunderstanding you but I believe this issue is for setting the --net flag when running containers, not the Docker daemon being on TCP or Unix sockets. Containers can be networked via the bridge interface by default, or with full host passthrough and control over the interface. This allows some services like Elasticsearch or Consul direct access to the interfaces they need to bind on. It's vaguely analogous to NAT vs host networking in VirtualBox.

[danbeaulieu]

@hatt yup you are right. I had some concepts confused.

In the mean time I have had success cloning this repo and making changes to the agent to do things it currently didn't do and then building and deploying that version in my cluster. The agent is pretty hackable.

[dilev01]

+1

[ankitcha]

+1

needed for Apache Flink installation which is based on Akka behind the scenes. It's been a smooth ride with ECS except this limitation. It would be really great if this could be resolved.

Thanks!

[robertjustjones]

+1

[ghost]

👍

[3lackraven]

Needed for an Akka based application

[forkfork]

+1

[3lackraven]

Hoping it's useful for someone:

I have "solved" with a workaround: I have installed a little 'discoverip' service (in my case jetty + a single line jsp) that return the remote ip address of an http request. Than in akka I set the hostname programmatically with the result of the service call

val result = ConfigFactory.empty()
...
.withValue("akka.remote.netty.tcp.hostname", ConfigValueFactory.fromAnyRef(Source.fromURL("http://mylittleservice:8080/discoverip")))
...

the jsp consist in the following trivial code:

<%= request.getRemoteAddr() %>

Of course this solution has the external service dependency drawback ... but it's 'quick and dirty' working.

[ChrisRut]

@3lackraven ,

That doesn't "solve" this issue. This issue is related to the performance of the userland proxy that Docker uses for it's networking by default.

Also, wouldn't it just be easier to get the remote IP from ec2's metadata:

PUBLIC_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)

[hatt]

Getting the IP is easy, but this request is for host device access, not the ability to check an interface's assigned IP address. That's just a side effect of it.

[valter-lisboa-credibilit]

Obrigado pelo retorno Samuel,

Entendi como proceder.

Galante, ok para você?

Atenciosamente

Enviado do Outlook Mobilehttps://aka.ms/blhgte

On Mon, Apr 18, 2016 at 4:19 PM -0700, "Matt Sharpe" <[email protected][e-mail]> wrote:

Getting the IP is easy, but this request is for host device access, not the ability to check an interface's assigned IP address. That's just a side effect of it.

You are receiving this because you commented.
Reply to this email directly or view it on GitHubhttps://github.com/aws/amazon-ecs-agent/issues/185#issuecomment-211627369

---

INFORMAÇÕES AOS DESTINATÁRIOS: As informações contidas e as anexadas a esta comunicação podem ser confidenciais, legalmente privilegiadas, ou ter de outra forma protegida a sua divulgação, sendo exclusivamente para o uso do(s) seu(s) destinatário(s). Se você não for o destinatário previsto desta comunicação, queira por gentileza excluir e destruir todas as cópias em seu poder, notifique o remetente que você recebeu esta comunicação por engano e esteja ciente de que a leitura ou a divulgação bem como a adoção de qualquer ação baseada nesta comunicação estão expressamente proibidas.
As mensagens de e-mail podem conter vírus ou outros defeitos, podem não ser reproduzidas fielmente em outros sistemas, ou podem ser interceptadas, excluídas ou interferidas sem o conhecimento do remetente ou do destinatário. A Credibilit não assume nenhuma responsabilidade em relação a estas ocorrências. Esteja ciente de que a Credibilit reserva-se o direito de interceptar, monitorar e reter mensagens de e-mail destinadas aos seus sistemas ou neles originadas, na medida do permitido pela legislação aplicável. Se você não se sente seguro com os riscos associados às mensagens de e-mail, poderá optar por não usar e-mail nas comunicações com a Credibilit.

---

INFORMAÇÕES AOS DESTINATÁRIOS: As informações contidas e as anexadas a esta comunicação podem ser confidenciais, legalmente privilegiadas, ou ter de outra forma protegida a sua divulgação, sendo exclusivamente para o uso do(s) seu(s) destinatário(s). Se você não for o destinatário previsto desta comunicação, queira por gentileza excluir e destruir todas as cópias em seu poder, notifique o remetente que você recebeu esta comunicação por engano e esteja ciente de que a leitura ou a divulgação bem como a adoção de qualquer ação baseada nesta comunicação estão expressamente proibidas.
As mensagens de e-mail podem conter vírus ou outros defeitos, podem não ser reproduzidas fielmente em outros sistemas, ou podem ser interceptadas, excluídas ou interferidas sem o conhecimento do remetente ou do destinatário. A Credibilit não assume nenhuma responsabilidade em relação a estas ocorrências. Esteja ciente de que a Credibilit reserva-se o direito de interceptar, monitorar e reter mensagens de e-mail destinadas aos seus sistemas ou neles originadas, na medida do permitido pela legislação aplicável. Se você não se sente seguro com os riscos associados às mensagens de e-mail, poderá optar por não usar e-mail nas comunicações com a Credibilit.

[3lackraven]

@ChrisRut
Well, yes, this doesn't solve any docker host device access issue.
My intention was to share a quick and dirty workaround for running a dockerized akka application (while waiting for a more proper solution).
And yes, yours is definitely a better solution. Thank you.

[bilalyasar]

+1

[errordeveloper]

Some of the comments here can be solved by using Weave Net for service
discovery, which works well for Akka and any legacy apps.

On Wed, 27 Apr 2016, 14:19 Bilal Yasar, [email protected] wrote:

+1

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/aws/amazon-ecs-agent/issues/185#issuecomment-215080247

[errordeveloper]

Here is the guide for deploying Weave Net in ECS:
https://www.weave.works/deploy-weave-aws-cloudformation-template/

On Wed, 27 Apr 2016, 19:13 Ilya Dmitrichenko, [email protected]
wrote:

Some of the comments here can be solved by using Weave Net for service
discovery, which works well for Akka and any legacy apps.

On Wed, 27 Apr 2016, 14:19 Bilal Yasar, [email protected] wrote:

+1

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/aws/amazon-ecs-agent/issues/185#issuecomment-215080247

[s7anley]

Yet another workaround for some of the cases mentioned here. If you need port of container, you can mount docker socket as volume and get it from Docker Remote API.

AWS_PRIVATE_IP=`curl http://169.254.169.254/latest/meta-data/local-ipv4`
CONTAINER_ID=$(cat /proc/self/cgroup | grep "docker" | sed s/\\//\\n/g | tail -1)
HOST_PORT=$(curl --unix-socket /var/run/docker.sock http:/containers/$CONTAINER_ID/json | jq -r '.NetworkSettings.Ports."26379/tcp"[0].HostPort')

[ChrisRut]

@errordeveloper & @s7anley ,
All these work-arounds don't solve the original problem that this issue was created for. They assume you are willing to live with the networking penalty that the userland proxy buys you. I think Hitler sums it up best:
"Docker is supposed to have better performance. Yet that **ing userland proxy is slower than a 28.8k modem"

• https://www.youtube.com/watch?v=PivpCKEiQOQ&t=3m5s

[xbglowx]

+1

[wdalmut]

+1

[ricoli]

this is very much needed. alongside --pid=host, for monitoring tools as well, such as new relic

[gaborho]

+1
It is recommended to run Consul in container with --net=host, as explained in Consul's official Docker image description:

Consul should always be run with --net=host in Docker because Consul's consensus and gossip protocols are sensitive to delays and packet loss, so the extra layers involved with other networking types are usually undesirable and unnecessary.

[wjimenez5271]

+1

[ryansch]

For all the +1s: I'm running consul on RancherOS as a system service under system-docker. I get to run it with --net=host and I still get to use ECS with the userland docker.

Bonus: userland docker has access to consul on startup so it can be used for overlay networking

[elhachmi]

+1

[rupamroy]

+1, Please let us know if there is any traction on supporting this.

[igrayson]

A workaround, for the consul image, without customizing the image.

consul client container:

$ ip_address=`ifconfig eth0 | grep -oP 'inet addr:\K\S+'`
$ docker run --rm -it \
  -e 'CONSUL_CLIENT_INTERFACE=eth0' \
  --name consul \
  consul agent -advertise="$ip_address" -retry-join=yourserver

some container which consumes your consul client:

$ docker run -it --rm --link consul:consul \
  -e 'CONSUL_RPC_ADDR=consul:8400' \
  consul consul members

Obviously, this goes against hashicorp's recommendation to use net=host.

[CpuID]

Any further updates on this issue? I would prefer to go with net=host for consul as per the HashiCorp guidelines, but cannot do so under ECS. cc @samuelkarp

[bpoweski]

@samuelkarp Any chance of this happening soon-ish?

[veqryn]

This and I also need --pid=host support...

[aaithal]

Hello,

We plan to support host networking mode for containers in ECS by adding the networkMode field in the task definition. networkMode can be set to one of the following values:

• bridge (default)
• host
• none

Note that port mapping behavior differs for each of these networking modes (discussed in more detail below).

bridge

If networkMode is not specified in the task definition (current behavior), or if the networkMode is specified as bridge, a network stack is created for each container in the task, on the default Docker bridge.

Port Mapping

In the bridge networking mode, container ports can be mapped to host ports as
documented here.

host

All of the containers in the task will share the network stack of the host in this mode.

Port Mapping

The task definition will require all host ports to match container ports in the host networking mode. If the host port is not specified, the host port will be same as container port in port mappings.

none

Containers in the task will not have access to any external routes in this mode.

Port Mapping

We do not allow any port mappings in the task definition when networkModeis set to none.

Task Definition Snippets

1. networkMode=host with container and host port mappings (host port and

container port are set to 80)

{
        "family": "awesome-app",
        "networkMode" : "host"
        "containerDefinitions": [
            {
                "name": "...",
                "image": "...",
                "portMappings": [
                  {
                    "containerPort": 80,
                    "hostPort": 80
                  },
                  ...
                ],
                ...
            },
            ...
        ]
}

2. networkMode=host with no host port mappings (host port is assumed to be

80)

{
        "family": "awesome-app",
        "networkMode" : "host"
        "containerDefinitions": [
            {
                "name": "...",
                "image": "...",
                "portMappings": [
                  {
                    "containerPort": 80,
                  },
                  ...
                ],
                ...
            },
            ...
        ]
}

3. networkMode=none (no port mapping)

{
        "family": "awesome-app",
        "networkMode" : "none"
        "containerDefinitions": [
            {
                "name": "...",
                "image": "...",
                "portMappings": [],
                ...
            },
            ...
        ]
}

[jayadatta]

Thanks Anirudh. Could you please let us know if there is any timeline by which we can expect this mode?

[reedflinch]

thanks, @aaithal. Any chance PidMode (--pid=host) will be included soon?

[oli-g]

Hi @aaithal, do you have an ETA for this change? We're really looking forward to it!

[dimahavrylevych]

+1 to @oli-g

[errordeveloper]

@aaithal this is great to hear! We also really need --pid=host...

[halfmatthalfcat]

@aaithal any movement on this? ETA?

[aaithal]

@jayadatta @oli-g @dimahavrylevych @halojedi20 Unfortunately, we cannot comment on the delivery date for this feature. However, please note that we have a PR (#468) for integrating our recently released Task IAM Roles with host networking mode. Whenever that is merged to the master branch, this feature should be available to you.

@reedflinch @errordeveloper Could you please create a new issue for --pid=host as we would like to keep this thread focused on --net=host? We would also appreciate if you could provide more details about your use-cases as well.

Thanks,
Anirudh

[aaithal]

Hi All,

Closing this issue as you can now launch containers in ECS with host networking mode as per this announcement. Docs for the same can be found here. Please reach out to us for any questions/comments/clarifications.

Thanks,
Anirudh

[errordeveloper]

@aaithal there is #473, created by @marinusd.

[bertildaniel]

@aaithal we are trying to get this to work and seeing a number of issues, one seems which I don't think is related to ecs-agent (the amazon UI (and potentially API) does not allow opening them same port for both tcp/udp, it insist on there only being on definition and it being either tcp/udp.)

The issue that does seem to be related to ecs-agent is that we are seeing on a task definition that works/worked fine in bridge networking, it has two container definitions and the second container is failing with CannotCreateContainerError: Container already exists.. Stays the same if I remove the old stopped container. Is there an undocumented limitation that host networking only supports a single container within the task definition? (I'm a bit unclear on why host networking is defined for the entire task and not on a container level)

Also this thread is the only place where the networkMode keyword is actually documentation, it's not in your official docs (yet).

[aaithal]

@bertildaniel thank you for reporting this.

Also this thread is the only place where the networkMode keyword is actually documentation, it's not in your official docs (yet).

As per my earlier response, networkMode is documented in our docs here.

The issue that does seem to be related to ecs-agent is that we are seeing on a task definition that works/worked fine in bridge networking

Could you please create a new issue for this with more details including a sample task definition that's failing for you and the error logs from the Agent for the same? That would really help us in debugging the issue.

Thanks,
Anirudh

[bertildaniel]

@aaithal If you go to that page, nowhere does it say what the keyword is it just talks generally about network mode

I will create another issue for it and get some logs for you, we have currently progressed with the workaround of splitting the task definition into two services which is working fine. Also AWS support has now confirmed the bugs that are in the ECS web interface.

[aaithal]

@bertildaniel thanks for pointing that out. The register-task-definition API documentation does seem to contain the keyword that you're looking for here. We'll also work on getting the keyword into the service documentation as well.

I will create another issue for it and get some logs for you, we have currently progressed with the workaround of splitting the task definition into two services which is working fine.

We're glad that you're unblocked in the interim. Please let us know by creating a new issue if/when you run into this again.

Thanks!

[Robbert1]

Not sure this is the place for it but it would be great if the Network Mode option would also be made available through CloudFormation version of the API for managing task definitions:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ecs-taskdefinition.html#cfn-ecs-taskdefinition-containerdefinition

[mixja]

@Robbert1 +1 - although it took ~6 months for ECS Service Deployment thresholds to become configurable in CF and even longer to support UDP port mappings, so don't hold your breath.

[jacohend]

+1

"CannotCreateContainer: container already exists" error for registrator and haproxy in host networking mode.

[ryansch]

@Robbert1 If you're still waiting for CF, could I suggest you look at terraform?

https://www.terraform.io/docs/providers/aws/r/ecs_task_definition.html#network_mode

[b-jensen]

+1 for "CannotCreateContainer: container already exists" error. This is a big problem, and the "separate services" workaround won't work for us.

[aaithal]

@b-jensen Can you please create a new issue with more details of the problem you're running into so that it can be tracked properly as the --net=host issue for this thread has already been resolved?

Thanks,
Anirudh

[b-jensen]

For those who may have found this issue while searching for the answer to the "CannotCreateContainerError: container already exis" problem, I filed an AWS Support ticket and discovered that the problem was that my task definition JSON specified both networkMode=host and links. Removing the links from the task definition JSON solved the problem. The error, "container already exis" is misleading at best.

[richardpen]

@b-jensen Thanks for pointing this out, we are already working on fixing the misleading error issue. Will let you know when we have an update.

[humayunjamal]

@aaithal

we are trying to get this to work and seeing a number of issues, one seems which I don't think is related to ecs-agent (the amazon UI (and potentially API) does not allow opening them same port for both tcp/udp, it insist on there only being on definition and it being either tcp/udp.)

The issue that does seem to be related to ecs-agent is that we are seeing on a task definition that works/worked fine in bridge networking, it has two container definitions and the second container is failing with CannotCreateContainerError: Container already exists.. Stays the same if I remove the old stopped container. Is there an undocumented limitation that host networking only supports a single container within the task definition? (I'm a bit unclear on why host networking is defined for the entire task and not on a container level)

Also this thread is the only place where the networkMode keyword is actually documentation, it's not in your official docs (yet).

+1

• we removed the links option as well but still same error

[b-jensen]

@humayunjamal , in host networking mode make sure that your task definition does not contain any links. The links are critical in bridge mode, but will kill you in host mode.

[richardpen]

@humayunjamal IIUC, there are two issues here:

• The host networking mode can't be combined with link option as per docker.
• The error returned "CannotCreateContainerError: Container already exists" is a little confusing, it may be option conflict. We are working on a fix which may need to create a PR to docker.

Could you provide the task definition here or send it to me at penyin (at) amazon.com so that we can figure out why it doesn't work even without link option in host networking mode?

Thanks,
Peng

[mmaring-handy]

any update here, I am having the same exact issue and its causing me many problems

[JDvorak]

+1

[marcoocram]

I'm having this same issue, with networkmode host and links message "container already exists" and without links, the containers can't connect between....any ideas?