Amazon-ecs-agent: Ports are assumed to be TCP

[owaaa]

I am looking into ECS but after looking a the source code it seems ports are assumed to be /tcp. In Docker you can specify a port as /udp for example -p 8301:8301/udp in the port mapping. Some of my containers require udp exchanges.

[rayh]

We also have a need for this for VoIP/RTP streams

[brian-silberbauer]

I need this too..

[mbrevoort]

+1 Just hit the wall trying to get Consul working on ECS because of this issue.

[bear]

adding my voice to getting this fixed - ECS is a non-starter without UDP IMO

[kretz]

+1

My use case is to use Logstash with UDP input.

[johnae]

+1 need udp

[ViteFalcon]

+1

(Github, roll out the option to +1 issues without having to use comments!)

[jbmcgill]

+1

Use cases are around logging and instrumentation infra.

• rsyslog
• statsd
• snmp

[johnae]

This has been an issue for almost four months, are there plans to implement this? I just really need udp...

[mirstan]

+1 because of statsd.

looks like a huge miss to me in general.

[cbbarclay]

Thanks a lot for the feedback -- seems like a popular request. We're on it.

[justingosan]

+1 because of winston-graylog :)

[ghellings]

+1 How is this labelled an enhancement and not just a bug?

[ViteFalcon]

It's a bug if they say the feature is supported and is not working as expected. Here, the feature doesn't exist and hence isn't a bug.

[WooDzu]

cc @myself Use case is WebRTC and possibly other RTP-related.

[danmayer]

+1 on this, are there workarounds?

[eldos-dl]

+1 need udp soon

[ViteFalcon]

@eldos-dl: This feature is 'Pending Release' as you'll notice in the label that got added 13 days ago.

[danmayer]

My guess is he is aware, like I was when I added a +1. I am looking for it to get released on ECS. Is there a timeline on that?

Not trying to rush or be rude or rush things, just very excited about ECS and UDP is a blocker for my usage.

[samuelkarp]

We have released support for UDP with the ECS Agent v1.2.0 and the AWS CLI/SDK. Please note that the following caveats apply:

• Upgrading an existing Container Instance with a previous version of the Agent to v1.2.0 does not enable UDP. A Container Instance must be initially registered with v1.2.0 to support UDP.
• UDP is not currently visible or settable in the ECS Console, but is visible and settable with the AWS CLI or SDK.

You can start using the ECS Agent v1.2.0 with UDP support by any of the following methods:

• Launching our new ECS-Optimized AMI (amzn-ami-2015.03.c-amazon-ecs-optimized)
• Installing ecs-init 1.2.0-1 on the Amazon Linux AMI (sudo yum install ecs-init)
• Following the instructions for other AMIs

All existing Task Definitions with portMappings defined will now show a protocol of tcp. To define a UDP port in your Task Definition, you can create a portMapping with the protocol of udp as follows:

{
    "hostPort": 53,
    "containerPort": 53,
    "protocol": "udp"
}

To use UDP in the AWS CLI, you will need to update your installation to version 1.7.33. To update your installation through pip, you can run pip install --upgrade awscli.

[eldos-dl]

Is this change reflected in AWS Elastic Beanstalk Multi-container Docker environments?

[samuelkarp]

@eldos-dl: I've asked my colleagues from AWS Elastic Beanstalk and will update the thread when I receive an answer.

[csakoda]

@samuelkarp Any response from your colleagues? I'm observing that multicontainer ElasticBeanstalk instances started with AMI aws-elasticbeanstalk-amzn-2015.03.0.x86_64-ecs-pv-201506120128 are not capable of listening on UDP via Dockerrun.aws.json

i.e., passing a portMapping like

           "portMappings": [
                {
                    "hostPort": 8302,
                    "containerPort": 8302,
                    "protocol": "udp"
                }
            ]

results in docker exposing the tcp 8302 port

[root@ip-10-100-10-141 ecs]# docker ps
CONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                                                                                      NAMES
4865be0754f1        progrium/consul:latest           "/bin/start '-server   49 seconds ago      Up 48 seconds       53/tcp, 53/udp, 8300-8301/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8302->8302/tcp, 8500/tcp   ecs-awseb-l0c8New-env-uwhapph2xw-26-consul-server-e48ea4b2fbbff1d5d701
71f360cc2c2a        amazon/amazon-ecs-agent:latest   "/agent"               13 minutes ago      Up 13 minutes       127.0.0.1:51678->51678/tcp                                                                 ecs-agent

I've already manually upgraded the ecs-agent running on the system:

[root@ip-10-100-10-141 ecs]# docker stop ecs-agent
[root@ip-10-100-10-141 ecs]# docker rm ecs-agent
[root@ip-10-100-10-141 ecs]# docker pull amazon/amazon-ecs-agent:latest
[root@ip-10-100-10-141 ecs]# initctl start ecs
[root@ip-10-100-10-141 ecs]# docker run -it -rm amazon/amazon-ecs-agent
Warning: '-rm' is deprecated, it will be replaced by '--rm' soon. See usage.
2015-07-10T21:22:59Z [INFO] Starting Agent: Amazon ECS Agent - v1.3.0 (097e4af)

Am I doing something wrong or is this just not going to work yet?

[samuelkarp]

@csakoda: My colleagues have confirmed that UDP is not supported in AWS Elastic Beanstalk. I will continue to update this thread as I become aware of more information.

As an aside (since you included some output here), you can check the version of your currently-running ECS Agent either through our ECS API (DescribeContainerInstances and look at the versionInfo) or through the Agent's introspection API on the host (curl localhost:51678/v1/metadata).

[sebest]

@samuelkarp
Hello,

Do you know if there is any update on the support for UDP by AWS Elastic Beanstalk multi container?

thanx

[ChrisRut]

It seems CloudFormation hasn't caught up to this change: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-portmappings.html
Notice the lack of a 'protocol' setting.

Would anyone from Amazon happen to know what the ETA is for this?

[samuelkarp]

@sebest: My colleagues have confirmed that UDP is not supported in AWS Elastic Beanstalk.
@ChrisRut: I've asked my colleagues from AWS CloudFormation and will update the thread when I receive an answer.

[samuelkarp]

@ChrisRut: My colleagues have confirmed that UDP is not supported in AWS CloudFormation. I will continue to update this thread as I become aware of more information.

[ChrisRut]

@samuelkarp , Thanks for the update, to be clear do they mean "it's not supported and won't ever be supported" or "it's not supported, and they are working on adding support" ?

[ChrisRut]

@samuelkarp , any updates?

[adinapoli]

This would be of use to us as well. We are in the process of migrating our existing infrastructure to Elastic Beanstalk but we hit a wall when it comes to use the official loggly syslog image:

https://www.loggly.com/docs/docker-syslog/

It needs to bind on port 514/udp, so we are out of luck.

[sebest]

@adinapoli This is not really an issue if you only need to access this syslog server from the containers that are on the same host.

You don't need to "expose" the port, just use the "links" feature:
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

Eg: name your loggly syslog container "syslog", in the containers that need to access syslog add "links": ["syslog"] and now you can access syslog from these containers using the "syslog" hostname.

you can read more about this feature here: http://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/

[adinapoli]

Hey @sebest , I have follow the "official" loggly guide:

https://www.loggly.com/docs/docker-syslog/

Although they links the containers together, they also specifically expose port 514/udp, which is impossible to do for me due to the aforementioned limitation.

[sebest]

@adinapoli , this is a common mistake: you do not need to expose a port if you only need to access it from docker container on the same host. Exposing the port is only useful if you want to be able to access this port from an external host.

See "links" as internal DNS on a private LAN and "expose" as the port forwarding on your firewall to expose a specific host from your private LAN to the internet.

[jessesanford]

Is there any externally available ticket or issue tracking for UDP being supported by beanstalk? It's been many months. Is it supported yet?

[sebest]

@therealjessesanford from what i know elastic beanstalk only support HTTP(S) externally , even TCP is not supported, so i doubt EB will ever support UDP especially as ELB only supports TCP.

[developerinlondon]

any updates on this for cloud formation yet? http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-portmappings.html

[ChrisRut]

@developerinlondon , Last I heard @samuelkarp was going to check with the CloudFormation team, Any updates @samuelkarp ?

[samuelkarp]

@sebest @adinapoli @jessesanford My colleagues have informed me that UDP should work in AWS Elastic Beanstalk now.

I do not have an update on AWS CloudFormation right now.

[adinapoli]

@samuelkarp Hey Samuel, thanks for the info! I don't know if was an happy coincidence, the result of the aforementioned support for UDP and whatnot, but I was able to get the loggly docker container working on EBS (with linking as well) by simply following their tutorial and not specifying anything explicitly when it came to exposing ports.

In other terms, the tutorial simply worked out of the box, whereas before it did not. In any case, I'm an happy man!

[ChrisRut]

@samuelkarp any updates on CloudFormation adding support for UDP ports yet?
We are approaching 1 yr since this was implemented, and still no CloudFormation support.

[andreausu]

@samuelkarp I was wondering that too, not having it in CloudFormation is very inconvenient.

[ChrisRut]

We gave up and migrated away from CloudFormation for the provisioning of ECS resources and are now making direct boto3 calls.

[CpuID]

@ChrisRut incase your unware of it - https://www.terraform.io/docs/providers/aws/index.html (ctrl+f ECS) :)

[simonbrady]

@ChrisRut @andreausu I asked about this in the CloudFormation forum and was told there's an open feature request for this: https://forums.aws.amazon.com/message.jspa?messageID=720619#720619

[ChrisRut]

It looks like the CloudFormation team finally updated to support Protocol: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions-portmappings.html

[brianantonelli]

+1 could really use this support in Dockerrun.aws.json for EB/ECS..

[samuelkarp]

@brianantonelli UDP ports should be supported in Elastic Beanstalk Multi-Container. If you are experiencing problems with your Dockerrun.aws.json, you can get help through the Elastic Beanstalk forum or through AWS Support.