Containers-roadmap: Fargate: support launching tasks in offline subnets

[copumpkin]

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
For various security/regulatory reasons, we often need to do work in completely "offline" VPCs with no internet connectivity. This means no public IPs, no NATs, no IGWs, or any other path to the internet. Fargate will allow me to try to submit a task to a subnet that can't reach the internet, but then will time out (not a great user experience!), presumably because it can't communicate its status back to the ECS control plane.

Are you currently working around this issue?
Not using Fargate for this sort of work 😦

Additional context
This could be in part connected to #1 because the common case is that someone will want to launch images from ECR into Fargate on a private subnet, but even independently of #1 (e.g., we run a private Docker registry) Fargate containers would ideally maintain some sort of separate pathway to the ECS control plane so that customer-level networking requirements don't prevent the basic service from working (or logging, or reporting statuses).

Somewhat interestingly, if I launch a Fargate task into a private subnet today, I do actually get error messages in the ECS control plane up until the container actually launches. So for example, if I point my task definition at my private registry and put in a bad image ID, the ECS control plane will pass along the Docker error message properly. But once it gets done with early provisioning activity and actually tries to launch the Fargate container, it just sits in the "waiting" state and eventually times out.

[clareliguori]

Hi @copumpkin, using Fargate in an isolated VPC is supported today. If you are seeing "DockerTimeoutError" and you have logging enabled, that is likely caused by the awslogs logging driver not being able to reach CloudWatch Logs. CloudWatch Logs supports PrivateLink, which will enable you to have an isolated VPC but still get container logs in Fargate.

[copumpkin]

@clareliguori wow, that's awesome. Indeed the culprit was awslogs and I just assumed the whole thing was timing out when it was just the logging. Would be nice from a UX standpoint to give a bit more feedback in that sort of situation since I banged my head against this for ages without realizing that it was a simple log driver breaking it 😄

Relatedly, can the awslogs driver configuration get an option for endpoint perhaps? That way we can point it at the specific privatelink endpoint and not have to muck with VPC (or broader)-wide DNS to convince ECS to send logs to the right place?

Either way, thanks for getting me unstuck!

[clareliguori]

Yep agreed this error message DockerTimeoutError: Could not transition to started; timed out is not helpful at all! I'll get that feedback back to the team

[copumpkin]

Thanks! And on the awslogs thing I asked, it turns out https://github.com/moby/moby/commit/998478d369ad3784c3cd28038d836c3c94b16936 added the field I wanted to the driver. Not sure if it's deployed widely yet (and isn't documented), but I'll test it soon.

[mailjunze]

You can run Fargate tasks in completely isolated subnet. With regards to Cloudwatch endpoint issues, If any, You would need to "Enable Private DNS Name" for com.amazonaws.eu-west-1.logs. ( By default "Private DNS" is not enabled for Cloudwatch endpoint)

Go to the Cloudwatch Endpoint in VPC console > Actions > Modify Private DNS name. Also, make sure you're using the latest Fargate platform Version(1.3.0).

Now that ECR endpoints are supported as well, everything(ECR/Cloudwatch) works like a breeze.