Containers-roadmap: [Fargate] [request]: Support for ARM based Tasks

[peterjuras]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

I would like to launch and run ARM based tasks on Fargate.

Which service(s) is this request for?

ECS - Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

ARM based instances are cheaper than other EC2 instance types while still allowing a great number of use cases like web servers. It would be great to have the same cost savings when using ARM with Fargate.

Are you currently working around this issue?

Maintaining your own ECS cluster with ARM based instances.

[janaka]

and EKS

[kcolemangt]

This is blocked because firecracker only runs on Intel. ARM and AMD are in developer preview now. Contribute or follow the progress:

https://github.com/firecracker-microvm/firecracker/milestone/7

https://github.com/firecracker-microvm/firecracker/labels/Feature%3A%20CPU%20Support%3A%20ARM

[akshayram-wolverine]

Hi! We are currently scoping this request. To help better understand your use case, would really appreciate answers/thoughts/opinions on the following questions or other questions if you feel they are pertinent:

1. How do you build your containers? Do developers build them on their laptop or it is done in a CI/CD system?
2. Do you build multi arch images today?
3. What if you switched transparently to ARM behind the scenes? Would you need developers to opt in?
4. If yes, how would you incentivize developers to opt in to ARM (eg. cost)? Would developers need ARM dev workstations to test?
5. What languages do your applications use?
6. What task sizes do you need on ARM?
7. Do you need any storage or network requirements?
8. What partner tools do you use?

[alex]

1. We build in CodeBuild
2. Not yet.
3. We can't swap the architecture for runs without making changes to container builds first.
4. Cost/performance are the obvious ones :-)
5. Go
6. Same as x86, we'd be porting existing systems (though we'd also really like it if there were large instance sizes for fargate)

[ecliptik]

How do you build your containers? Do developers build them on their laptop or it is done in a CI/CD system?

Our images running on EKS are built either using Jenkins or Gitlab CI/CD, usually a combination of docker build, docker tag and docker push.

Do you build multi arch images today?

We use docker buildx to create a multi-arch manifest and then push both architecture images to ECR and then the manifest. Typical workflow follows the steps in this guide: https://aws.amazon.com/blogs/containers/introducing-multi-architecture-container-images-for-amazon-ecr/

What if you switched transparently to ARM behind the scenes? Would you need developers to opt in?

Most likely not, as all of our apps are architecture agnostic, java, python, ruby, although there may be a few architecture specific dependencies we may need to tweak

If yes, how would you incentivize developers to opt in to ARM?

Although when we've discussed it with them they were mostly excited about the cost savings and potential for ARM to out-perform x86.

What languages do your applications use?

java, scala, python, ruby

What task sizes do you need on ARM?

Usually 4-8 CPUs and 8Gb memory, although there are some workloads that could benefit from 32+ CPUs and 64+Gb memory.

Do you need any storage or network requirements?

We run almost everything stateless, and as long as there is enough scratch storage on a pod for some simple temp files it's not really noticed. Network-wise having up to 10GiB would be nice.

What partner tools do you use?

Gitlab is probably the biggest 3rd party tool that would be involved with building multi-arch docker images and running CI/CD jobs on Fargate. See this comment for how we would like to Fargate for running Gitlab jobs and building images: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4358#note_433886014

[sjurtf]

1. Using Gitlab CI with Gitlab runners on EC2 (arm/x86). Images are also built locally on dev machines for local/dev environments.
2. Yes
3. If the provided image to a task definition is multi arch, AWS could, in my option, transparently select the underlying architecture to run that given task on. That said, I would expect ARM runners to cost less than x86_64 runners, similar to what we see today with EC2 Graviton pricing. So to be able to force ARM for cost reasons would be highly appreciated. Making the auto-selection of architecture opt-in during a beta is probably a good idea.
4. Cost savings/performance gains
5. Golang / Java / Python
6. Same as today's configurations. I would also point out these issues (#110 and #79), which I would love to see some development on. They suggest to decouple CPU and RAM configuration and additionally add lower configurations to use for small services. Some would also like to see higher configurations of CPU and RAM.
7. No
8. -

[janaka]

pertinent:
For context the uncertainty is because we are only just starting the process of making the move from ECS to EKS. Moves to Arm would come in that timeline.

1. CI/CD
2. Not yet.
3. Not sure yet. Maybe ideally not.
4. .NET Core / C#
5. Not sure yet
6. Not sure yet
7. Not sure yet

[akshayram-wolverine]

Thanks for the responses and noted on both ECS and EKS support! @janaka @sjurtf @ecliptik What about dev workstations to help developers test and signal if applications are ARM compatible? Is that process easy today? Would you like it simplified?

[pieterza]

I would like Fargate to support ARM based tasks because:

• I'd like to switch our CI/CD worker nodes to use less expensive ARM cores to build/test/publish docker images.
  This allows me to for example use fewer actual instances to achieve the same number of jobs per hour.

• Unless we cross-compile back to x86 (buildx or what have you), we cannot deploy the arm images to Fargate

Thus, we can only switch our (CPU heavy) build nodes (c5 class) to cheaper ARM based instances once the resulting images can run on Fargate.

[janaka]

@akshayram-wolverine I don't have an answer for you on the dev workstation question because we are early on in this journey. Workstations are on x86 still. Hardware will change organically over the coming years as refresh cycles come in.

Our main interest is the price-performance gain for web app workloads. So expecting that to be reflected in Fargate pricing. The code build use case comes hand-in-hand with the web app hosting use case. Hope that helps.

[kapilt]

docker buildx and buildkit, significantly ease multi-arch building from an x86 dev environment, which automates the cross build via qemu

https://github.com/docker/buildx#building-multi-platform-images
https://mirailabs.io/blog/multiarch-docker-with-buildx/

also looking at a cost consideration for ARM fargate, images builds via pipelines (github actions and codebuild) using buildx for cross arch.