Containers-roadmap: [EKS/fargate] : Support for ISTIO service mesh sidecars

[studees]

We are very keen to migrate our EKS workloads over to the recently released Fargate integration.

Unfortunately it appears as though it is not currently possible to deploy pods that use ISTIO injected sidecars, which seems to significantly limit the potential use of fargate.

The istio-proxy sidecar itself can be injected successfully but we've not managed to apply the subsequent iptables configuration to direct all traffic via the envoy proxy within the sidecar.

Istio supports two mechanisms for this:

1) istio-init container - this fails as under fargate the pod cannot be deployed within the required NET_ADMIN permission
2) istio-cni - this fails as we cannot schedule daemonsets on fargate, and even if we were to manually reconfigure as a sidecar then we cannot mount the required /opt/cni/bin host volume.

Does the migration to firecracker and the associated move of the fargate agent to outside the VM change the security posture for granting pods NET_ADMIN access? This appears to be the most flexible option, but we'd like to request support for network sidecar proxies be provided by whatever means...

Thank you!

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?

Which service(s) is this request for?
This could be Fargate, ECS, EKS, ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

Are you currently working around this issue?
How are you currently solving this problem?

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[ofiliz]

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

[dracmic]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate?
Best regards.

[eightnoteight]

Hello @ofiliz ,

I am new in Fargate stuff ... So ask you directly maybe a stupid question: Can I use istio sidecar injector with Fargate?
Best regards.

hey @dracmic ,
yes, all components of istio which doesn't use istio-proxy are possible to run in fargate, only istio-proxy container needs NET_ADMIN to add the iptable rules

Please note that Fargate already supports App Mesh as a managed solution for both ECS and EKS. Customers don't need to run any init containers or additional CNI plugins to setup any iptables rules, which also means that no privileges (NET_ADMIN etc.) are required.

@ofiliz
it's awesome that app mesh as a managed solution is supported in EKS, but I think the support for istio is very much needed because app mesh doesn't offer as much flexibility as istio does in terms of custom protocols, adding filters and many more things.

fargate as a managed solution solves many of management and reliability problems of self-managing ec2 instances but restrictions like being unable to use istio with it is a huge blocker from adopting fargate in production workloads

[Dhua-v]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

[studees]

the NET_ADMIN is a priviliged user, whom even I wouldnt allow if fargate does. Hence, Istio now provides ISTIO-CNI, we need to use it. https://github.com/istio/istio/tree/master/cni

As far as I'm aware Fargate does not allow us to deploy CNI, or any other daemonset either.

Aside from that, relying on daemonsets for single pod worker nodes will be very inefficient, and exarcebates the existing race conditions of the CNI pod being scheduled & starting before the application pod is scheduled

[Dhua-v]

@studees can we use this cni plugin from fargate https://github.com/aws/aws-app-mesh-roadmap/issues/3
I can raise a pr in istio, but I am not aware of the aws side permissions.

[wmorgan]

This isn't Istio-specific, the same applies to @Linkerd or to any service mesh that does injection via iptables.

[Dhua-v]

Ya we can make atleast istio support locked cloud services

[wmorgan]

@Dhua-v it's not about Istio adding support for Fargate, it's about Fargate adding support for this type of pod network programming, which will allow any service mesh (or other tool) to do proxy sidecar injection.

[Dhua-v]

@wmorgan aws seems to have provided this
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ProxyConfiguration.html

But I am not aws guy, cant say how we can utilize that.