Aws-app-mesh-roadmap: [request]: Provide support for Envoy's zone aware routing

[rlafferty]

Tell us about your request
Support the ability to use envoy's zone aware routing.

Which integration(s) is this request for?
App Mesh and potentially Cloud Map

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
It would be beneficial to be able to use Envoy's zone aware routing to "preference" backends in the same AZ. This would reduce cross-AZ latency where possible. I believe the expectation is that the zone information would be provided by the service discovery tool - so in this case, Im unsure if Cloud Map already provides/retains that information.

Are you currently working around this issue?
Not currently able to, so going "without it"

[lavignes]

Hi @rlafferty. Thanks for opening this issue. This is definitely a worth-while feature to add.
I know today that ECS will publish Cloud Map attributes such as AVAILABILITY_ZONE:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

So hypothetically today, you could create a virtual nodes for each AZ that matches on the AVAILABILITY_ZONE Cloud Map attribute as a work-around. But that is obviously not ideal.

We could specify the AZ by querying the EC2 metadata endpoint and pass it along to Envoy's bootstrap locality: https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/base.proto#envoy-api-msg-core-node

Then we can query for zonal instances as an opt-in parameter in VirtualNode service discovery.. Not sure exactly how this should look yet, but we can start thinking about it.

[joeykhashab]

@lavignes I am currently experimenting with the work around that you were suggesting. I am still a little unsure about how the zone-specific logic would work, when I setup my zone specific virtual nodes I still need to setup zone specific virtual services, virtual routers, and routes to them correct?

Once I do that, I guess I need to create a custom envoy docker image to handle the logic of "if in zone A, route the request using the zone A virtual router" right?

[lavignes]

Hi @joeykhashab

I still need to setup zone specific virtual services, virtual routers, and routes to them correct?

For now, yes unfortunately. Hence why this issue exists :(

In the route, you could specify two virtual nodes weighted targets: one with an AVAILABILITY_ZONE Cloud Map attribute and another without. Then you could weight the traffic primarily towards the first to gain some fail-over capabilities if the preferred AZ is seeing an outage.

Once I do that, I guess I need to create a custom envoy docker image to handle the logic of "if in zone A, route the request using the zone A virtual router" right?

That would not be necessary. You'd simply configure the backends of your virtual nodes to use the zone-specific virtual service. The Envoy image contains just some information about reaching our management server to download configuration. So you aren't able to target any specific virtual routers etc. It is all fetched at run-time.

All that said, I wouldn't recommend doing this manually. It would require creating a lot of duplicate mesh resources. My example above is more hypothetical as it does allow for routing to a specific AZ via service discovery, but there is no nice mechanism for handling fail-over.

[joeykhashab]

I see. OK I think I am understanding the work around better now. Thanks for your response @lavignes.

[claydanford]

Any update on this after 8 months?

[inakianduaga]

How realistic is it that this will be implemented in the next 6 to 12 months? We are currently considering AppMesh to potentially remove our ALB LCU costs by dropping ALBs for all internal traffic, but it feels that won't help if we get hit with CrossAZ costs for the traffic

[Rob-Johnson]

just adding a +1 for wanting to see this functionality. From what I can see, the method involving cloudmap attributes won't provide the same capabilities that envoy has natively, particularly making sure that the amount of traffic sent within the same az is kept proportionate with the capacity of the upstream cluster.