Aws-app-mesh-roadmap: Bug: Delayed Envoy Configuration Update When Removing a Route

[AKBarcenas]

Summary
When deleting a route in App Mesh there is a possibility of the Envoy remaining out of sync with this change for more than 10 minutes in some observed cases. The issue here seems to be tied with the order in which the route was created with respect to other routes. Specifically, if the route being deleted is not the newest route that has been created then there will be some delay between the route deletion and the Envoy receiving an update of that change. An example here would be creating routes 1, 2, 3 starting with the lowest number first. In this case, deleting route 3 would not result in a delay, but deleting either route 1 or 2 would result in a delay.

Steps to Reproduce
This specifically is an EKS example, but this can be reproduced outside of using EKS.

1. Deploy the following template in EKS: https://github.com/aws/aws-app-mesh-examples/blob/master/walkthroughs/howto-k8s-http-headers/v1beta2/manifest.yaml.template

2. Delete the color-route-blue route from the template

3. Verify that the route has been deleted for virtual router color_howto-k8s-http-headers via either the App Mesh console or AWS CLI.

4. Exec into the front pod and check the Envoy config, you will see the route for blue virtual node will still be there.

        "routes": [
         {
          "match": {
           "prefix": "/",
           "headers": [
            {
             "name": "color_header",
             "exact_match": "blue"
            }
           ]
          },
          "route": {
           "weighted_clusters": {
            "clusters": [
             {
              "name": "cds_egress_howto-k8s-http-headers_blue_howto-k8s-http-headers_http_8080",
              "weight": 1
             }
            ],
            "total_weight": 1
           }
          }
         },

Are you currently working around this issue?
A workaround for this problem is to force the route being deleted to be the latest route by recreating the route. For example, if the route color-route-blue from above was recreated and deleted again then the Envoy should receive an update within the expected time frame (5 min)

[dastbe]

We're working on a general fix for scenarios where certain updates (specifically, resource deletes) are not distributed to Envoys in a timely fashion.

[dastbe]

As an update, we're now rolling out a fix for this issue to all regions.

[dastbe]

As a (final) update, we've rolled out this fix worldwide! delete things to your heart's content.