Aws-app-mesh-roadmap: Bug: Route is not updated on Envoy when added to a router with no routes.

Created on 14 Sep 2019  路  2Comments  路  Source: aws/aws-app-mesh-roadmap

Describe the bug
Route is not getting updated on Envoy when added to a router with no routes.

This happens to applications that use a backend virtual-service that is associated with virtual-router with no routes.

Platform
ECS (but believe it is platform independent)

To Reproduce
Steps to reproduce the behavior:

  1. Clone https://github.com/kiranmeduri/aws-app-mesh-examples/tree/route-bug-2019-09-13
  2. cd walkthrough/route-bug
  3. Run deploy.sh: This will create a mesh and its parts where color-app's virtual-router has no routes.
  4. Now if you curl the front endpoint you should see 404. This is expected.
  5. Now go ahead and un-comment the L-206 deploy_color_routes
  6. Run deploy.sh: This will add the route for color-app
    ....Wait for some time to confirm eventual distribution ~1min (I waited 10mins)...
  7. Now if you curl front endpoint you will still see 404 when 200 OK is expected
  8. Now if you restart front-app's ECS task you will see that curl will return 200 OK.

Note that if you have an existing app using router with routes and delete all the routes this issue will not reproduce.

Bug Working on it High
Source
picture kiranmeduri
👀1 👍1

Most helpful comment

By reproducing the whole steps in this Github issue for a few times along side with other validations, we didn鈥檛 meet the bug and we confirmed that the bug is fixed now. Below is an example of one of the reproduce processes:

  1. Follow the step 1 - 3 to create a mesh and its parts where color-app's virtual-router has no routes. The outputs related to the mesh are:
deploy infra...
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - route-bug-infra
{
    "mesh": {
        "meshName": "route-bug",
        ...
    }
}
deploy app...
{
    "virtualNode": {
        "meshName": "route-bug",
        ...
        "virtualNodeName": "color-node"
    }
}
{
    "virtualRouter": {
        "meshName": "route-bug",
        ...
        "virtualRouterName": "color-router"
    }
}
{
    "virtualService": {
        "meshName": "route-bug",
        ...
        "virtualServiceName": "color.route-bug.local"
    }
}
{
    "virtualNode": {
        "meshName": "route-bug",
        ...
        "virtualNodeName": "front-node"
    }
}
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - route-bug-app
Public endpoint:
http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color
  1. If we list routes for the router now, we can see that there is no routes for this router:
aws appmesh-preview list-routes --mesh-name route-bug --virtual-router color-router

{
    "routes": []
}
  1. Then follow step 4 we curl the front endpoint we got 404:
curl http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 404</p>
        <p>Message: Not Found.</p>
        <p>Error code explanation: 404 - Nothing matches the given URI.</p>
    </body>
</html>
  1. Then follow step 5 - 6 to update route in deploy.sh and run it again to add the route for color-app:
deploy infra...
Waiting for changeset to be created..
No changes to deploy. Stack route-bug-infra is up to date
deploy app...
{
    "route": {
        "meshName": "route-bug",
        "metadata": {
            "arn": "arn:aws:appmesh-preview:us-west-2:*:mesh/route-bug/virtualRouter/color-router/route/color-route",
            "uid": "746d35d1-e8ad-4801-ab2b-77cd1c56c09f",
            "version": 1
        },
        "routeName": "color-route",
        "spec": {
            "httpRoute": {
                "action": {
                    "weightedTargets": [
                        {
                            "virtualNode": "color-node",
                            "weight": 100
                        }
                    ]
                },
                "match": {
                    "prefix": "/"
                }
            }
        },
        "status": {
            "status": "ACTIVE"
        },
        "virtualRouterName": "color-router"
    }
}
Waiting for changeset to be created..
No changes to deploy. Stack route-bug-app is up to date
Public endpoint:
http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color
  1. After the update success, wait for around 1 minutes. If we list the routes again we can see the new route:
aws appmesh-preview list-routes --mesh-name route-bug --virtual-router color-router

{
    "routes": [
        {
            "arn": "arn:aws:appmesh-preview:us-west-2:*:mesh/route-bug/virtualRouter/color-router/route/color-route",
            "meshName": "route-bug",
            "meshOwner": "370566534788",
            "resourceOwner": "370566534788",
            "routeName": "color-route",
            "virtualRouterName": "color-router"
        }
    ]
}
  1. Then we curl front endpoint again and this time we can get the response green from the color service:
curl http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color

green%

From the results above, we can observe that the route is successfully getting updated on Envoy when we added route to a router with no routes before, and this bug is fixed now.

picture flashyang on 5 May 2020
👍2

All 2 comments

By reproducing the whole steps in this Github issue for a few times along side with other validations, we didn鈥檛 meet the bug and we confirmed that the bug is fixed now. Below is an example of one of the reproduce processes:

  1. Follow the step 1 - 3 to create a mesh and its parts where color-app's virtual-router has no routes. The outputs related to the mesh are:
deploy infra...
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - route-bug-infra
{
    "mesh": {
        "meshName": "route-bug",
        ...
    }
}
deploy app...
{
    "virtualNode": {
        "meshName": "route-bug",
        ...
        "virtualNodeName": "color-node"
    }
}
{
    "virtualRouter": {
        "meshName": "route-bug",
        ...
        "virtualRouterName": "color-router"
    }
}
{
    "virtualService": {
        "meshName": "route-bug",
        ...
        "virtualServiceName": "color.route-bug.local"
    }
}
{
    "virtualNode": {
        "meshName": "route-bug",
        ...
        "virtualNodeName": "front-node"
    }
}
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - route-bug-app
Public endpoint:
http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color
  1. If we list routes for the router now, we can see that there is no routes for this router:
aws appmesh-preview list-routes --mesh-name route-bug --virtual-router color-router

{
    "routes": []
}
  1. Then follow step 4 we curl the front endpoint we got 404:
curl http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 404</p>
        <p>Message: Not Found.</p>
        <p>Error code explanation: 404 - Nothing matches the given URI.</p>
    </body>
</html>
  1. Then follow step 5 - 6 to update route in deploy.sh and run it again to add the route for color-app:
deploy infra...
Waiting for changeset to be created..
No changes to deploy. Stack route-bug-infra is up to date
deploy app...
{
    "route": {
        "meshName": "route-bug",
        "metadata": {
            "arn": "arn:aws:appmesh-preview:us-west-2:*:mesh/route-bug/virtualRouter/color-router/route/color-route",
            "uid": "746d35d1-e8ad-4801-ab2b-77cd1c56c09f",
            "version": 1
        },
        "routeName": "color-route",
        "spec": {
            "httpRoute": {
                "action": {
                    "weightedTargets": [
                        {
                            "virtualNode": "color-node",
                            "weight": 100
                        }
                    ]
                },
                "match": {
                    "prefix": "/"
                }
            }
        },
        "status": {
            "status": "ACTIVE"
        },
        "virtualRouterName": "color-router"
    }
}
Waiting for changeset to be created..
No changes to deploy. Stack route-bug-app is up to date
Public endpoint:
http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color
  1. After the update success, wait for around 1 minutes. If we list the routes again we can see the new route:
aws appmesh-preview list-routes --mesh-name route-bug --virtual-router color-router

{
    "routes": [
        {
            "arn": "arn:aws:appmesh-preview:us-west-2:*:mesh/route-bug/virtualRouter/color-router/route/color-route",
            "meshName": "route-bug",
            "meshOwner": "370566534788",
            "resourceOwner": "370566534788",
            "routeName": "color-route",
            "virtualRouterName": "color-router"
        }
    ]
}
  1. Then we curl front endpoint again and this time we can get the response green from the color service:
curl http://route-Publi-YFQYS65PNL5S-834992577.us-west-2.elb.amazonaws.com/color

green%

From the results above, we can observe that the route is successfully getting updated on Envoy when we added route to a router with no routes before, and this bug is fixed now.

flashyang picture flashyang on 5 May 2020
👍2

thanks for diving into this and closing it out @flashyang

lavignes picture lavignes on 5 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lybrant picture lybrant  路  6Comments
bcelenza picture bcelenza  路  3Comments
dastbe picture dastbe  路  8Comments
jtoberon picture jtoberon  路  5Comments
CharlyF picture CharlyF  路  4Comments