Amplify-console: Bug: Reverse proxy rewrite swallows 404 response codes due to trailing slash redirect

[aethr]

Describe the bug
Our Amplify Vue.js app uses reverse proxy routes configured through the CloudFormation customRules property to proxy API requests through the Amplify app domain. This is partly to avoid CORS configuration on the back-end services.

We have our Amplify Vue.js SPA delivered via the URL:
https://ourservice.com

If we have Service A operating at the URL:
https://service-a.ourservice.com/v0/endpoint

We create a proxy rewrite URL like:
https://ourservice.com/api/service-a/v0/endpoint

Requests from the Vue.js app to this URL do not run afoul of CORS origin issues, and we are able to obfuscate the providers of back-end services. All back-end services use API GW in AWS in the same account, but are not part of the Amplify app.

Some of these back-end services may return 404 responses when we attempt to fetch resources that don't exist. Previously, these 404 responses were returned successfully through the proxy route. Recently, these 404 errors have been getting transformed into 301 redirects by the Amplify proxy route.

To Reproduce
Steps to reproduce the behavior:

1. Create an Amplify proxy route with status: 200 to an external service, like:

{
  "status": "200", 
  "source": "/api/service-a/<*>", 
  "target": "https://service-a.amplifydomain.com/<*>"
}

1. Make a GET request to the proxy route for a resource that doesn't exist:

curl -I https://amplifydomain.com/api/service-a/resource/invalid-resource-id

1. See status code of 301, and location header that adds a trailing slash, but also throws away all other parts of the path:

< HTTP/2 301 
< content-type: application/json
< content-length: 0
< server: CloudFront
< date: Tue, 04 Aug 2020 06:34:29 GMT
< location: /invalid-resource-id/

Expected behavior
Making this same request directly to the back-end service results in a 404 error with a JSON payload describing the error (this is back-end service behaviour). We would expect the reverse proxy rewrite route to return the exact same status code and payload.

$ curl -v 'https://service-a.amplifydomain.com/resource/invalid-resource-id'
< HTTP/2 404 
< date: Tue, 04 Aug 2020 06:51:42 GMT
< content-type: application/vnd.api+json
< content-length: 131
...
{"errors":[{"detail":"Unable to find entity with ID: invalid-resource-id","status":"404","code":"NotFoundErr"}]}

What is Configured?

• Amplify app is configured via CloudFormation template
• App is front-end only, with no back-end Amplify component
• Current app configuration from aws cli:

$ aws amplify list-apps
{
    "apps": [
        {
            ... REDACTED ...
            "customRules": [
                {
                    "status": "200", 
                    "source": "/api/service-a/<*>", 
                    "target": "https://service-a.redacted.com/<*>"
                }, 
                {
                    "status": "200", 
                    "source": "</^[^.]+$|\\.(?!(css|gif|ico|jpg|js|png|txt|svg|woff|ttf|map|json)$)([^.]+$)/>", 
                    "target": "/index.html"
                }
            ], 
            "platform": "WEB", 
            "enableBasicAuth": false, 
            "appId": "d190cn89m1ywxj", 
            ... REDACTED ...
        }
    ]
}

Additional context
We believe this was working correctly until recently. We do not believe that 200 reverse proxy redirects should modify responses in flight, they should deliver them as-is. These API proxy routes should neither have trailing slash "corrected" or responses re-written.

Tickets like this seem to indicate that our desired behaviour did work previously, but that behaviour may have changed: https://github.com/aws-amplify/amplify-js/issues/2775

[kahdojay]

@aethr thanks for the detailed report - can you share your appId (if not d190cn89m1ywxj) and the region?

[aethr]

Hi @kahdojay thanks for looking into this. Confirming the appId is the one from the output above, d190cn89m1ywxj.

Region is ap-southeast-2.

[aethr]

I've been told that this is related to #70 , which has a fix being deployed, and I've been instructed to try re-deploying my app to see if the issue is resolved. I'll report back here once I've had a chance to verify.

[ganipcanot]

Working on a fix which is actively being deployed: https://github.com/aws-amplify/amplify-console/issues/70#issuecomment-699150598.

should be live in eu-west-2 now... slowly rolling through our pipeline to all regions.

For now you can test in eu-west-2. For ap-southeast-2 it will probably be ready by early/mid next week.

[aethr]

@ganipcanot thanks for updating this ticket.

My team member deployed our Amplify app to eu-west-2 this morning but unfortunately we are not seeing a resolution to the 404 responses being mangled and returned as 301s. I'll update this ticket today or tomorrow with some more details of what we tried, but it will essentially be the same as above, just in eu-west-2.

[aethr]

As I mentioned last week, my team member deployed our Amplify app to eu-west-2 last week, appId d1qcmfdxdovis8. We crafted a request that our back-end API will respond with a 404 by requesting a resource with an id that we know doesn't exist.

When we hit the API directly, we get:

$ curl -v 'https://direct.api.endpoint/v0/resource/12345' \
>   -H 'authorization: A_VALID_TOKEN'
< HTTP/2 404 
< date: Tue, 13 Oct 2020 05:28:03 GMT
< content-type: application/vnd.api+json
< content-length: 130
< SOME HEADERS REDACTED
{"errors":[{"detail":"Unable to find entity with ID: 12345","status":"404","code":"NotFoundErr"}]}

When we send the same request to our Amplify proxy route, we get:

curl -v 'https://dev.d1qcmfdxdovis8.amplifyapp.com/api/service/v0/resource/12345' \
>   -H 'authorization: A_VALID_TOKEN'
< HTTP/2 301 
< content-type: application/vnd.api+json
< content-length: 0
< server: CloudFront
< date: Tue, 13 Oct 2020 05:28:09 GMT
< location: /12345/
< x-cache: Error from cloudfront
< SOME HEADERS REDACTED

Both the transformation from 404 to 301 and the mangled location header make it impossible for us to handle the missing resource in a reasonable way in our app.

[thiagoarruda]

I'm also having this issue on sa-east-1, it's completely breaking our error handling, as we have quite a few endpoints that may return a 404. Any ETA on the fix?

Response when hitting the API directly:

< HTTP/1.1 404 Not Found
< Server: Cowboy
< Connection: keep-alive
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 133
< Date: Wed, 14 Oct 2020 13:07:27 GMT
< SOME HEADERS REDACTED

Response when going through the Amplify proxy:

< HTTP/2 301 
< content-type: application/json; charset=utf-8
< content-length: 0
< server: Cowboy
< x-powered-by: Express
< date: Wed, 14 Oct 2020 13:08:11 GMT
< location: /senha/recuperar/
< x-cache: Error from cloudfront
< SOME HEADERS REDACTED

[aethr]

My team would also really appreciate an indication of when this is going to be worked on.

[aethr]

My team would like to tear down the Amplify app we created in eu-west-2 (app id d1qcmfdxdovis8), since this was only created to test the fix in #70 .

If you'd like us to leave it up for your investigation, please let me know before Tues, Oct 20th.

[abhi7cr]

Hello @aethr,

I will be investigating the above app this week, and will get back to you with my findings. Apologies for the delay.

[aethr]

Thanks for the update @abhi7cr , I'll leave the app up for a while, please let me know when you're finished so I can tear it down.

[abhi7cr]

Hi @aethr, @thiagoarruda,

I was able to reproduce the 404s getting converted to 301s with the location header set to the invalid-resource. I have added this to our backlog, and will update here when the fix is in-place.

Thanks for your patience.

[aethr]

Hi @abhi7cr , thank you for the update. I'm going to tear down the d1qcmfdxdovis8 stack if your team doesn't need it anymore.

[abhi7cr]

Hi @aethr, yes we don't need it anymore. You can delete it, thanks!

[thiagoarruda]

Hi @abhi7cr, any updates on this? I found another behavior that seems to be related:

I'm using a rule to redirect most requests to index.html for our SPA to handle, using the following regex with a 200 (Rewrite): </^[^.]+$|\.(?!(css|gif|ico|jpg|js|png|txt|svg|woff|ttf|map|json|webmanifest|xml)$)([^.]+$)/>

With this rule, files with the above extensions that exist on the server can be retrieved, however, if I try to access a file that doesn't exist, say, http://mysite.com/invalid-file.xml, instead of returning a 404, it also gets converted to a 301, which ends up loading the SPA (not the desired behavior).

We caught this because we don't serve a apple-touch-icon-precomposed.png file, so it should return a 404 when requested, but instead it's returning the contents of index.html.

[aethr]

@abhi7cr is there any update on this? The bug has been in production for more than 4 months.

Having a regression go unaddressed (and with absolutely minimal feedback) for so long is making us lose confidence in Amplify as a product.