Brave-browser: wayback machine being used when website uses CloudFlare DDoS protection

[kjozwiak]

Description

When visiting a website that has CloudFlares DDoS protection, Brave will display the Wayback Machine until the website completes loading after passing CloudFlares checks. Technically, this isn't a 404 and the website does exists.

Note: I'm not really sure if this is an issue as it recovers and loads the website without any issues. I guess the question is if Brave should be displaying the Wayback Machine when it runs into a website using CloudFlares DDoS protection service.

Steps to Reproduce

1. Install a version of brave that have Wayback Machine (1.4.95 CR: 80.0.3987.122 in this case)
2. load https://www4.mymangalist.org/

Actual result:

Expected result:

Wayback shouldn't be displayed if a website is using CloudFlares DDoS protection or other services that re-routes traffic.

Reproduces how often:

100% reproducible using the above STR.

Brave version (brave://version info)

Brave | 1.4.95 Chromium: 80.0.3987.122 (Official Build) (64-bit)
-- | --
Revision | cf72c4c4f7db75bc3da689cd76513962d31c7b52-refs/branch-heads/3987@{#943}
OS | macOS Version 10.15.3 (Build 19D76)

Version/Channel Information:

• Can you reproduce this issue with the current release? Yes, reproducible
• Can you reproduce this issue with the beta channel? Yes, reproducible
• Can you reproduce this issue with the dev channel? Yes, reproducible
• Can you reproduce this issue with the nightly channel? Yes, reproducible

Other Additional Information:

• Does the issue resolve itself when disabling Brave Shields? N/A
• Does the issue resolve itself when disabling Brave Rewards? N/A
• Is the issue reproducible on the latest version of Chrome? N/A

Miscellaneous Information:

CCing @rebron @brave/legacy_qa

[rebron]

cc: @tomlowenthal @simonhong I don't think we should be showing Wayback machine prompt in this case (503)

[tomlowenthal]

I think that 503 (service unavailable) is generally an HTTP status code for which we should show the Wayback prompt. It's a little odd that Cloudflare serves that response in this case, but the Wayback prompt goes away as soon as Cloudflare's check succeeds and you're directed to the site in question with a 200 response. I think that this is working as intended.

[FozzieHi]

This is a little annoying when visiting a page even if it's temporary. Instead of looking at the HTTP status code, is it possible to look at the 503 as well as the page title? It seems to be static.

Either that or add an option to disable the popup on certain HTTP statuses but then again that's an entirely different issue.