Magento2: Customer account link broken after crawl

[monotheist]

Preconditions (*)

1. Magento 2.3 with cache and FPC enabled

Steps to reproduce (*)

1. Run site check in woorank.com like below.
2. https://www.woorank.com/en/www/pr-20241.instances.magento-community.engineering

Expected result (*)

1. Below link should not throw 404 page not found
2. https://pr-20241.instances.magento-community.engineering/customer/account/create/

Actual result (*)

1. https://pr-20241.instances.magento-community.engineering/customer/account/create/
2. Above link is broken. See screenshot below
   

[magento-engcom-team]

Hi @ethniccode. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• [ ] Summary of the issue
• [ ] Information on your environment
• [ ] Steps to reproduce
• [ ] Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me $VERSION instance

where $VERSION is version tags (starting from 2.2.0+) or develop branches (for example: 2.3-develop).
For more details, please, review the Magento Contributor Assistant documentation.

@ethniccode do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• [ ] yes
• [ ] no

[monotheist]

@magento-engcom-team give me 2.3 instance

[magento-engcom-team]

Hi @shikhamis11. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [x] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [x] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 6. Add label Issue: Confirmed once verification is complete.

• [ ] 7. Make sure that automatic system confirms that report has been added to the backlog.

[shikhamis11]

@magento-engcom-team give me 2.3-develop instance

[magento-engcom-team]

Hi @shikhamis11. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @shikhamis11, here is your Magento instance.
Admin access: https://i-20255-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[magento-engcom-team]

@shikhamis11 Thank you for verifying the issue. Based on the provided information internal tickets MAGETWO-97606 were created

[shikhamis11]

@ethniccode I did the same you explained and it actually showing 404 on account creation page.

[monotheist]

@shikhamis11 This issue appears on my production system overnight when Google crawls. I was struggling to reproduce this issue. Looks like any crawler can be used to reproduce this issue. Not only account create page, but also home page breaks. You can temporarily overcome the issue by disabling FPC

[shikhamis11]

okay let me check if it is also there in magento 2.2 versions

[shikhamis11]

@magento-engcom-team give me 2.2-develop instance

[magento-engcom-team]

Hi @shikhamis11. Thank you for your request. I'm working on Magento 2.2-develop instance for you

[magento-engcom-team]

Hi @shikhamis11, here is your Magento instance.
Admin access: https://i-20255-2-2-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[monotheist]

@shikhamis11 No this is not there till 2.2.6. I am facing this issue only after I upgraded from 2.2.6 to 2.3

[shikhamis11]

@ethniccode yes this issue is not in 2.2 versions

[monotheist]

@magento-engcom-team Is there a plan to fix this issue. If yes, in which release please?

[monotheist]

@shikhamis11 Is there any temporary fix to this?

[monotheist]

@orlangur Wondering why there aren't any discussions on this issue? Isn't this critical?

[monotheist]

@DanielRuf Hi, Just wanted to check if you could shed some light on this issue. Isn't this a critical issue? I have disabled FPC in my production store and awaiting a fix for this.

[monotheist]

@buskamuza Hi there, Are you aware of this bug? Would this get fixed in 2.3.1?

[orlangur]

@ethniccode could you please not spam everybody? For discussions you may join #github channel in Magento Slack.

[monotheist]

@orlangur This is not spam. I am looking for answer to my question. If Magento is encouraging the user community to report bugs then they should take responsibility to answer promptly?

[monotheist]

@mage2pratik @verklov @Alex-Lunkov @rogyar @dmytro-ch @ronak2ram @hitarthpattani Sorry to bother you. I wanted to check if you could shed some light on this issue. Isn't this issue affecting all merchants who are using magento 2.3? I was wondering if there is a temporary fix or workaround?

[DanielRuf]

Please reread https://github.com/magento/magento2/issues/20255#issuecomment-453823333

[xantek-imc]

As a workaround try updating the .htaccess file to add the HTTP HEAD block below the already existing TRACE and TRACK block. Then flush the cache and 404 should not return.

############################################
## TRACE and TRACK HTTP methods disabled to prevent XSS attacks

RewriteCond %{REQUEST_METHOD} ^TRAC[EK]
RewriteRule .* - [L,R=405]

############################################
## HEAD HTTP methods disabled due to https://github.com/magento/magento2/issues/20255

RewriteCond %{REQUEST_METHOD} HEAD
RewriteRule .* - [L,R=405]

Testing this issue I found the following in my access log:
"HEAD /customer/account/create/ HTTP/1.1" 404 3708 "-" "Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)"

This got me thinking the HTTP HEAD request was causing the 404 to get stuck in the FPC for the requested URI. To prove this open a page on a Magento 2.3 site, open browser developer tools, and make a HEAD request by pasting the following into the Javascript Console:

var xmlhttp = new XMLHttpRequest();
  xmlhttp.open("HEAD", "/customer/account/create/", true); 
  xmlhttp.onreadystatechange=function() {
      if(xmlhttp.readyState==4) { 
         console.log(xmlhttp.getAllResponseHeaders());
      }
  }
  xmlhttp.send(null);

Which returns: HEAD https://{site.testing.domain}/customer/account/create/ 404 (Not Found)

Now the customer account creation page return 404 and the FPC cache needs to be flushed to clear.

Based on the above finds I took the quick route of blocking all HEAD request using the existing .htaccess. Obviously, NGINX would require config addition. I've read about the potential traffic drawbacks of rejecting all HEAD request so I imagine a more graceful/targeted workaround could be developed.

Credit for detailing how to make HEAD requests in a browser: https://www.laurencegellert.com/2013/03/using-your-browser-as-a-dev-tool-to-test-head-requests/

Edit: example NGINX config addition

## HEAD HTTP methods disabled due to https://github.com/magento/magento2/issues/20255
if ( $request_method = HEAD ) {
   return 405;
}