Magento2: Varnish does not appear to be configured for a public folder web root (at least, with NGINX)

[andrewhowdencom]

Preconditions

1. Magento 2.4-develop
2. Varnish 4/5/6

Steps to reproduce

1. Set up a server according to the provided recommendations, with NGINX as the HTTP host and using the public folder
2. Export varnish as shown by Magento
3. Install varnish configuration

Expected result

1. Server works

Actual result

1. Varnish doesn't work (specifically, it returns a 503 and complains about upstream being down). Additionally, there are a bunch of messages in the NGINX error log about "404 /pub/health_check.php not found"

File is here: https://github.com/magento/magento2/blob/2.2-develop/app/code/Magento/PageCache/etc/varnish5.vcl

As you can see, it has the pub hardcoded. I might suggest doing detection in the application, and adjusting the probe root accordingly.

[orlangur]

I believe it's handled properly in other lines of the same file: if (req.url ~ "^/(pub/)?(media|static)/") {

Not sure why root index.php it still not removed in favor of pub/index.php though.

[andrewhowdencom]

I'm not sure -- I don't know if Varnish caches it's own probes? To me that wouldn't make sense. Docs aren't helpful here:

https://varnish-cache.org/docs/4.0/reference/vcl.html#reference-vcl-probes

Additionally, tailing nginx logs seems to support this (with the pass directive disabled):

127.0.0.1 - - [01/Nov/2017:17:23:30 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:23:35 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:23:40 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:23:45 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:23:50 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:23:55 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:00 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:05 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:10 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:15 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:20 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"
127.0.0.1 - - [01/Nov/2017:17:24:25 +0100] "GET /health_check.php HTTP/1.1" 200 5 "-" "-" "-"

[andrewhowdencom]

Oh also I patched this to remove the pub. Specifically, we route based on virtualhosts rather than ports:

backend default {
    .host = "127.0.0.1";
    .port = "80";
    .first_byte_timeout = 600s;
    .probe = {
        # SITEWARDS CHANGES START
        # The normal probe configuration does not allow setting specific headers for request[1]. So, the below snippet
        # manually constructors the request to include the Host header.
        #
        # [1] https://varnish-cache.org/lists/pipermail/varnish-misc/2011-October/021298.html
        .request =
            "GET /health_check.php HTTP/1.1"
            "Host: magento.local"
            "Connection: close";
        # SITEWARDS CHANGES END
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

[magento-engcom-team]

Hi @andrewhowdencom
Could you please let us know if the issue is still actual? Thank you

[andrewhowdencom]

Looks like it. As mentioned, i customised the VCL in production for us as above, for various reasons. But the 2.2 develop branch still has the path hard-coded.

I imagine the rule "if you need two versions of index.php you'll probably need two versions of health_check.php" holds true.

https://github.com/magento/magento2/blob/2.2-develop/app/code/Magento/PageCache/etc/varnish5.vcl#L13

[magento-engcom-team]

@andrewhowdencom, thank you for your report.
We've created internal ticket(s) MAGETWO-83718 to track progress on the issue.

[erfanimani]

This is still an issue.

Preconditions

• Magento 2.3.1
• Varnish 4
• Run Magento out of pub (web root being pub — the proper way to run PHP apps)

Steps to reproduce

• Click Export VCL for Varnish 4

Expected result

• Nothing breaks

Actual result

• Website down because health check URLs have pub in them for whatever reason

Specifically the following lines:

Line 12: .url = "/pub/health_check.php"; (why is /pub there? of course backend origin is going to 404, making Varnish think that the node is down..)

Line 65:

# Bypass health check requests
if (req.url ~ "/pub/health_check.php") {
    return (pass);
}

[erfanimani]

How is this still open after multiple years? Or maybe we're all expected to investigate the exported VCL file and catch the error ourselves?

[m2-assistant[bot]]

Hi @engcom-Alfa. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

• [ ] 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Alfa
Thank you for verifying the issue. Based on the provided information internal tickets MC-30080 were created

Issue Available: @engcom-Alfa, _You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself._

[m2-assistant[bot]]

Hi @engcom-Alfa. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• 
  
  1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
  
  

• 1. Verify that the issue is reproducible on 2.4-develop branch
     Details

     - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
     - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
     - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
• 1. If the issue is not relevant or is not reproducible any more, feel free to close it.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[engcom-Alfa]

Hi @andrewhowdencom .

According to the Magento documentation in part Advanced Varnish configuration described that: The health_check.php script is located in the pub directory. If your Magento root directory is pub, then be sure to change the path in the url parameter from /pub/health_check.php to health_check.php.
If you manually change the path to .url = "/health_check.php"; everything work correctly.

1. Install Varnish (ver. 4 for example);
2. Configure Varnish and your web server;

Manual testing scenario:

1. Set document root to /pub (devdocs);
2. Go to Admin->Stores->Configuration->Advanced->System->Full Page Cache;
3. Choose "Varnish Cache" from "Caching Application" drop down and click on "Save" button;
4. Then click on "Export VCL for Varnish 4" button and open it;
5. Change /pub/health_check.php to health_check.php and Save;
6. Copy the file to the /etc/varnish/ directory;
7. Open home page;

Actual Result: :heavy_check_mark: Server works

@andrewhowdencom Could you take a look?
Thanks!

[engcom-Alfa]

@andrewhowdencom , I am closing this issue now due to inactivity.
Please reopen and update if you wish to continue.
Thank you for the collaboration!

[andrewhowdencom]

I think the desired result on this one would be to detect what mode the store is running in (given that it itself can introspect and determine what the suitable varnish configuration would be), and export things appropriately without requiring developer input.