Magento2: No Varnish Purge Request from Nginx

[smaadjahangir]

Hi,

I've following webstack:

Nginx > Varnish > Apache.

The problem i am seeing is that Nginx is not forwarding Purge request to Varnish. I am getting (status 400 ) invalid request when ever purge request is made. I have a hunch that Nginx is adding trailing slash to either URL or "X-Magento-Tags-Pattern" header is getting modified on the way. Nginx Error:

client sent invalid request while reading client request line, client: 127.0.0.1, server: example.com, request: "PURGE HTTP/1.1"

Regards,
Maad

[smaadjahangir]

Can someone tell me the exact request which Nagento2 forward to Webserver. I'll try to dig the actual problem by issuing the request manually to Nginx.

[smaadjahangir]

I think for some reason Nginx is not forwarding "X-Magento-Tags" and "X-Magento-Tags-patterns" header. OR the URL is getting null when Nginx is parsing request.

This person is facing similar issue:

https://community.magento.com/t5/Technical-Issues/Issue-with-purging-Varnish-cache/m-p/23784/thread-id/190

[smaadjahangir]

Has anyone faced this issue. I am unable to sort it and purging Varnish is critical functionality that i'll need.

[boldhedgehog]

I have faced this, and the forum topic above had been created by me.

At this moment, I run a request with cURL to purge the cache.

curl -X 'PURGE' -H'X-Magento-Tags-Pattern: .*' localhost:6081

[smaadjahangir]

Hi. The thing is that i want it to work with Magento Flush cache which one perform from Admin. Doing it manually via ssh is an extra process. I'll check if this somehow can be incorporated with PHP call that Magento make.

[smaadjahangir]

@joanhe Have you checked this issue?

[toddwolaver]

I am seeing a similar issue, though I'm not using Nginx. Varnishlog shows the following when a PURGE request is made:

*   << Request  >> 98458     
-   Begin          req 98457 rxreq
-   Timestamp      Start: 1451936193.270818 0.000000 0.000000
-   Timestamp      Req: 1451936193.270818 0.000000 0.000000
-   HttpGarbage    "PURGE%00"
-   ReqAcct        47 0 47 28 0 28
-   End            

It appears this may be a result of an NULL path being set on the $server URI object in sendPurgeRequest (vendor/magento/module-cache-invalidate/Model/PurgeCache.php)

Via hard coding hack, adding a path of '/' to the $server URI object will purge the Varnish cache for me.

$server->setPath('/');

I've tried explicitly defining the cache server with a 'path' via http_cache_hosts in the env.php however only the host and port values are used.

[alankent]

Yes, it looks like a bug that no path is being supplied sometimes. Any information about the sequence of operations performed to trigger this case would be helpful. (Or maybe print out a stack trace (e.g. throw an exception) if NULL is passed into the function.) (Note: the "PURGE%00" is suspicious - a NUL byte sneaking in somewhere? Or maybe that is just because no path name was supplied?)

[toddwolaver]

In my case I am fairly certain the "PURGE%00" is a NULL byte due to the NULL path.

I don't see how a path would be set, where in my case the path is always null.

Magento\PageCache\Model\Cache\Server::getUris either explicitly sets the path to NULL or doesn't set a path (depending on if http_cache_hosts configuration is set or not in env.php).

    public function getUris()
    {
        $servers = [];
        $configuredHosts = $this->config->get(ConfigOptionsListConstants::CONFIG_PATH_CACHE_HOSTS);
        if (null == $configuredHosts) {
            $httpHost = $this->request->getHttpHost();
            $servers[] = $httpHost ?
                UriFactory::factory('')->setHost($httpHost)->setPort(self::DEFAULT_PORT)->setScheme('http') :
                UriFactory::factory($this->urlBuilder->getUrl('*', ['_nosid' => true])) // Don't use SID in building URL
                    ->setScheme('http')
                    ->setPath(null)
                    ->setQuery(null);

        } else {
            foreach ($configuredHosts as $host) {
                $servers[] = UriFactory::factory('')->setHost($host['host'])
                    ->setPort(isset($host['port']) ? $host['port'] : self::DEFAULT_PORT)
                    ->setScheme('http');
            }
        }
        return $servers;
    }

It could be possible that some Varnish configurations may accept the NULL byte in the Varnish request, however I have been unable to get it to work.

[jasperzeinstra]

I was also struggling with the purge request. In the end I solved my problem by connecting to port 6081. So my env.php contains this http_cache_hosts:

    'http_cache_hosts' => array (
        0 => array (
            'host' => '127.0.0.1',
            'port' => '6081'
        ),
    )

[gwillem]

@jasperzeinstra then you are just lucky that Varnish does not choke on the syntax error, but the behaviour is undefined (flush all or nothing). The real fix is figuring out why the URI could go missing, as @alankent pointed out.

[gordonknoppe]

Internal issue MDVA-56.

[davidalger]

The null byte in the request is not a bug in Magento/Zend code. I've verified it simply shows up that way in the varnishlog output if you don't include a path in the request. I tested by manually sending requests using telnet.

The culprit actually lies in the combination of the implementation of \Zend\Uri\Uri::toString, and the Magento code not explicitly setting a path. The toString method which renders the URI only appends the slash automatically in the case of a query or fragment being set in addition to the hostname:

if ($this->path) {
    $uri .= static::encodePath($this->path);
} elseif ($this->host && ($this->query || $this->fragment)) {
    $uri .= '/';
}

So the Magento code needs to set the path in all cases. Welcome feedback, but I believe the following implementation of the method will solve this problem across all cases:

    public function getUris()
    {
        $servers = [];
        $configuredHosts = $this->config->get(ConfigOptionsListConstants::CONFIG_PATH_CACHE_HOSTS);

        if (is_array($configuredHosts)) {
            foreach ($configuredHosts as $host) {
                $servers[] = UriFactory::factory('')
                    ->setHost($host['host'])
                    ->setPort(isset($host['port']) ? $host['port'] : self::DEFAULT_PORT)
                ;
            }
        } elseif ($this->request->getHttpHost()) {
            $servers[] = UriFactory::factory('')->setHost($this->request->getHttpHost())->setPort(self::DEFAULT_PORT);
        } else {
            $servers[] = UriFactory::factory($this->urlBuilder->getUrl('*', ['_nosid' => true]));
        }

        foreach ($servers as $key => $value) {
            $servers[$key]->setScheme('http')
                ->setPath('/')
                ->setQuery(null)
            ;
        }
        return $servers;
    }

I've tested it both with and without the http_cache_hosts configuration in env.php and it works in both cases. The only one I have not tested is coming from the CLI, which is what would cache the trailing else block.

If no-one here sees something incorrect with this implementation, I'm going to PR the change.

[davidalger]

Fixed in 2.0.6

[nobuttsalex]

Edit: Sorry I meant 2.1.2

Edit 2: Found the solution, should have been 80 in config and not 8080 and 127.0.0.1 wasn't enough, had to put the domain name in and have a record for each website

I'm experiencing the same problem in 2.1.1. I'm using nginx for ssl termination on 443, apache on port 8080 serving Magento 2 and varnish on port 80. I have the following entry in env.php

'http_cache_hosts' =>
array (
0 =>
array (
'host' => '127.0.0.1',
'port' => '8080',
),
),

But clearing the full page cache in the admin does not clear the varnish cache. The only way I can do it currently is killing the varnish process and restarting it.

Is there something I may have missed or has this bug come back?

[nobodyfamouz]

We are having the same issue as nobuttsalex. Any ideas?

[giacmir]

@nobuttsalex Purge requests should be processed by varnish and not by apache. So you must use port 80 (varnish) and not 8080 (apache).

[dimple-vasoya]

How can we clear the varnish cache for the particular URL?