Magento2: Normalize User Agent

[miguelbalparda]

Preconditions

1. Magento 2.2
2. Varnish enabled

Steps to reproduce

1. Hit a page with curl -A "iPhone 7" -I http://m2varnish.troutzilla.com/
2. Get a MISS
3. Hit the same page with a slightly different UA
4. Get a MISS

Expected result

1. Hit a page with curl -A "iPhone 7" -I http://m2varnish.troutzilla.com/
2. Get a MISS
3. Hit the same page with a slightly different UA
4. Get a HIT

Actual result

1. User agent is not normalized, leading to a higher cache fragmentation

Adding something like

if (req.http.User-Agent ~ "iP(?:hone|ad|od)|BlackBerry|Palm|Googlebot-Mobile|Mobile|mobile|mobi|Windows Mobile|Safari Mobile|Android|Opera (?:Mini|Mobi)") {
    set req.http.X-Normalized-User-Agent = "mobile";
} else {
    set req.http.X-Normalized-User-Agent = "other";
}

and use Vary: X-Normalized-User-Agent,Accept-Encoding or

if (req.http.User-Agent ~ "iP(?:hone|ad|od)|BlackBerry|Palm|Googlebot-Mobile|Mobile|mobile|mobi|Windows Mobile|Safari Mobile|Android|Opera (?:Mini|Mobi)") {
    set req.http.User-Agent = "mobile";
} else {
    set req.http.User-Agent = "other";
}

should do the trick.

[orlangur]

Hm, do we have any server side user-agent specific logic at all to distinguish other/mobile?

Another place with similar comparison https://github.com/magento/magento2/blob/2.2-develop/.htaccess#L163

[miguelbalparda]

I got that piece of code from Nexcess_Turpentine I'm open to suggestions or changes but I do believe this change might have value even if we make it configurable and optional from the backend. Also, it is not so much to distinguish mobile from others but to make the cache less fragmented.

[cjarvis86]

Changing the Vary to be set as X-Normalized-User-Agent or another name instead of User-Agent will tell Varnish to cache the content based on that header content instead of User-Agent header and still allow the application and backend web server to see the true User-Agent instead of other or mobile.

I tested overwriting the Vary to X-Normalized-User-Agent instead of User-Agent in Apache and it looks to be working as you would expect, grouping the cache into mobile and other based on the VCL logic shown above but preserving the User-Agent string to be seen in the application and web service logs.

Perhaps the VCL logic could be included and the Vary header coming from the application being adjusted to match if a normalize user agent function was enabled in the admin?

[magento-engcom-team]

@miguelbalparda, thank you for your report.
We've created internal ticket(s) MAGETWO-83987 to track progress on the issue.

[kirmorozov]

Let me incorporate one of known solutions:
https://github.com/varnishcache/varnish-devicedetect
License allows.

[kirmorozov]

Will resolve it #mageconf

[magento-engcom-team]

@kirmorozov thank you for joining. Please accept team invitation here and self-assign the issue.

[maximbaibakov]

Hi @miguelbalparda

Just tested this on 2.2.5 and 2.2-develop branch - can't reproduce the issue.

Also, can't see the reason, why do we need to cache a page based on User-Agent. Ideally, the cache should not be depended on User-Agent.

Is it still reproducible for anyone? Or we can close it?

Regards,
Maxim

[arnoudhgz]

I second @maximbaibakov on this, you should not want page cache per user-agent. Can this ticket be closed?

[sdzhepa]

Hello @miguelbalparda

Due to comments

1. https://github.com/magento/magento2/issues/11726#issuecomment-412246586
2. https://github.com/magento/magento2/issues/11726#issuecomment-412301834

And no activities(answers/confirmation) issue is not actual anymore.

Thank you for feedback and collaboration!

[mrbig]

Hello @sdzhepa ,
I was able to reproduce this issue with Magento 2.2.5.
I realized that the cause is that in the default .htaccess there is commented line that adds User-Agent to Vary header: https://github.com/magento/magento2/blob/310648c7f2d1280502f0b92e3dcc28894f7faf86/.htaccess#L112-L113
Some time ago someone thought those lines could be useful, and enabled it - that messed up the varnish cache.

Maybe a review of the default .htaccess would be useful, and the deprecated configs could be removed completely.

What do you think?

[killday]

Magento 2.3.1 upgraded from 2.2.0
I've got the same problem. I try to remove the User-Agent by the lines in .htaccess but with no success.
Did anyone solve this and how?

[an4rei]

SetEnv dont-vary 1 in .htaccess fixed it for me on magento 2.4.1 and varnish 6