Trellis: Exclude non-production environments from being indexed by search engines (since WP 5.3))

[adamtomat]

• [x] I've read the guidelines for Contributing to Roots Projects
• [x] This request isn't a duplicate of an existing request
• [x] This is not a personal support request that should be posted on the Roots Discourse community

Summary

WordPress 5.3 has moved away from using the "robots.txt" to discourage search engines to adding a meta tag to prevent a site from being listed in search engines.

As part of this, it is also recommended that the HTTP Header X-Robots-Tag: noindex, nofollow is added to ALL requests (including images, pdfs, videos etc). I believe this should be something Trellis should handle out-of-the-box.

Motivation

This impacts every non-production environment. WordPress will add this meta tag to all pages it renders but it will not be applied to any assets etc.

I suggest that this header should be added by default to non-production environments (e.g. staging) on all requests.

This solution is recommended by one of the core team working on WordPress. At the bottom of this article he says:

Excluding development servers from search engines.

The most effective method to exclude development sites from being indexed by search engines is to include the HTTP Header X-Robots-Tag: noindex, nofollow when serving all assets for your site: images, PDFs, video and other assets.

As most non-HTML assets are served directly by the web server on a WordPress site, the core software is unable to set this HTTP header. You should consult your web server’s documentation or your host to ensure these assets are excluded on development sites.

Additional context

Here is a link to the core WP ticket discussing this:
https://core.trac.wordpress.org/ticket/43590

[retlehs]

good call! we should adapt the existing mu-plugin in bedrock for this

https://github.com/roots/bedrock/blob/ff5e28ac74f1e9aac1b44f8d197054a34eb85079/web/app/mu-plugins/disallow-indexing.php

[adamtomat]

It's not (just) a Bedrock issue though. From my understanding that mu-plugin is still doing the right thing?

That won't be called when requesting assets (e.g. a PDF) as nginx handles this as it's a file rather than routing it through WP (at least that's my understanding of how Trellis has setup nginx).

[swalkinshaw]

@adamtomat do you want to try contributing this change? It shouldn't be too bad and I can help direct you to the right place if you want.

[adamtomat]

@swalkinshaw yeah I'm up for that. Think this should be on by default, but can be disabled via config?

And I assume it's just optionally adding the header to the nginx conf under certain envs? (As opposed to conditionals within the config itself?

[swalkinshaw]

Think this should be on by default, but can be disabled via config?

Yep 👍

And I assume it's just optionally adding the header to the nginx conf under certain envs? (As opposed to conditionals within the config itself?

Yeah I'd just use a conditional like this: https://github.com/roots/trellis/blob/76337fd20e1a137be6487d473d2d8d4ee7997e9e/roles/wordpress-setup/templates/wordpress-site.conf.j2#L28-L30

Maybe define a named block for it as well to make it easier to override.

I might be even be okay with skipping an explicit config option for this and just making people override the template since I assume it would be quite rare? 🤷‍♂

[TangRufus]

Instead of {% if env == 'development' -%}, we can make a variable for that, similar to h5bp_cache_file_descriptors_enabled

https://github.com/roots/trellis/blob/76337fd20e1a137be6487d473d2d8d4ee7997e9e/roles/wordpress-setup/defaults/main.yml#L42-L44

[adamtomat]

Do you mean something like this? @TangRufus

not_prod: "{{ env != 'production' }}"
robots_tag_header: "{{ item.value.robots_tag_header | default({}) }}"
robots_tag_header_enabled: "{{ robots_tag_header.enabled | default(not_prod) }}"

[TangRufus]

Do you mean something like this?

Yes.

I mean:

not_prod: "{{ env != 'production' }}"
robots_tag_header_enabled: "{{ robots_tag_header.enabled | default(not_prod) }}"

Add the robots_tag_header if you see fit.

[swalkinshaw]

Closed by https://github.com/roots/trellis/pull/1158.

Thanks for bringing this up and implementing it @adamtomat 😄