Freecodecamp: Noindex /learn

[QuincyLarson]

As part of cleaning up our Google footprint, we want to de-index /learn. We don't want people to arrive from Google to the middle of a challenge (because they most likely won't be ready to sit down and code right then and there).

So we need to add this to all pages of /learn

<meta name="robots" content="noindex">

[RandellDawson]

What about keeping indexes on the introduction pages for each curriculum section?

[raisedadead]

I do not think we would want to index anything from learn on the public search engines.

[QuincyLarson]

@RandellDawson We can create /news landing pages for each of these certifications and its subsections. These can be more keyword rich and betted designed for Google. Our current /learn pages aren't designed with traffic from Google in mind.

[RandellDawson]

@raisedadead Based on what @QuincyLarson said above, is this a way we can just remove the introduction/landing pages from the curriculum?

[raisedadead]

Yes, I believe so.

[hassaanp]

@raisedadead are you working on adding the meta tags? If not, I can give it a go.

[raisedadead]

@hassaanp if you are interested in this here is a plugin that I intend to add to the client application:

https://github.com/itgalaxy/generate-robotstxt

Feel free to work on this.
Thanks.

[hassaanp]

@raisedadead - acknowledged.

[hassaanp]

Pushed a PR.

I have tested the code on my local setup by building the client and serving it via npx.
The learn page shows that the meta tag for noindex is added. The appropriate robots.txt is also generated.
I have used this plugin to generate the robots.txt file. If I am not wrong, the plugin is built upon the one referenced by @raisedadead. I just found it more convenient since it could be plugged directly into the Gatsby config.

[raisedadead]

Thanks for researching this and using a more customisable plugin. In this case, let's take this further and make sure that the only /learn/* paths are not indexed (which I assume the current PR does).

Also, I see there are options for picking up env values for host, etc. Lets implement those and configure this to be able to deploy for www.freecodecamp.org, www.freecodecamp.dev and so on.

[hassaanp]

The default values are picked from the siteMetadata in the config, which I assume will have the appropriate ${siteMetadata.siteUrl} defined when deploying for either mentioned URLs.

Yes, this PR is only targetting the /learn/* path. I have pushed another commit that removes the noindex header in the unmount lifecycle method to ensure it is not persisted across other paths.

[raisedadead]

Gosh! That reminds me that we need to make this dynamic:

https://github.com/freeCodeCamp/freeCodeCamp/blob/882cddd80076b232882c8960a784499d95e816e7/client/gatsby-config.js#L12-L14

Will you be interested in handling that in this same PR. If not let me know we can tackle this separately.

[hassaanp]

Apologies for the delay.
I have pushed a commit to make the siteUrl dynamic via the .env
Kindly review @raisedadead

[raisedadead]

Left some additional comments, this is mostly ready.