Docs-website: Automate the updates of doc site index pages

[jerelmiller]

~After speaking with @djsauble, we've determined that keeping around the auto-generated index pages for each path segment in the URL doesn't make a ton of sense. We would like to keep around the "table of contents" pages for each landing page section, but only for those pages.~

~There are a few reasons we think this is a good idea:~

• ~These are auto-generated, which means we don't actually link to them anywhere~
• ~Maintaining these will be a nightmare. Any addition/removal of a page in one of the subpaths would need a change reflected in these pages. Unless we have some tooling to auto-generate this content, its going to be a nightmare to keep these up-to-date.~
• ~These pages are only there so that if someone where to start hacking with the URL and removing path segments, they'd see a page with some content instead of a 404 page. We think that if a user starts messing with the URL manually, it's expected that you'd see a 404 page for pages that don't exist.~

MMF 9 Scope

We want the site to automatically build these pages so content contributors do not have to manually update them.

Directory landing pages would already exist, so would not need to be accounted for here. Depending on how they're implemented, Agent API index pages and agent specific release notes may need to be accounted for in this work.

Acceptance criteria

• [ ] As a content contributor, I do not have to update index files manually
• [ ] As a content contributor, when I run the site locally I can generate those pages so I can see them on front-end with minimal effort

[jerelmiller]

cc @austin-schaefer

[austin-schaefer]

Hey @jerelmiller @djsauble , I don't think we can remove these. We do actually link to auto-generated indexes below the top level in many places and they serve a specific purpose.

Here's just a small selection of these auto-generated index pages that get substantial traffic:

• https://docs.newrelic.com/docs/infrastructure/install-infrastructure-agent (~900 hits a montH)
• https://docs.newrelic.com/docs/integrations/prometheus-integrations (630 hits a month)
• https://docs.newrelic.com/docs/synthetics/synthetic-monitoring/scripting-monitors (~2100 hits a month)
• https://docs.newrelic.com/docs/apis/synthetics-rest-api (680 pages)
• https://docs.newrelic.com/docs/integrations/amazon-integrations (400 hits a month)
• etc etc

The flexibility to link to automatically generated index pages for various depths of the taxonomy is a pretty useful tool taht we often leverage to help point people into a particular type of concept without having the create a hand-written page that collects that content together.

[jerelmiller]

The flexibility to link to automatically generated index pages for various depths of the taxonomy is a pretty useful tool taht we often leverage to help point people into a particular type of concept without having the create a hand-written page that collects that content together

This was a detail I was unaware of. Thanks for letting me know!

I think we probably need to rethink these auto-generated index pages then (at least in their current form). There are some problems with the way they exist today. These are only generated during the migration, which means that any future updates will cause these pages to get out-of-date pretty quickly. This means that as of now, it would require a manual edit to update theses TOC pages.

I think we need to instead do some behind-the-scenes magic to generate these at build, instead of at migration time, which would allow the site to handle management of the data.

Glad to know these are still important!

[austin-schaefer]

Awesome, thanks @jerelmiller . Updating at build makes a lot of sense to me.

Also worth noting: Some of those pages have redirects to them.

[djsauble]

@austin-schaefer What's the functional difference between category pages and these auto-generated index pages? It seems like both are essentially listings of content, with the exception that category pages get a sidebar nav and auto-generated index pages don't. Could we just convert auto-generated index pages to category pages?

[austin-schaefer]

I'm fine converting them into category pages, so long as that's done automatically. There's hundreds of them across the site so it's not viable authoring them by hand.

[jpvajda]

Shouldn't these types of page just follow the pattern of this page? https://docs-preview.newrelic.com/docs/agents . I just consider any list page of links an index page and any page that has boxes on it, a category page like: https://docs-preview.newrelic.com/docs/insights. Just want to be clear on the terminology we are using.

I also don't know if doing this at build time makes sense from a performance perspective if we run this job at every build of the site. I'll let @jerelmiller weigh in on that, but I do agree these need to stay and also need to auto-generation via some automatic job we setup. Perhaps it runs on a time interval vs. every build, if we have performance concerns.

[austin-schaefer]

Plus one for auto-generating and for every page. No strong opinion on when it happens, though I think we wouldn't want it to be _too_ infrequent so we don't have long lags between making changes and having those changes reflected in the site.

[jerelmiller]

Shouldn't these types of page just follow the pattern of this page?

Yes. The problem here isn't the pattern so much as the content itself. Right now the content for that page is an MDX file: https://github.com/newrelic/docs-website/blob/develop/src/content/docs/agents/index.mdx

Any updates to the title of the page, routes, new pages, moving pages, etc, would require a manual update to that MDX file. No bueno. What I'm suggesting is that rather than generating that MDX and writing it to that file, we could do some magic behind the scenes and generate it. That means any title change, new page, etc. would automatically be handled with no human intervention.

I also don't know if doing this at build time makes sense from a performance perspective

In regards to this, there is no real way around it. I doubt its going to hurt performance much. We are already building these pages during our build process, so likely the real-world impact would be negligible. The only difference between what I'm proposing and the current state would be where the data for the page comes from. In the current state, it comes from an MDX file. In a automated world, we'd do a GraphQL query to determine what all the subpages are and build the HTML off of that. Likely we won't have any perceived performance hit (famous last words 😄 )

[jpvajda]

@jerelmiller Thanks for clarifying. What MMF would this work fall into?

[jerelmiller]

Thats a great question. Based on our timeline right now, I'd say this should be a fast-follow item after we go live with the docs site. We are covered right now by the migration script since it automatically regenerates these pages each time we sync. That should get us to launch. The issue will show up as soon as we stop using the migration script and go live with the site. Are we ok making this a fast-follow item post GA?

[jpvajda]

@jerelmiller I think we'll have to discuss what automation needs (if any) are required to launch the site. @austin-schaefer