Docs-website: [Content] Migration is pulling in dead content

[austin-schaefer]

Description

The migration script is pulling in content from somewhere that no longer exists. For example, these pages:

• https://www.docs-preview.newrelic.com/docs/servers/new-relic-servers-windows/installation-configuration/installing-servers-windows
• https://www.docs-preview.newrelic.com/docs/servers/servers-pages/servers-ui/new-relic-servers-overview-page

These docs were removed/EoLed a long time ago and redirect to Infra docs.

Steps to Reproduce

1. Go to https://www.docs-preview.newrelic.com/
2. Click on any of the links in the Servers category

Expected Behavior

Redirect to appropriate destination

Additional context

It's possible these pages are still hanging out somewhere in Drupal as "dummy docs," in which case it would be good to have @roadlittledawn help investigate/resolve.

Acceptance Criteria

• [ ] Assess whether or not a document is a "dummy doc"
• [ ] Redirects are set up to replicate the behavior of "dummy docs"
• [ ] "Dummy docs" files are discarded and _not_ turned into MDX files

[jerelmiller]

@austin-schaefer we are currently pulling that data from the docs endpoints: /api/migration/content/<type>/list. As you mentioned, we likely need to do a little work on the Drupal side to ensure these "dead" docs don't get synced as real pages.

[roadlittledawn]

we're pulling from dev, so that may be the cause here. dev is behind prod database. i haven't synced that in...month or more

[austin-schaefer]

@roadlittledawn : Hmm, I think these pages were deleted summer 2019-ish, so shouldn't be impacted by a belated sync.

[roadlittledawn]

ah those look like dummy docs. dummy docs are going to be hard to detect because they essentially look like a normal node. @austin-schaefer do we think that all dummy docs have text Redirect to WHATEVER in their body?

[roadlittledawn]

and are all "archived" docs unpublished? i can't remember what the workaround process was for archiving. I _think_ we unpublished them and set a redirect if applicable, right?

[roadlittledawn]

I think we'll eventually have to address this issue as part of redirects MMF. We don't currently have any redirect functionality in place.

It will be tricky to catch "dummy doc" and "archived" doc redirects. "Dummy docs" are a workaround for adding doc links to multiple categories/nav menus. "Archived" docs are old docs that still exist on the back end of the site (content is not available to front end) to redirect to another URL. Both types essentially masquerade as a redirect (or sometimes, collection of redirects) to another URL.

One way to do this might be to add this functionality to migration script:

1. Install gatsby-plugin-redirect-from plugin
2. When migration runs, use docId field in JSON to ask the redirects resource what URLs should redirect to _this_ doc, and check if this doc redirects to another URL.
3. If this doc redirects to another URL, it's likely either a "dummy doc" / "archived" doc and we need to somehow take all the URLs that redirect to this doc and add them to the doc that this doc redirects to. Or output to console so user can take note what URLs will not be redirected automatically.
4. If this doc does not redirect to another URL, safe to assume it is not a dummy doc / archived doc, so we can add the URLs from step 2 that redirect to _this_ doc to frontmatter field redirect_from in this doc/mdx file.

[jpvajda]

I'd like to propose we fix this in MMF11 as It would be nice to have all migration tasks done as we going into the final stages of the project, @jerelmiller / @zstix / @roadlittledawn what do you think?

[roadlittledawn]

@jpvajda I believe the remaining work here is mostly dummy docs. There have been several instances of rogue content coming over which I've been able to fix by tweaking Drupal source (wrong URL paths for release notes and landing pages, filtering certain special nodes out of the JSON, etc).

After doing some of the redirect work and looking at dummy docs currently in the repo, I think it's going to be pretty tough to programmatically find and remove dummy docs. Maybe we add to a production readiness checklist / audit after migration but before launch? For example, I was able to search src/content/docs for Redirects to (a common but not standard thing for a TW to enter for the body content of a dummy doc) and quickly determine what was a dummy doc. One PR for that and if many eyes are on the repo, maybe we catch all the others?

[jpvajda]

@roadlittledawn what are dummy docs used for?

[roadlittledawn]

@jpvajda essentially a Drupal node masquerading as a redirect. it was a workaround to add links to the same piece of content in more than one category's menu. more info in this comment above