Docs-website: [Umbrella Issue] Gatsby Cloud Evaluation

[jpvajda]

Overview

This issue captures all the related work occurring to decide if moving our Gatsby Sites to Gatsby Cloud make sense.

• docs.newrelic.com
• opensource.newrelic.com
• developer.newrelic.com

Other sites to consider

• private docs previews? We host a private instance of our Docs site for writers to create preview or beta docs that aren't ready to be made available on our main Docs Site, we use the amplify preview functionality to make those available to customers.

Acceptance Criteria

Business

• [x] Get a detailed quote from Gatsby
• [x] Confirm we have approval to actually make the switch from a procurement point of view.
• [x] Evaluate the plan and ensure hosting and build numbers align with our needs.
• [x] Ensure we understand the concierge service, build audits and the cost for assistance in migrating to V3 and Gatsby Cloud.
• [x] Decide of all of sites move to Gatsby Cloud now, or later.

Sites to migrate

• [x] create new tickets for these migrations

- Docs --> Sprint 3
- OSS ---> Sprint 4
- Dev  ---> Sprint 4
- Private docs ---> Sprint 5
- Bulk-edit private docs ---> Sprint 5

DOCs #1933
OSS newrelic/opensource-website#757
DEVnewrelic/developer-website#1270
Private DOCs newrelic/docs-website-private#5
BETA DOCs newrelic/private-docs-site#45

Engineering

• [x] Complete upgrade to V3 on all sites and the Theme

DOCS https://github.com/newrelic/docs-website/issues/1809
OSS https://github.com/newrelic/opensource-website/issues/748
DEV https://github.com/newrelic/developer-website/issues/1248

• [x] Spin up a test account on Gatsby Cloud https://github.com/newrelic/docs-website/issues/1945
• [x] Setup integration with our Github Docs repository
• [x] Run tests cases and evaluate Gatsby Cloud
• [x] Make a decision to migrate to Gatsby Cloud
• [x] Define the migration approach and create a document that outlines what we'll need to consider.

Test Cases

• [x] Evaluate several incremental build tests to see how Gatsby Cloud handles these cases with varying degree of changes.
• [x] Test a cold build with no cache for build time results
• [x] Test a warm build with cache for build time results
• [x] Other tests TBD by @zstix

Current Benchmarks

_A “cold build” will start from scratch, while a “warm build” will reuse what it can from a cache._

Cold Builds

relates to issue #1661

TBD

Warm Builds

TBD

What to expect from Incremental builds on Gatsby Cloud.

Incremental builds will function very similarly for MDX content as it does for other types of content (e.g. CMS content) - the degree of incrementalism is predicated on the degree to which a component is shared among other MDX files. An existing cache is a basis for incremental builds. Needs something to validate against. So, if the changed content is only local to that MDX file, and there is an existing cache, then only that MDX file would be rebuilt. If the content, however, is shared among multiple MDX files.. and there's further sharing among the dependent MDX files, then there would be more page queries that become invalidated, thereby resulting a slower incremental build.

Current Issues with Amplify

Queing of releases on Amplify as build times are slow.

• If we merge to main at 9am, and then again at 9:30 but the 9am merge has NOT built on the public site yet, how does this work? Does it queue the second merge? Does it combine both and start from scratch?
• It should queue it up. You’ll get the first merge deployed, then once that is done, it will start building the 2nd one that is all handled at the amplify level. So we’ll need Gatsby Cloud to handle this case.

Build/deploy times are very unpredictable:

• Perhaps the worst issue is how unpredictable builds times are. This makes it very hard to forecast to stakeholders when their content will go live.
• Even after merging into Main, we have no idea when changes will go live. Even some insight or view into the current status, or even better an ETA after merging into main, would be great.
• We had a critical release (rollback of docs on a feature that got released, and then itself got rolled back). Unpredictable build times meant it took over two hours to build and deploy the site. See #1394.

Slow deploy times:

• Creates collisions when lots of things are going on.
• Much slower to deploy content than the business is used to. (For example, release notes used to go live in 1–2 minutes and can now take hours.)
• This is particularly worrisome if we need to fix a "bad" doc/deploy, or when trying to time docs content release with a product release.

Slow preview build times blocking writers and stakeholders:

• PR reviews have to wait for a while before they reasonably check on things.

Unreliable previews:

• The previews themselves sometimes just fail for unknown reasons.

Amplify is a black box for those without access

• Writers have no way to check on the status of builds, leaving them unable to communicate to stakeholders the status of their requests.

Unreliable cache.

• We've enabled incremental builds, but it doesn't seem to work very often (if at all). Incremental build times in our GH actions will be pretty quick (~2 min) but will still take the full ~50 min in Amplify.
• I’ve had an Amplify check go for 18 hours without completion and haven’t yet seen an Amplify check successfully complete.

We have an entire GitHub action to build the site for PRs just so that we can get a reliable way to check failures.

• We should not need a duplicate build to work around Amplify's shortcomings

Provisioning step gets stuck for way longer than it should, resulting in even longer build times. The build is already ~50 min.

• We had an incident on 3/3 that needed a quick turnaround, but it took ~10-20 min just to get past the provision step, which tacked on unneeded turnaround time

Hitting Rate limits on Amplify after many builds a day...

[CWL_THROTTLING_EXCEPTION: The maximum request rate permitted by Amazon CloudWatch Logs has been exceeded for your account. Error message: Rate exceeded]
CreateLogStream
50 transactions per second (TPS/account/Region), after which transactions are throttled. You can request a quota increase

Releases (PR from Develop to Main) sometimes take hours

• because as new PRs are merged to Develop, they are added to the PR to main, checks start again, etc. Our only option currently appears to be asking everyone to not merge PRs for designated time periods while we wait for release merges to go through. This creates more cognitive burden. (to clarify - I kicked off a release last week that had only about 4 commits, but by the end, there were more like 20 commits in the PR to main)

One release (currently) not detected & deployed

On Thursday April 1st, we merged https://github.com/newrelic/docs-website/pull/1660 into main. The merge was detected by Amplify, built, and deployed. On Friday April 2nd, we merged https://github.com/newrelic/docs-website/pull/1674 into main and Amplify _did not_ detect it. This meant that work we intended to be released was stuck in limbo until we could manually trigger another release. (This is a big red flag, in my opinion - Z)

Gatsby Cloud Benchmark Testing

Gatsby has set up Docs Site on an instance of Gatsby Cloud for benchmarking and here some initial results. This doesn't include an upgrade to Gatsby V3, which should also improve build times.

Cold Build Test

Warm Build Test

[jpvajda]

I've asked Luke if we eventually move the Dev Site and the OSS to Gastby Cloud, if there is an additional cost for that, I've been thinking about it lately I don't think we need to migrate those sites off Amplify at this time, but may consider it in the future to consolidate our sites into one place, but I don't feel it's neccessary to that now. @zstix / @polfliet thoughts?

[zstix]

If we're not experiencing as many problems with the other sites being on Amplify, then I suppose we don't _need_ to migrate them. That said, I'm a little hesitant to ask our engineers to support two different hosting platforms.

In my mind, it sorta depends on how long it would take to migrate _all_ our sites to the same platform (and the cost difference).

[jpvajda]

@zstix totally agree. I'd like to be on one platform ideally for all our gatsby sites, perhaps this is a follow up task to move the OSS / Dev site over after we move the Docs Site over. Could be considered sustaining work.

[jpvajda]

@zstix After our discussion here is a proposed migration schedule, let me know what you think.

Sites to migrate

    Docs --> Sprint 3
    OSS ---> Sprint 4
    Dev ---> Sprint 4
    Private docs ---> Sprint 5
    Bulk-edit private docs ---> Sprint 5

[jpvajda]

Need to answer the following: to setup our enterprise Gatsby cloud test account (this account will convert to our actual account if we decide to switch to gatsby cloud) is there a service account we use for this type of stuff? It's tied to Github authentication.

Gatsby suggests we tie this to new relic the org, not an individual account.

[jpvajda]

the recommendation was:

As far as I know, "service accounts" aren't a specified thing with GitHub. I believe the closest thing they have is what's coined a machine user (which is more or less just a bot). That's really just a personal account that we limit to your use case and we'd grant it membership to the newrelic organization.

[jpvajda]

https://github.com/newrelic/docs-website/issues/1933
https://github.com/newrelic/opensource-website/issues/757
https://github.com/newrelic/developer-website/issues/1270
https://github.com/newrelic/docs-website-private/issues/5
https://github.com/newrelic/private-docs-site/issues/45

[jpvajda]

Currently blocked on vendor review, will take ~7-10 business days

[jpvajda]

jira VRI-8985 for the security review

[jpvajda]

This is now requiring GM approvals. 🧷

[zstix]

I took a shot at comparing Amplify and Gatsby Cloud. Getting a PR-for-PR comparison is a little tricky, since PRs are sometimes merged before both completed, but I was able to get a enough data to draw some initial conclusions:

Overall Build Time

| PR | AWS Amplify | Gatsby Cloud |
| - | - | - |
| #2338 | 4:10 | 13:20 |
| #2326 | 58:47 | 36:24 |
| #2320 | Failed | 39:30 |
| #2277 | 58:46 | 40:47 |
| #2335 | 5:42 | 23:51 |
| #2260 | ??:?? | 11:59 |

_Assessment_: For this set of PRs, Gatsby Cloud doesn't seem to be significantly better. We can continue to compare builds as PRs come in. Anecdotally, it seems like Gatsby builds come in about 10 - 20 minutes earlier, but those results are varied.

Run Page Queries

The run page queries step (compiling all the data to display for each page) is the slowest part of any build. Here's the breakdown for some of those PRs:

| PR | AWS Amplify | Gatsby Cloud |
| - | - | - |
| #2326 | 2158.588s (~36min) | 1513.17s (~25min) |
| #2320 | 2307.101s (~38min) | 1654.34s (~28min) |
| #2277 | 2062.123s (~34min) | 1672.67s (~28min) |
| #2260 | 2351.070s (~39min) | 443.92s (~7min) |

_Assessment_: Gatsby is faster at our slowest step in the build process. It's not a massive improvement, but Gatsby Cloud consistently outperforms.