Docs-website: whats-new-ids.json changes needs local build to update prior to submitting a PR

[austin-schaefer]

Description

Every other part of the site gets updated when we merge from develop to main as part of the build process. But there's a totally bizarre thing where whats-new-ids.json doesn't get updated unless you build _locally_, first.

This has bit us a couple times, most notably with a fix @mmfred did to fix a duplicate post earlier today as she didn't build locally before merging to develop.

Steps to reproduce

1. Create or delete a WN post
2. Build locally
3. Notice that this file changes: what's-new-ids.json
4. merge to develop

5. push to main

6. Create or delete a WN post

7. DO NOT Build locally
8. merge to develop
9. push to main
10. Notice that this file doesn't change: what's-new-ids.json

Expected behavior

local build shouldn't be needed to update what's-new-ids.json and that build process that runs when the site deploys, should detect these changes and update what's-new-ids.json

[austin-schaefer]

@jpvajda @zstix : Odd bug we might want to look at. Not a dealbreaker but it's led to some confusing issues with WN being out of sync. Perhaps more importantly, it's a "gotcha" that's (AFAIK) the only exception to the rule that the process of building locally shouldn't affect the actual build/publish process.

[jpvajda]

@austin-schaefer thanks for reporting this, we'll consider it for Sprint 1 as one of our primary focuses in the next few months is reducing contributor toil.

[jpvajda]

@austin-schaefer It's not clear exactly the issue here can you clarify the following?

1. Create or delete a WN post
2. Build locally

3. Notice that this file changes for some reason

   what file? do you mean the what's-new-ids.json Also, is this what you'd expect as you just modified a file that relates to this JSON file.

   1. Discard that change, open a PR, and try merging
   2. Notice that the json file doesn't get updated

ok, but that's because you discarded the change.

So what is your are expecting to happen here? What is the workflow you are expecting?

[austin-schaefer]

@jpvajda :

Yes, the file is whats-new-ids.json.

There are two issues:

1. A local build is _required_ for this JSON file to update. The actual build/deploy process in Amplify doesn't seem to build this file. That is not true of anything else on the site. This has resulted in several instances of the whats-new.json file being out of date, because no one had rebuilt the site locally and pushed this updated json file.
2. AFAIK, building locally via yarn && yarn start shouldn't result in any changes being picked up by git anyway. No other local build process results in changes to site files that need to be manually pushed up.

[jpvajda]

@austin-schaefer I see.. that helps to clarify. so the expectation here is that if a what's new post is updated in any way, that a local build shouldn't be needed, and that build process that runs when the site deploys, should detect these changes?

[jpvajda]

I clarified and update this issue, since there is a work around, we'll put this work in a future sprint as Sprint 1 has already started.

I'm not 100% sure if we can support this with in the build workflow so I'll have to checking with @zstix on that, if it's a small amount of work, we could consider if for sprint 2. In the mean time i suggest that this is documented for contributors in Style Guide so they know if they are modifying what's new posts a local build is required prior to submitting a PR. - @austin-schaefer

[austin-schaefer]

Example of doing this as part of an otherwise unrelated content edit #1642.

[paperclypse]

Just wanted to add a comment to say that we've been running into a lot of merge conflicts with the what's-new-ids-json file, sometime even in branches where no work is being done related to what's new.

Anything you want to add @zuluecho9?

[zuluecho9]

I didn't read all of above but we are still having some issues/confusions around this. I attempted to pinpoint what seems to be going wrong here with a slack message that I'll repost here:

"so I'm a bit confused about the 'what's news' stuff: based on yesterday's experience, we pushed it up with the new PHP agent what's new post, without anything being added to the js file, and it seems to have worked fine. But when I pull down develop locally and examine the js file, it still doesn't have that php agent entry in the index. So I imagine this means it's building that file when it goes to main, which means our develop branch is out of date. Which I think would explain all the problems we keep having here, in terms of having merge conflicts and such, and having Dayna point out what's new posts in the UI that aren't in docs, and such."

Here's an example of the kinds of duplicate what's new post we keep running into. Also notice there's a lack of comma after that post, which I think is related to the fact that the last entry in the js file lacks a comma, and somehow it's reordering the list. I don't have a clue what's going on, but we are having quite a bit of problems.

[austin-schaefer]

This was causing a fair amount of confusion/churn in our team retro today, would be great to fix the weirdness here.

Per our retro, it sounds like @roadlittledawn confirmed we should _not_ keep/merge changes to this file from local builds.

[jpvajda]

There is a short term fix we can put in place, @roadlittledawn and @paperclypse discussed, if we can clarify that, I can update the AC and we can take this in in Sprint 3, or if urgent enough have a hero take a look.

[roadlittledawn]

yep yep. if we move the code that generates the what's new IDs to a different Gatsby lifecycle like onPostBuild and figure out how we can red those IDs when generating the what's new JSON endpoint, that should help address the main issues here. that's a similar pattern we use for other JSON / RSS XML output via gatsby-plugin-generate-json, gatsby-plugin-generate-doc-json, gatsby-plugin-release-note-rss.

[paperclypse]

I simply confirmed with @roadlittledawn last week that we should be discarding changes to the whats-new-id.json file when we see them. This wasn't consistently being done, I don't think.

[LizBaker]

for what its worth, because i'm not sure an explanation has been given for why this happens:

this json file is needed for an endpoint being used by a whats new app in nr1. it does update on build, as everyone has seen locally, and that includes when it gets deployed. when it's deployed the updates live in memory and you don't see it in the repo, the deploy doesn't have a way to come back and commit the changes, nor does it really need to. just like how a build will create all the static html files for the site but we don't have those committed.
so everything is still working just fine, but updating and committing it manually resulted in some errors/struggles. the file is basically a seed for an automatic process happening in the build, and you can technically update the seed, but that has no value.

that being said, content creators shouldn't have to look at/worry about this, so we're gonna smooth it out!

[jpvajda]

So it sounds like whats-new-ids.json should be treated like related-pages.json That lives on the Dev Site, you never update this file as a contributor, you just let the build handle that. Is this a good way of understanding this @LizBaker ?