React: Move website source to new reactjs.org repo

[bvaughn]

Now that the new website has launched I propose we do the following clean-up steps:

• Move Gatsby source and dynamic content (*.md and *.yml files) to new repo
• Update external tools like Netlify and Crowdin to use this new repo
• Delete Jekyll website source

This will offer several benefits, including:

• Improve focus and simplify triage process for both the website and framework repos
• Cut down on noise from Netlify preview comments on PRs that aren't related to website

I'm creating this issue mostly to facilitate discussion around the new proposed directory structure (below).

Proposed Directory Structure

├── content                              # Markdown and YML source
│   ├── blog
│   │   ├── markdown-files-here...
│   │   └── nav.yml
│   ├── community
│   │   ├── markdown-files-here...
│   │   └── nav.yml
│   ├── docs
│   │   ├── markdown-files-here...
│   │   └── nav.yml
│   ├── tutorial
│   │   ├── markdown-files-here...
│   │   └── nav.yml
│   ├── acknowledgements.yml
│   ├── authors.yml
│   └── index.md
├── plugins                              # Custom plug-ins
│   ├── gatsby-remark-use-jsx
│   └── gatsby-transformer-authors-yaml
├── src                                  # Gatsby site source goes here
│   ├── components
│   ├── css
│   ├── layouts
│   ├── pages
│   └── templates
├── static                               # Images and other statically-linked content
│   └── large-logo.svg
├── CONTRIBUTING.md
├── LICENSE.md
├── README.md
└── package.json

cc @ericnakagawa, @flarnie, @joecritch

[joecritch]

Liking this! I think this will streamline efforts from both sides.

[Mathspy]

Another benefit could be that since you're moving the docs (along with their license) is being able to display the MIT license in the navigation bar per this discussion: https://github.com/facebook/react/pull/10891

[gaearon]

There's a whole lot of benefits. My favorite one is hopefully people will stop trying to read docs on GitHub. We'll also reduce the issue noise and hopefully have more community members with commit access. I'm unreasonably excited about this.

@bvaughn Let me know if I can help. I love deleting/moving things. Better than real work!

[bvaughn]

Let me know if I can help. I love deleting/moving things. Better than real work!

😆 Thanks Dan! I enjoy it too. I'll take care of this one. 😁

[yangshun]

@bvaughn Would you require help in this effort? I am keen to help if you do.

[bvaughn]

Thanks for the offer, but I think I'm good for the bulk of it. Help would be great once the project has been moved and I file a bunch of issues. 😄

[bvaughn]

One thing I did not consider when creating this issue was how to best keep scripts/error-codes/codes.json in sync.

To a lesser extent, this also applies to the way we read the current React version from the package.json file but that was already a little janky. I think it's okay for us to just move this to the site-constants file and accept the fact that we'll need to bump it now and again. (Please let me know if this seems too onerous.)

I see a couple of options. I'm currently leaning toward the first one.

Option 1

The website's build scripts could load the latest error codes from the facebook/react repo at build time. I think this could be done by way of a Gatbsy source plugin.

• pros:
  
  
  
  • Adds no complexity to the facebook/react repo
  
  
  • No change in user-facing behavior or performance on website
  
  
• cons:
  
  
  
  • Additional point of failure for reactjs.org repo CI (eg GitHub API is down, error codes JSON gets moved, etc)
  
  
  • Website must be separately re-built whenever error codes are updated
  
  

Option 2

The ErrorDecoder could just-in-time load the latest error codes from the facebook/react repo when the user opens the error page.

• pros:
  
  
  
  • Website automatically stays in-sync with the latest error codes when they are updated
  
  
  • Doesn't add complexity to build step
  
  
• cons:
  
  
  
  • Added latency for user's looking up error messages
  
  
  • Additional point of failure for reactjs.org website (eg GitHub API is down, error codes JSON gets moved, etc)
  
  

Option 3

We could fork the error codes JSON and store a copy in the new repo.

• pros:
  
  
  
  • More flexibility over which error codes are live on the website
  
  
• cons:
  
  
  
  • Too easy to forget about the forked file during a release
  
  

[bvaughn]

I wrote a Gatsby source plugin that loads the data from the React repo (using GitHub's user content API):

const request = require('request-promise');

const errorCodesUrl = 'http://raw.githubusercontent.com/facebook/react/master/scripts/error-codes/codes.json';

exports.sourceNodes = async ({ boundActionCreators }) => {
  const { createNode } = boundActionCreators;

  const jsonString = await request(errorCodesUrl);

  createNode({
    id: 'error-codes',
    children: [],
    parent: 'ERRORS',
    internal: {
      type: 'ErrorCodesJson',
      contentDigest: jsonString,
    },
  });
};

I think this is reasonable for now. Let's talk more if anyone has concerns though.

[gaearon]

Sounds good to me. (But does this mean we can't run website offline now?)

[bvaughn]

Sounds good to me. (But does this mean we can't run website offline now?)

To run the website offline we'll need to first re-enable service workers (and figure out why they were causing intermittent problems). But this change won't preclude that. The plug-in I wrote embeds the data at build (or dev startup) time so that the end result will be the same as before.

[bvaughn]

I believe it should be possible to preserve history in this new repo by doing the following:

# git filter-branch only supports a single folder
# So start by creating one pruned repo per folder we want to preserve
# Copy history from the "docs" folder
# (This step will take a long time because there's a lot of history to copy)
git clone [email protected]:facebook/react.git react-docs
cd react-docs
git filter-branch --prune-empty --subdirectory-filter docs master

cd ..

# Copy history from the "www" folder
git clone [email protected]:facebook/react.git react-www
cd react-www
git filter-branch --prune-empty --subdirectory-filter www master

cd ..

# Create a new repo to unify our filtered repos
mkdir reactjs.org
cd reactjs.org
git init

# At least one commit is needed before the below steps will work
git commit --allow-empty -m "Initial (empty) commit"

# Merge docs changes
git remote add react-docs ../react-docs
git fetch react-docs
git merge -s ours --no-commit react-docs/master --allow-unrelated-histories
git read-tree --prefix=docs -u react-docs/master
git commit -m "Import docs"

# Merge www changes
git remote add react-www ../react-www
git fetch react-www
git merge -s ours --no-commit react-www/master --allow-unrelated-histories
git read-tree --prefix=www -u react-www/master
git commit -m "Import www"

# Move things around (as outlined in the issue description) commit by commit

Edit 1: git log --follow seems to suggest this didn't work correctly. This may be related to the --allow-unrelated-histories flag. Not sure.

Edit 2: I was able to get this working by only copying history for the docs folder. I think that's okay since the www folder is much newer and its history is less useful.

[Daniel15]

The website's build scripts could load the latest error codes from the facebook/react repo at build time

We do something similar for the Babel site - It loads all the package readmes from the main Babel repo as part of the build. The script is here: https://github.com/babel/website/blob/master/scripts/download-readmes.js

I believe it should be possible to preserve history in this new repo by doing the following

I'm glad you could work this out. I was splitting one of Yarn's repositories into two separate ones (https://github.com/yarnpkg/releases and https://github.com/yarnpkg/release-infra) and it took forever to work out how to actually do it 😛

[bvaughn]

Gatsby's plug-in system made this ridiculously easy to do. 😄

1. Pull data into graphql via a plug-in: https://github.com/reactjs/reactjs.org/blob/master/plugins/gatsby-source-react-error-codes/gatsby-node.js
2. Read data into the template from graphql: https://github.com/reactjs/reactjs.org/blob/be1f9a60d80261eb70eded915b266078b606524c/src/pages/docs/error-decoder.html.js#L101-L118