Gatsby: [v2] Builds are inconsistent

[oltodo]

I realized that builds in Gatsby v2 are inconsistent. It means two builds based on the same code base couldn't provide the same the result (in the public folder). That can cause troubles when building your Gatsby project on multiple load-balanced servers (for instance).

To reproduce the case:

• $ gatsby new my-blazing-fast-site https://github.com/gatsbyjs/gatsby-starter-default\#v2
• $ cd my-blazing-fast-site
• $ yarn build
• $ mv public previous
• $ rm -rf .cache
• $ yarn build
• $ diff -rq previous public

Unless you're lucky, you should have a diff like this:

Which is caused by a random sorting of pages' dataPaths:

[kakadiadarpan]

I'm able to reproduce this. Just create a new Gatsby(v2) project, and follow the steps provided in the description. This doesn't happen in v1.

Environment

System:

OS: macOS High Sierra 10.13.6
CPU: x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
Shell: 3.2.57 - /bin/bash

Binaries:

Node: 10.9.0 - /usr/local/bin/node
Yarn: 1.9.4 - /usr/local/bin/yarn
npm: 6.2.0 - /usr/local/bin/npm

Browsers:

Chrome: 68.0.3440.106
Safari: 11.1.2

npmPackages:

gatsby: next => 2.0.0-rc.9
gatsby-plugin-manifest: next => 2.0.2-rc.1
gatsby-plugin-offline: next => 2.0.0-rc.2
gatsby-plugin-react-helmet: next => 3.0.0-rc.1

npmGlobalPackages:

gatsby-cli: 2.0.0-rc.1

[kakadiadarpan]

Hi @oltodo, can you run this command gatsby info --clipboard and provide the result of that?

[oltodo]

Sure

__System:__

OS: macOS Sierra 10.12.6
CPU: x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Shell: 5.2 - /bin/zsh

__Binaries:__

Node: 10.7.0 - ~/.nvm/versions/node/v10.7.0/bin/node
Yarn: 1.9.4 - ~/.nvm/versions/node/v10.7.0/bin/yarn
npm: 6.4.0 - ~/.nvm/versions/node/v10.7.0/bin/npm

__Browsers:__

Chrome: 68.0.3440.106
Firefox: 59.0.2
Safari: 11.1.2

__npmPackages:__

gatsby: next => 2.0.0-rc.7
gatsby-plugin-manifest: next => 2.0.2-rc.1
gatsby-plugin-offline: next => 2.0.0-rc.2
gatsby-plugin-react-helmet: next => 3.0.0-rc.1

__npmGlobalPackages:__

gatsby-cli: 1.1.58

[kakadiadarpan]

Hi @oltodo, thanks for sharing the environment info. We'll look into this.

[oltodo]

Thanks @kakadiadarpan!

[pieh]

Hey, this is happening because we run graphql queries in parallel and order of inserting entries to dataPaths is not guaranteed. (dataPaths are partial paths to .json files that contain results of graphql queries - files are located in public/static/d/${dataPath}.json).

Here's where we are writing file that has random order of keys: https://github.com/gatsbyjs/gatsby/blob/faf58c8a05453763301f5de631e5efc95abd788e/packages/gatsby/src/internal-plugins/query-runner/pages-writer.js#L120-L126

We could process jsonDataPaths before writing it to ensure keys are sorted and therefore consistent between builds (f.e. using something like _(jsonDataPaths).toPairs().sortBy(0).fromPairs() - not tested). Would you be up for checking this out and creating PR for it @oltodo?

[oltodo]

Hi @pieh, sure I totally can submit a PR. I wanted to be sure that is something possible before starting any code.

Do you think sorting can be applied to the pagesData object too? I've filed an issue a few days ago about random sorting on the aforesaid object. So I was able to solve this by adding a sort filter in my GraphQL queries (used to create new pages) but I think it should be handled by the page-writer.js file.

[pieh]

Right, yeah I think this would be good idea to do this for pagesData as well (I guess sort by path there).

[oltodo]

Fixed through https://github.com/gatsbyjs/gatsby/pull/7878