Gatsby: Running gatsby build on fresh v2 install results in a large number of static subdirectories

[kmcaloon]

Description

Running gatsby build on all fresh installs using v2 results in the creation of the directory public/static/d with 998 empty subdirectories numbered 1-998.

Steps to reproduce

Install gatsby from scratch or starter theme, in this case I used gatsby new my-site https://github.com/gatsbyjs/gatsby-starter-hello-world#2

Run gatsby build

Environment

  System:
    OS: OS X El Capitan 10.11.6
    CPU: x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 9.5.0 - /usr/local/bin/node
    Yarn: 1.3.2 - ~/.yarn/bin/yarn
    npm: 5.6.0 - /usr/local/bin/npm
  Browsers:
    Chrome: 67.0.3396.99
    Firefox: 59.0.2
    Safari: 11.1.2
  npmPackages:
    gatsby: ^2.0.0-beta.29 => 2.0.0-beta.34 
    gatsby-image: ^2.0.0-beta.6 => 2.0.0-beta.6 
    gatsby-link: ^2.0.0-beta.4 => 2.0.0-beta.4 
    gatsby-plugin-catch-links: ^2.0.2-beta.2 => 2.0.2-beta.2 
    gatsby-plugin-google-analytics: ^2.0.0-beta.2 => 2.0.0-beta.2 
    gatsby-plugin-manifest: ^2.0.2-beta.2 => 2.0.2-beta.2 
    gatsby-plugin-offline: ^2.0.0-beta.3 => 2.0.0-beta.3 
    gatsby-plugin-react-helmet: v2 => 2.0.11 
    gatsby-plugin-sass: ^2.0.0-beta.5 => 2.0.0-beta.5 
    gatsby-plugin-sharp: ^2.0.0-beta.3 => 2.0.0-beta.5 
    gatsby-plugin-sitemap: ^2.0.0-beta.2 => 2.0.0-beta.2 
    gatsby-remark-images: ^2.0.1-beta.4 => 2.0.1-beta.6 
    gatsby-source-filesystem: ^2.0.1-beta.3 => 2.0.1-beta.3 
    gatsby-transformer-remark: ^2.1.1-beta.3 => 2.1.1-beta.3 
    gatsby-transformer-sharp: ^2.1.1-beta.3 => 2.1.1-beta.3 
  npmGlobalPackages:
    gatsby-cli: 1.1.58

File contents

gatsby-config.js:
```{
"name": "custom-site",
"version": "2.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"classnames": "^2.2.6",
"gatsby": "^2.0.0-beta.29",
"gatsby-image": "^2.0.0-beta.6",
"gatsby-link": "^2.0.0-beta.4",
"gatsby-plugin-catch-links": "^2.0.2-beta.2",
"gatsby-plugin-google-analytics": "^2.0.0-beta.2",
"gatsby-plugin-manifest": "^2.0.2-beta.2",
"gatsby-plugin-offline": "^2.0.0-beta.3",
"gatsby-plugin-react-helmet": "v2",
"gatsby-plugin-sass": "^2.0.0-beta.5",
"gatsby-plugin-sharp": "^2.0.0-beta.3",
"gatsby-plugin-sitemap": "^2.0.0-beta.2",
"gatsby-remark-images": "^2.0.1-beta.4",
"gatsby-source-filesystem": "^2.0.1-beta.3",
"gatsby-transformer-remark": "^2.1.1-beta.3",
"gatsby-transformer-sharp": "^2.1.1-beta.3",
"mobx": "^5.0.3",
"mobx-react": "^5.2.3",
"prop-types": "^15.6.2",
"react": "^16.4.1",
"react-dom": "^16.4.1",
"react-helmet": "^5.2.0",
"react-loadable": "^5.4.0"
},
"scripts": {
"start": "npm run develop",
"build": "gatsby build",
"develop": "gatsby develop",
"serve": "gatsby serve"
}
}

package.json:

{
"name": "custom-site",
"version": "2.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"classnames": "^2.2.6",
"gatsby": "^2.0.0-beta.29",
"gatsby-image": "^2.0.0-beta.6",
"gatsby-link": "^2.0.0-beta.4",
"gatsby-plugin-catch-links": "^2.0.2-beta.2",
"gatsby-plugin-google-analytics": "^2.0.0-beta.2",
"gatsby-plugin-manifest": "^2.0.2-beta.2",
"gatsby-plugin-offline": "^2.0.0-beta.3",
"gatsby-plugin-react-helmet": "v2",
"gatsby-plugin-sass": "^2.0.0-beta.5",
"gatsby-plugin-sharp": "^2.0.0-beta.3",
"gatsby-plugin-sitemap": "^2.0.0-beta.2",
"gatsby-remark-images": "^2.0.1-beta.4",
"gatsby-source-filesystem": "^2.0.1-beta.3",
"gatsby-transformer-remark": "^2.1.1-beta.3",
"gatsby-transformer-sharp": "^2.1.1-beta.3",
"mobx": "^5.0.3",
"mobx-react": "^5.2.3",
"prop-types": "^15.6.2",
"react": "^16.4.1",
"react-dom": "^16.4.1",
"react-helmet": "^5.2.0",
"react-loadable": "^5.4.0"
},
"scripts": {
"start": "npm run develop",
"build": "gatsby build",
"develop": "gatsby develop",
"serve": "gatsby serve"
}
}
```

[jquense]

Can confirm, it's a bit alarming :p

[ghost]

I also have the same issue. Public/static/d has 998 empty folders.

[pieh]

It's result of https://github.com/gatsbyjs/gatsby/pull/6226/commits/266f5ae81c52a69e436115826e39dade35d3f3c4

so it's not strictly a bug.

And some context for this:

kylemathews:
turns out that it gets way slower to write large numbers of files to the same directory e.g. 1000+
went from ~1.2 ms / file to ~12 ms / file
so now randomly splitting files into 100 folders haha
now 650% faster :smile:
to build a 10k page site
Just built a 10k site (w/ tiny files and no queries) in 39 seconds :tada:

But this could probably be adjusted and dependent on page count and not hardcoded to 999 subdirs (which is suitable for very large websites).

[KyleAMathews]

Yeah, we could create the directories on demand so there's not quite as many.

I do find it interesting that people look so much around Gatsby's public folder. I never pay it any attention at all — it'd be like gawking at a binary file a compiler generates ;-)