Gatsby: Gatsby with large amount of markdown pages

Created on 24 Sep 2018 · 19Comments · Source: gatsbyjs/gatsby

3k markdown pages with Gatsby

Relevant information

I have tried to compile 3k markdown pages with gatsby and it gets heap out of memory issue. I searched and found the benchmark which is mentioned for large amount of markdown pages, but there is no graphql in the gatsby-node.js and not more number of plugins used in the application.

Is this cause of performance issue? If large amount of markdown pages used in an application, should it need to configure minimal plugins and ignore graphql?

How could use large amount of markdown pages with required plugins and config in gatsby?

Environment (if relevant)

 System:
    OS: Windows 10
    CPU: x64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
  Binaries:
    Yarn: 1.3.2 - C:\Users\ajithr\AppData\Roaming\npm\yarn.CMD
    npm: 6.4.1 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: 42.17134.1.0
  npmPackages:
    gatsby: ^2.0.0 => 2.0.7
    gatsby-plugin-catch-links: ^2.0.2 => 2.0.2
    gatsby-plugin-feed: ^2.0.5 => 2.0.5
    gatsby-plugin-google-analytics: ^2.0.5 => 2.0.6
    gatsby-plugin-manifest: ^2.0.2 => 2.0.2
    gatsby-plugin-nprogress: ^2.0.5 => 2.0.5
    gatsby-plugin-offline: ^2.0.5 => 2.0.5
    gatsby-plugin-react-helmet: ^3.0.0 => 3.0.0
    gatsby-plugin-sharp: ^2.0.5 => 2.0.5
    gatsby-plugin-sitemap: ^2.0.1 => 2.0.1
    gatsby-plugin-styled-components: ^3.0.0 => 3.0.0
    gatsby-plugin-typography: ^2.2.0 => 2.2.0
    gatsby-remark-autolink-headers: ^2.0.6 => 2.0.6
    gatsby-remark-copy-linked-files: ^2.0.5 => 2.0.5
    gatsby-remark-images: ^2.0.1 => 2.0.1
    gatsby-remark-prismjs: ^3.0.0 => 3.0.0
    gatsby-remark-responsive-iframe: ^2.0.5 => 2.0.5
    gatsby-remark-smartypants: ^2.0.5 => 2.0.5
    gatsby-source-filesystem: ^2.0.1 => 2.0.1
    gatsby-transformer-json: ^2.1.1 => 2.1.1
    gatsby-transformer-remark: ^2.1.1 => 2.1.3
    gatsby-transformer-sharp: ^2.1.1 => 2.1.1

File contents (if changed)

gatsby-config.js:

const config = require("./src/templates/site-config");

const pathPrefix = config.pathPrefix === "/" ? "" : config.pathPrefix;

module.exports = {
  siteMetadata: {
    siteUrl: config.siteUrl + pathPrefix
  },
  pathPrefix: config.pathPrefix,
  plugins: [
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `${__dirname}/src/pages`,
        name: 'pages',
        ignore: [`templates/**/`]
      },
    },
    {
      resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 590,
            },
          },
          {
            resolve: `gatsby-remark-responsive-iframe`,
            options: {
              wrapperStyle: `margin-bottom: 1.0725rem`,
            },
          },
          'gatsby-remark-prismjs',
          'gatsby-remark-copy-linked-files',
          'gatsby-remark-smartypants',
          'gatsby-remark-autolink-headers'
        ],
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    `gatsby-plugin-catch-links`,
    `gatsby-plugin-sitemap`,
    {
      resolve: `gatsby-plugin-google-analytics`,
      options: {
        trackingId: config.googleAnalyticsID,
      },
    },
    {
      resolve: "gatsby-plugin-nprogress",
      options: {
        color: config.themeColor
      }
    },
    `gatsby-plugin-feed`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: config.siteTitle,
        short_name: config.siteTitle,
        description: config.siteDescription,
        start_url: config.pathPrefix,
        background_color: config.backgroundColor,
        theme_color: config.themeColor,
        display: "minimal-ui",
        icons: [
          {
            src: "/logos/logo-192x192.png",
            sizes: "192x192",
            type: "image/png"
          },
          {
            src: "/logos/logo-512x512.png",
            sizes: "512x512",
            type: "image/png"
          }
        ]
      }
    },
    `gatsby-plugin-offline`,
    `gatsby-plugin-styled-components`,
    `gatsby-plugin-react-helmet`,
    {
      resolve: 'gatsby-plugin-typography',
      options: {
        pathToConfigModule: 'src/utils/typography',
      },
    },
  ],
}

package.json:

{
  "name": "gatsby-starter-blog",
  "description": "Starter Gatsby Blog",
  "version": "1.0.0",
  "author": "Kyle Mathews <[email protected]>",
  "bugs": {
    "url": "https://github.com/gatsbyjs/gatsby-starter-blog/issues"
  },
  "dependencies": {
    "babel-plugin-styled-components": "^1.7.1",
    "core-js": "^2.5.7",
    "gatsby": "^2.0.0",
    "gatsby-plugin-catch-links": "^2.0.2",
    "gatsby-plugin-feed": "^2.0.5",
    "gatsby-plugin-google-analytics": "^2.0.5",
    "gatsby-plugin-manifest": "^2.0.2",
    "gatsby-plugin-nprogress": "^2.0.5",
    "gatsby-plugin-offline": "^2.0.5",
    "gatsby-plugin-react-helmet": "^3.0.0",
    "gatsby-plugin-sharp": "^2.0.5",
    "gatsby-plugin-sitemap": "^2.0.1",
    "gatsby-plugin-styled-components": "^3.0.0",
    "gatsby-plugin-typography": "^2.2.0",
    "gatsby-remark-autolink-headers": "^2.0.6",
    "gatsby-remark-copy-linked-files": "^2.0.5",
    "gatsby-remark-images": "^2.0.1",
    "gatsby-remark-prismjs": "^3.0.0",
    "gatsby-remark-responsive-iframe": "^2.0.5",
    "gatsby-remark-smartypants": "^2.0.5",
    "gatsby-source-filesystem": "^2.0.1",
    "gatsby-transformer-json": "^2.1.1",
    "gatsby-transformer-remark": "^2.1.1",
    "gatsby-transformer-sharp": "^2.1.1",
    "lodash": "^4.17.11",
    "prismjs": "^1.15.0",
    "react": "^16.5.1",
    "react-dom": "^16.5.1",
    "react-helmet": "^5.2.0",
    "react-router-dom": "^4.3.1",
    "react-typography": "^0.16.13",
    "styled-components": "^3.4.9",
    "typeface-merriweather": "0.0.43",
    "typeface-montserrat": "0.0.43",
    "typography": "^0.16.17",
    "typography-theme-wordpress-2016": "^0.15.10"
  },
  "devDependencies": {
    "eslint": "^4.19.1",
    "eslint-plugin-react": "^7.11.1",
    "gh-pages": "^1.2.0",
    "gulp": "^3.9.1",
    "json-loader": "^0.5.7",
    "prettier": "^1.14.2",
    "shelljs": "^0.8.2"
  },
  "homepage": "https://github.com/gatsbyjs/gatsby-starter-blog#readme",
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "main": "n/a",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/gatsbyjs/gatsby-starter-blog.git"
  },
  "scripts": {
    "dev": "gatsby develop",
    "lint": "./node_modules/.bin/eslint --ext .js,.jsx --ignore-pattern public .",
    "test": "echo \"Error: no test specified\" && exit 1",
    "format": "prettier --trailing-comma es5 --no-semi --single-quote --write 'src/**/*.js' 'src/**/*.md'",
    "develop": "gatsby develop",
    "build": "gatsby build",
    "deploy": "gatsby build --prefix-paths && gh-pages -d public",
    "fix-semi": "eslint --quiet --ignore-pattern node_modules --ignore-pattern public --parser babel-eslint --no-eslintrc --rule '{\"semi\": [2, \"never\"], \"no-extra-semi\": [2]}' --fix gatsby-node.js"
  }
}

gatsby-node.js:

const _ = require('lodash')
const Promise = require('bluebird')
const path = require('path')
const { createFilePath } = require('gatsby-source-filesystem')

var fs = require('fs');
var shelljs = require('shelljs');
var createTab = require('./build/tab.js').createTab;
var common = require('./build/common.js');

var gcsecx = null;
exports.onCreateNode = ({ node, actions, getNode }) => {
    const { createNodeField } = actions
    let slug;

    if (node.internal.type === `MarkdownRemark`) {
        const value = createFilePath({ node, getNode })
        const fileNode = getNode(node.parent);
        const parsedFilePath = path.parse(fileNode.relativePath);
        if (node.internal.content) {
            node.internal.content = createTab(node);
        }
        if (parsedFilePath.name !== "index" && parsedFilePath.dir !== "") {
            slug = `/${parsedFilePath.dir}/${parsedFilePath.name}/`;
        } else if (parsedFilePath.dir === "") {
            slug = `/${parsedFilePath.name}/`;
        } else {
            slug = `/${parsedFilePath.dir}/`;
        }
        if (!gcsecx) {
            gcsecx = common.getGcse();
        }
        createNodeField({ node, name: "parentPath", value: parsedFilePath.dir });
        createNodeField({ node, name: "childPath", value: parsedFilePath.name });
        createNodeField({ node, name: "slug", value: slug });
        createNodeField({ node, name: "gcse", value: gcsecx });
        createNodeField({ node, name: "header", value: getHeader(node.rawMarkdownBody.match(/[#]+ [^\n]+/g)) });
    }
}

exports.createPages = ({ graphql, actions }) => {
    const { createPage } = actions

    return new Promise((resolve, reject) => {
        const layout = path.resolve('./src/templates/layout.js')
        resolve(
            graphql(
                `
          {
            allMarkdownRemark {
              edges {
                node {
                      html
                      id
                      frontmatter {
                          title
                          description
                          component
                      }
                      fields {
                          slug
                          parentPath
                          childPath
                          gcse
                          header
                      }
                  }
              }
            }
          }
        `
            ).then(result => {
                if (result.errors) {
                    console.log(result.errors)
                    reject(result.errors)
                }
                let chapters = [];
                // Create blog posts pages.
                const posts = result.data.allMarkdownRemark.edges;

                for (var i = 0; i < posts.length; i++) {
                    if (!chapters.length) {
                        chapters = getChapters(posts);
                    }
                    var post = posts[i];
                    const previous = i === posts.length - 1 ? null : posts[i + 1].node;
                    const next = i === 0 ? null : posts[i - 1].node;

                    createPage({
                        path: post.node.fields.slug,
                        component: layout,
                        context: {
                            slug: post.node.fields.slug,
                            previous,
                            next,
                            chapters: chapters,
                            gcse: post.node.fields.gcse,
                            header: post.node.fields.header
                        },
                    })
                }
            })
        )
    })
}

let getChapters = (chapters) => {
    var data = require('./data');
    let treeChapters = [],
        parents = [];
    let common = {

    }

    chapters.forEach(chapter => {
        let parentPath = chapter.node.fields.parentPath;
        if (parents.indexOf(parentPath) !== -1) {
            return;
        }
        parents.push(parentPath);
        let dataManager = new data.DataManager(chapters);
        let query = new data.Query().where('node.fields.parentPath', 'equal', parentPath);
        let childrens = dataManager.executeLocal(query);

        let parentId = chapter.node.frontmatter.component;
        if (chapter.node.frontmatter.component === 'Common') {
            parentId = common[parentPath];
        }

        if (parentId && parentId.length) {
            var childObj = getChildOrder(parentPath, childrens);
            let parentObj = {
                id: parentPath,
                child: childObj,
                name: parentId
            }
            treeChapters.push(parentObj);
        } else {
            childrens = getChildOrder(parentPath, childrens);
            for (var i = 0; i < childrens.length; i++) {
                treeChapters.push(childrens[i]);
            }
        }
    });
    return treeChapters;
}

let getChildOrder = (parentPath, childrens) => {
    var parent = parentPath.length ? '/' + parentPath : parentPath;
    var summaryPath = './src/pages' + parent + '/summary.txt';
    if (fs.existsSync(summaryPath)) {
        var orderedList = [];
        var summaryFile = fs.readFileSync(summaryPath, 'utf8').replace(/.md/g, '');
        var orders = summaryFile.split('\r\n');
        for (var i = 0; i < orders.length; i++) {
            for (j = 0; j < childrens.length; j++) {
                let childPath = childrens[j].node.fields.childPath;
                let childFile = './src/pages' + parent + '/' + childPath + '.md';
                if (!fs.existsSync(childFile)) {
                    console.log(childFile + ' not found');
                    continue;
                }
                if (orders[i] === childPath) {
                    orderedList.push({
                        parentId: parentPath,
                        id: parentPath + '/' + childPath,
                        name: childrens[j].node.frontmatter.nodeName || getChildName(childPath)
                    });
                }
            }
        }
        return orderedList;
    } else {
        return childrens;
    }
}

let getChildName = (childName) => {
    var splitted = childName.split('-');
    var name = splitted[0].charAt(0).toUpperCase() + splitted[0].slice(1);
    for (var i = 1; i < splitted.length; i++) {
        name = name + splitted[i].charAt(0).toUpperCase() + splitted[i].slice(1);
    }
    return name;
}

let getHeader = (headers) => {
    var toc = '';
    if (headers && headers.length) {
        toc = '<span>Contents</span>\n<ul>';
        for (var header of headers) {
            var hLevel = header.match(/#/g).length;
            if (hLevel > 1) {
                var hName = header.replace(/[#]+ /, '').trim();
                var hId = hName.toLowerCase().replace(/[ ]+/g, '-');
                toc += `\n  <li class="doc-anchor-h${hLevel}"><a href="#${hId}">${hName}</a></li>`;
            }
        }
        toc += '\n</ul>';
    }
    return toc;
}

Error

```success source and transform nodes — 126.943 s
success building schema — 4.738 s
⠐ createPages
<--- Last few GCs --->

[17700:000001A072EE6310] 1338918 ms: Mark-sweep 1373.9 (1570.6) -> 1373.9 (1570.6) MB, 1924.8 / 0.1 ms allocation failure GC in old space requested
[17700:000001A072EE6310] 1340517 ms: Mark-sweep 1373.9 (1570.6) -> 1373.9 (1549.1) MB, 1591.3 / 0.1 ms last resort GC in old space requested
[17700:000001A072EE6310] 1342166 ms: Mark-sweep 1373.9 (1549.1) -> 1373.9 (1536.1) MB, 1648.4 / 0.1 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000017216425879
1: /* anonymous */ [D:\Important\apps\docs\node_modules\lodash\defaults.js:~33] [pc=000003B2EEFC8D81](this=00000297B2175861

Gatsby: Gatsby with large amount of markdown pages

3k markdown pages with Gatsby

Relevant information

Environment (if relevant)

File contents (if changed)

Error

Most helpful comment

All 19 comments

Related issues