Gatsby: Converting site from remark to mdx increases compile and build times up to 8x

[janosh]

I just ported my entire personal site from Remark to MDX. Compile times went from 17 sec (Remark) to 135 sec (MDX) and build times increased from 32 sec to 193 sec (in each case as measured by an actual clock).

This is way more than I expected so I was wondering if perhaps I did something wrong in following the MDX guide and if there is potential for easy optimization.

Relevant information

Here's the commit that made the transition.

Environment

  System:
    OS: macOS 10.14.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 12.9.1 - /usr/local/bin/node
    Yarn: 1.17.3 - /usr/local/bin/yarn
    npm: 6.10.3 - /usr/local/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 76.0.3809.87
    Firefox: 68.0.2
    Safari: 12.1.2
  npmPackages:
    gatsby: 2.15.1 => 2.15.1 
    gatsby-image: 2.2.15 => 2.2.15 
    gatsby-plugin-algolia: ^0.3.3 => 0.3.3 
    gatsby-plugin-catch-links: 2.1.6 => 2.1.6 
    gatsby-plugin-favicon: ^3.1.6 => 3.1.6 
    gatsby-plugin-google-analytics: 2.1.12 => 2.1.12 
    gatsby-plugin-lodash: ^3.1.5 => 3.1.5 
    gatsby-plugin-manifest: 2.2.12 => 2.2.12 
    gatsby-plugin-mdx: ^1.0.35 => 1.0.35 
    gatsby-plugin-netlify-cache: ^1.2.0 => 1.2.0 
    gatsby-plugin-offline: 3.0.0 => 3.0.0 
    gatsby-plugin-react-helmet: 3.1.5 => 3.1.5 
    gatsby-plugin-sharp: 2.2.18 => 2.2.18 
    gatsby-plugin-styled-components: 3.1.3 => 3.1.3 
    gatsby-remark-autolink-headers: ^2.1.8 => 2.1.8 
    gatsby-remark-code-titles: ^1.1.0 => 1.1.0 
    gatsby-remark-copy-linked-files: ^2.1.13 => 2.1.13 
    gatsby-remark-embed-video: 1.7.1 => 1.7.1 
    gatsby-remark-emojis: ^0.3.2 => 0.3.2 
    gatsby-remark-images: 3.1.19 => 3.1.19 
    gatsby-remark-katex: ^3.1.6 => 3.1.6 
    gatsby-remark-responsive-iframe: 2.2.10 => 2.2.10 
    gatsby-remark-smartypants: 2.1.5 => 2.1.5 
    gatsby-remark-sub-sup: ^1.0.0 => 1.0.0 
    gatsby-remark-vscode: ^1.2.0 => 1.2.0 
    gatsby-source-filesystem: 2.1.18 => 2.1.18 
    gatsby-transformer-remark: 2.6.19 => 2.6.19 
    gatsby-transformer-sharp: 2.2.12 => 2.2.12 
    gatsby-transformer-yaml: 2.2.7 => 2.2.7 

Perhaps related

I also started getting a bunch of messages during the build process:

[BABEL] Note: The code generator has deoptimised the styling of undefined as it exc
eeds the max of 500KB.

[Mrazator]

I am having the same problem as you. However, as I am working with a lot of files (thousands of them and some of them are thousands of lines long), it is a blocker for me (as I cannot afford to have 40min+ build times). It also seems to me, that the cache mechanism just does not work with so many files - as whatever I change, It always starts the build process and re-generation of cache again. Btw. I am also encountering the same Babel message (Error) and it drives me crazy - as I cannot get rid of it by any of these:

• setting compact: true in .babelrc
• setting NODE_OPTIONS=--max_old_space_size=4096
• ???

I also posted the problem to the similar issue in gatsby-mdx, more info here: https://github.com/ChristopherBiscardi/gatsby-mdx/issues/411

[janosh]

@johno Let me know if I can try and help implement the static flag for MDX elements you mentioned at Gatsby Days London.

Also, how difficult do you think it would be to convert or extend gatsby-remark-vscode (the syntax highlighter I'm using ) into something that is based on React components like prism-react-renderer? And could that then be used in MDX's compositional approach to syntax highlighting?

[johno]

@Mrazator can you include which plugins you're using?

It also seems to me, that the cache mechanism just does not work with so many files - as whatever I change, It always starts the build process and re-generation of cache again.

This definitely sounds like a bug since we shouldn't need to rebuild the entire cache when a single file changes 🤔

Btw. I am also encountering the same Babel message (Error) and it drives me crazy

This typically only occurs with plugins that add a large amount of DOM nodes to the document (which we've only seen from syntax highlighting plugins so far).

---

Let me know if I can try and help implement the static flag for MDX elements you mentioned at Gatsby days London.

I'll set aside some time this week to write up an RFC so we can discuss implementation 👍

Also, how difficult do you think it would be to convert or extend gatsby-remark-vscode (the syntax highlighter I'm using ) into something that is based on React components like prism-react-renderer and could be used in MDX's compositional approach to syntax highlighting?

@janosh I'm not very familiar with gatsby-remark-vscode myself, but it looks like it handles lots of things like downloading extensions for other languages so I suspect it might be tricky to get it to run on the client (and potentially too large of a footprint).

---

I'll be digging into this issue over the next week and see what the bottlenecks are. MDX is doing a lot more so it __will__ result in longer build times, but it shouldn't be this significant so we'll look into optimizations to get the times down. There hasn't been much work done in this area so there's prolly some low hanging fruit.

[janosh]

@janosh I'm not very familiar with gatsby-remark-vscode myself, but it looks like it handles lots of things like downloading extensions for other languages so I suspect it might be tricky to get it to run on the client (and potentially too large of a footprint).

I'm confused now. How is the compositional approach related to running on the client? I thought the benefit of composition would stem from not ending up with a large number of DOM nodes generated by some remark plugin (which MDX then needs to handle) by instead rerouting certain parts (i.e. code blocks) of the MDX file straight into the package that knows what to do with them so that MDX can just forget about them.

[johno]

I'm confused now. How is the compositional approach related to running on the client? I thought the benefit of composition would stem from not ending up with a large number of DOM nodes generated by some remark plugin that MDX then needs to handle by instead rerouting certain parts (i.e. code blocks) of the MDX file straight into the package that knows what to do with them so that MDX can just forget about them.

The compositional approach occurs via the MDXProvider so it is rendered on the server during build _and_ on the client after rehydration. It becomes part of the site's bundle. This is why we require a "static" mechanism where we can "compile away" the client portion (which was what we discussed a bit in person last week).

Not sure if that clarified things or not?

[janosh]

It certainly helped. I guess I need to do a little more reading to understand how MDX really works...

[Mrazator]

Thanks for quick response.

@Mrazator can you include which plugins you're using?

Here you go:

"dependencies": {
    "@emotion/core": "^10.0.17",
    "@emotion/styled": "^10.0.17",
    "@mdx-js/mdx": "^1.5.0",
    "@mdx-js/react": "^1.5.0",
    "gatsby": "^2.15.28",
    "gatsby-cli": "^2.7.53",
    "gatsby-image": "2.2.23",
    "gatsby-plugin-antd": "^2.0.2",
    "gatsby-plugin-catch-links": "^2.1.12",
    "gatsby-plugin-emotion": "^4.1.9",
    "gatsby-plugin-less": "^3.0.9",
    "gatsby-plugin-manifest": "2.2.20",
    "gatsby-plugin-mdx": "^1.0.46",
    "gatsby-plugin-offline": "3.0.11",
    "gatsby-plugin-react-helmet": "3.1.10",
    "gatsby-plugin-sharp": "^2.2.27",
    "gatsby-remark-autolink-headers": "2.1.13",
    "gatsby-remark-images": "^3.1.25",
    "gatsby-remark-images-medium-zoom": "^1.2.1",
    "gatsby-remark-images-zoom": "^1.1.0",
    "gatsby-remark-mathjax": "^1.0.0",
    "gatsby-remark-prismjs": "^3.3.16",
    "gatsby-remark-relative-images": "^0.2.3",
    "gatsby-source-filesystem": "2.1.28",
    "gatsby-transformer-remark": "^2.6.27",
    "gatsby-transformer-sharp": "2.2.19",
    ...
  },
  "devDependencies": {
    "@babel/core": "^7.6.2",
    "@babel/plugin-proposal-class-properties": "^7.4.4",
    "@babel/plugin-syntax-dynamic-import": "^7.2.0",
    "@babel/plugin-transform-runtime": "^7.6.2",
    "@babel/preset-env": "^7.6.2",
    "@babel/preset-react": "^7.0.0",
    "@babel/preset-stage-0": "^7.0.0",
    "@babel/register": "^7.6.2",
    "@types/react": "16.9.4",
    "@types/react-dom": "16.9.1",
    "@types/react-helmet": "5.0.11",
    "babel-cli": "^6.26.0",
    "babel-eslint": "^10.0.3",
    "babel-loader": "^8.0.6",
    "babel-minify-webpack-plugin": "^0.3.1",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "eslint": "^6.5.1",
    "eslint-config-airbnb": "^18.0.1",
    "eslint-config-airbnb-base": "^14.0.0",
    "eslint-config-prettier": "^6.3.0",
    "eslint-config-react": "^1.1.7",
    "eslint-plugin-babel": "^5.3.0",
    "eslint-plugin-import": "^2.17.2",
    "eslint-plugin-jsx-a11y": "^6.2.1",
    "eslint-plugin-prettier": "^3.1.1",
    "eslint-plugin-react": "^7.15.0",
    "eslint-plugin-react-hooks": "^2.1.1",
    "eslint-plugin-relay": "^1.3.12",
    "gatsby-plugin-remove-trailing-slashes": "^2.1.9",
    "webpack": "^4.0.0",
   ...

  }

This typically only occurs with plugins that add a large amount of DOM nodes to the document (which we've only seen from syntax highlighting plugins so far).

I tested it without using prismjs and emotions, and still have the same problem. Maybe the actual document is just too big (6K+ lines)?

[johno]

I tested it without using prismjs and emotions, and still have the same problem. Maybe the actual document is just too big (6K+ lines)?

Hmmm, the compiled output of a 6k line document should be much less than 500KB if prismjs is turned off. Do you mind creating a minimal reproduction I could debug @Mrazator?

[Mrazator]

Do you mind creating a minimal reproduction I could debug @Mrazator?

I will try to do it asap :)

[Mrazator]

JFYI @johno
I am sorry that I didn't answer yet but it looks that I am not going to be able to create a minimal reproduction in the nearest future.

I partially "solved" the problem by splitting those big files into smaller ones.

[johno]

I'm going to close this in favor of #22521 where we're tracking additional performance work on gatsby-plugin-mdx, and #25066 where we're tracking plugin compatibility with MDX which will be related to prism/vscode plugins.