Gatsby: "update schema" step really slow after upgrading from 2.17 to 2.18
Description
Site with 20k pages, currently using version 2.17.6. We pass JSON data via context (hence not using GraphQL), so we have switched off type inference for SitePage.context to speed up compilation, as stated here.
These are the timings we get using 2.17.6:
2.17.6 timings
=======================================
success source and transform nodes - 4.247s
success building schema - 1.292s
success createPages - 23.121s
success createPagesStatefully - 0.079s
success onPreExtractQueries - 0.001s
success update schema - 0.051s
After upgrading from 2.17.6 to 2.18.12, "update schema" step is taking ~40 seconds instead of 0.051s:
2.18.12 timings
=======================================
success source and transform nodes - 5.167s
success building schema - 1.206s
success createPages - 27.748s
success createPagesStatefully - 0.108s
success onPreExtractQueries - 0.000s
success update schema - 41.675s
It's this problem related to "update schema" step not taking into account that type inference is disabled for SitePage.context?
Steps to reproduce
Our repo is private, but we are willing to provide access to any Gatsby employee (maybe @pvdz)
Expected result
"update schema" step should not take any time, as it did with v2.17.6.
Actual result
It takes ~40 seconds instead of ~0s
Environment
System:
OS: Linux 4.15 Ubuntu 18.04.3 LTS (Bionic Beaver)
CPU: (4) x64 Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Shell: 4.4.20 - /bin/bash
Binaries:
Node: 12.12.0 - ~/.nvm/versions/node/v12.12.0/bin/node
Yarn: 1.21.1 - /usr/bin/yarn
npm: 6.13.4 - ~/.nvm/versions/node/v12.12.0/bin/npm
Languages:
Python: 2.7.17 - /usr/bin/python
Browsers:
Chrome: 79.0.3945.88
Firefox: 71.0
npmPackages:
gatsby: ^2.18.12 => 2.18.17
gatsby-plugin-emotion: ^4.1.18 => 4.1.18
gatsby-plugin-eslint: 2.0.8 => 2.0.8
gatsby-plugin-manifest: ^2.2.34 => 2.2.34
gatsby-plugin-react-helmet: ^3.1.18 => 3.1.18
gatsby-plugin-webpack-bundle-analyzer: ^1.0.5 => 1.0.5
gatsby-source-filesystem: ^2.1.43 => 2.1.43
gatsby-transformer-json: ^2.2.22 => 2.2.22
npmGlobalPackages:
gatsby-cli: 2.8.22
gatsby: 2.1.19
asilgag
All 6 comments
There have been some changes to graphql. Could you post the whole output before and after? I'm half expecting (or hoping? :) ) that the time is moved to a different step and that the total runtime is about the same.
If that's not the case I'm definitely going to look into it as this is not a regression I'd be keen on keeping.
pvdz
on 6 Jan 2020
@vladar anything on your radar that might relate to this?
pvdz
on 6 Jan 2020
This is the whole output from 2.17.6:
success open and validate gatsby-configs - 0.855s
success load plugins - 0.875s
success onPreInit - 0.091s
success delete html and css files from previous builds - 0.011s
success initialize cache - 0.015s
success copy gatsby files - 0.022s
success onPreBootstrap - 0.005s
success source and transform nodes - 3.775s
success building schema - 1.190s
success createPages - 22.467s
success createPagesStatefully - 0.059s
success onPreExtractQueries - 0.002s
success update schema - 0.046s
success extract queries from components - 0.372s
success write out requires - 0.100s
success write out redirect data - 0.002s
success Build manifest and related icons - 0.025s
success onPostBootstrap - 0.080s
⠀
info bootstrap finished - 33.478 s
⠀
success Building production JavaScript and CSS bundles - 8.861s
success Rewriting compilation hashes - 0.004s
success run queries - 60.079s - 19896/19896 331.17/s
success Building static HTML for pages - 67.070s - 19888/19888 296.53/s
info Done building in 166.188489269 sec
And this the one from 2.18.12:
success open and validate gatsby-configs - 0.927s
success load plugins - 0.855s
success onPreInit - 0.092s
success delete html and css files from previous builds - 0.010s
success initialize cache - 0.009s
success copy gatsby files - 0.022s
success onPreBootstrap - 0.004s
success createSchemaCustomization - 0.005s
success source and transform nodes - 3.814s
success building schema - 1.223s
success createPages - 26.515s
success createPagesStatefully - 0.077s
success onPreExtractQueries - 0.002s
success update schema - 37.295s
success extract queries from components - 0.386s
success write out requires - 0.109s
success write out redirect data - 0.003s
success Build manifest and related icons - 0.023s
success onPostBootstrap - 0.083s
⠀
info bootstrap finished - 75.192 s
⠀
success Building production JavaScript and CSS bundles - 9.096s
success Rewriting compilation hashes - 0.005s
success run queries - 27.439s - 19896/19896 725.09/s
success Building static HTML for pages - 67.882s - 19888/19888 292.98/s
info Done building in 177.231419989 sec
As you can see:
- "update schema" is slower (0s vs 37s)
- "createPages" is slower (22s vs 26s)
- "run queries" is significantly faster (60s vs 27s)
All in all, the difference between two versions is about ~11 seconds.
Since we have switched off type inference for SitePage.context, shouldn't "update schema" step be faster, or even bypassed? We effectively know that we haven't changed any field after createPages, so could be opt-out of that schema update just like we did with type inference for SitePage.context?
asilgag
on 7 Jan 2020
More info about this issue.
We have removed the "update schema" step (following directions from https://github.com/gatsbyjs/gatsby/issues/14797#issuecomment-502354892) and our build time has gone down from 177 sec to 127 sec, 50 seconds less:
success open and validate gatsby-configs - 0.851s
success load plugins - 1.102s
success onPreInit - 0.188s
success delete html and css files from previous builds - 0.009s
success initialize cache - 0.007s
success copy gatsby files - 0.027s
success onPreBootstrap - 0.002s
success createSchemaCustomization - 0.004s
success source and transform nodes - 0.074s
success building schema - 0.271s
success createPages - 24.446s
success createPagesStatefully - 0.073s
success onPreExtractQueries - 0.000s
success extract queries from components - 0.180s
success write out requires - 0.096s
success write out redirect data - 0.001s
success Build manifest and related icons - 0.023s
success onPostBootstrap - 0.065s
⠀
info bootstrap finished - 29.771 s
⠀
success Building production JavaScript and CSS bundles - 8.173s
success Rewriting compilation hashes - 0.001s
success run queries - 25.558s - 19888/19888 778.15/s
success Building static HTML for pages - 64.330s - 19888/19888 309.16/s
info Done building in 126.490407588 sec
Done in 126.91s.
We haven't detected any issue with the resulting public folder, so we think that the problem comes from Gatsby not properly honoring that we have switched off type inference for SitePage.context. It's that assumption ok?
asilgag
on 9 Jan 2020
I need to check this when I have a moment and then will get back to you. Minimum reproduction could have helped a lot (disabling inference for SitePage and then showing that inference still occurs, i.e. SitePageContext type gets additional fields from data).
vladar
on 9 Jan 2020
Confirmed this. A PR with a fix: #20662
vladar
on 17 Jan 2020
Related issues
omrllm
·
83Comments
dlindahl
·
81Comments
antoinerousseau
·
139Comments
cusspvz
·
128Comments
freiksenet
·
131Comments
Most helpful comment
Confirmed this. A PR with a fix: #20662