Create-react-app: Enforce hoisting of some packages to fix giant disk size with Yarn
Yarn has known issues deduplicating dependencies, sometimes failing to hoist dependencies shared between a ton of packages (such as babel-runtime) and producing installs 5 times bigger than npm: https://github.com/yarnpkg/yarn/issues/2306.
While we could disable Yarn integration (https://github.com/facebookincubator/create-react-app/issues/1390), I'd like to explore an alternative approach. We should be able to determine which package causes most duplication, and then hoist it manually by making it an explicit dependency of react-scripts. This would ensure Yarn deduplicates it. This is a temporary workaround until Yarn actually fixes the algorithm (tracked in https://github.com/yarnpkg/yarn/issues/2306).
Please don't hesitate to help with this. You would need to:
- Reproduce the issue with giant (~1GB) install locally with Yarn.
- Figure out which dependencies get duplicated. (I don't know the easiest way but you could probably look at Babel packages and see which dependencies end up in
node_modulesof each of them.) - Add the duplicated dependencies to
react-scriptsitself to hoist them. - Verify that the issue is resolved.
gaearon
All 18 comments
I've been looking to get more involved in open source work. I'd be glad to look into this and see if I can fix it.
droidenator
on 16 Jan 2017
Yeah me too. I have created a list of duplicate modules installed by create-react-app, but am not sure what to do next.
Can I run create-react-app locally from a cloned copy?
Would it be just a matter of adding the duplicates to the Package.json in react-scripts?
sambanks
on 16 Jan 2017
Can I run create-react-app locally from a cloned copy?
Yes, see instructions here.
Would it be just a matter of adding the duplicates to the Package.json in react-scripts?
Should be.
gaearon
on 16 Jan 2017
Great I'll give it a crack thanks
sambanks
on 16 Jan 2017
According to pkgcount --du --sort size, after a fresh create-react-app, there's 54 copies of [email protected] coming in at 373.36 MB:
PACKAGE # SIZE
…
[email protected] 1 618.5 kB
[email protected] 1 659.46 kB
[email protected] 1 659.46 kB
[email protected] 1 679.94 kB
[email protected] 1 700.42 kB
[email protected] 1 708.61 kB
[email protected] 1 712.7 kB
[email protected] 1 794.62 kB
[email protected] 1 798.72 kB
[email protected] 2 868.35 kB
[email protected] 1 909.31 kB
[email protected] 2 933.89 kB
[email protected] 1 950.27 kB
[email protected] 1 983.04 kB
[email protected] 1 1.01 MB
[email protected] 1 1.15 MB
[email protected] 1 1.44 MB
[email protected] 1 1.54 MB
[email protected] 1 1.81 MB
[email protected] 1 1.88 MB
[email protected] 1 1.95 MB
[email protected] 53 2.17 MB
[email protected] 1 2.42 MB
[email protected] 1 2.54 MB
[email protected] 1 2.56 MB
[email protected] 1 2.75 MB
[email protected] 1 3.28 MB
[email protected] 1 4.4 MB
[email protected] 1 5.08 MB
[email protected] 1 8.4 MB
[email protected] 53 52.32 MB
[email protected] 54 373.36 MB
Other duplicates are far less significant, so deduping core-js should be the key priority.
timoxley
on 16 Jan 2017
FYI this is the same issue as npm 2 (why we didn't suggest using npm 2 for Babel 6).
Luckily after dropping Node < 4 we should be able to remove babel-runtime (core-js/regenerator) from babel itself after some work in https://github.com/babel/babel/issues/5118. Otherwise workaround is to make babel-runtime an explicit dependency for now.
(This is because almost all the babel plugin packages have babel-runtime (since we use https://babeljs.io/docs/plugins/transform-runtime/) to support Symbol, Promise, Map/Set, etc which are in Node 4.)
And as for a timeline we are just going to do another 6.x + fixes and 7.x work is already starting https://github.com/babel/babel/wiki/Babel-7 thanks to contributors! The scope of the changes isn't huge so it should be an easy upgrade (at least I intend it to be)
hzoo
on 16 Jan 2017
I don't mean to imply that I'm somehow more qualified to prioritize tasks than anyone else, especially @gaearon, but I fail to see why this issue is a big deal.
At the end of the day, disk space is cheap, this has no impact on build size or, if I understand correctly, build time. Its just feels to me like complaints about the size of node_modules are almost like saying "back in my day we used to walk uphill both ways to hand pick minified jQuery plugins and lovingly lay them down in the vendor directory."
Certainly this duplication bug is a problem that yarn should strive to fix, but I don't really see how this matters enough to warrant temporary workarounds in CRA.
omrigilad
on 16 Jan 2017
Disk space may be cheaper but it's not universal truth especially for older machines running out of space. If you work on many projects it's annoyingly each each of them takes a whole gig.
It also impacts install times negatively because the linking step takes a long time.
gaearon
on 16 Jan 2017
I was just tracking down where the [email protected] module comes from. It looks like [email protected] referenced by [email protected] is using that. All other package are using core-js@^2.4.0.
I have no idea how feasible it would be to update react to use the next version of fbjs (which is in 0.9.0-alpha.1)
FWeinb
on 16 Jan 2017
I can't categorically disagree with anything you've said, but I just don't think that annoyance is enough of a reason to "dirty" react-scripts' package.json with incorrect dependencies, let alone disable yarn integration entirely.
omrigilad
on 16 Jan 2017
@mmgm The link time alone is a bit of a killer for me, taking up to 2 minutes for an app that really isn't so big. I'm on a not so powerful laptop but it's still a little crazy.
tbillington
on 17 Jan 2017
@FWeinb oh interesting. Sounds like a similar issue with when Babel 6 was still being compiled with Babel 5 causing core-js@1 and core-js@2 deps and fixed with https://github.com/babel/babel/pull/3438 (to have the same core-js/babel-runtime version)
Also FYI: https://github.com/babel/babel/pull/3340 for core-js@1 to core-js@2 update in Babel a while ago.
hzoo
on 17 Jan 2017
@hzoo
Yes. Having the same babel-runtime should be the second goal. core-js@2 is responsible for a major chunk of memory. So trying to remove that should benefit create-react-app greatly.
FWeinb
on 17 Jan 2017
So I've done some investigation into deduping some of the dependencies:
Baseline:
--- /create-react-app/test-1 ---------------------------------------------------------------------------------------
543.5 MiB [##########] /node_modules
188.0 KiB [ ] yarn.lock
60.0 KiB [ ] README.md
32.0 KiB [ ] /public
24.0 KiB [ ] /src
4.0 KiB [ ] package.json
4.0 KiB [ ] .gitignore
Adding core-js as a top level dependency reduces the size significantly:
--- a/packages/react-scripts/package.json
+++ b/packages/react-scripts/package.json
@@ -28,6 +28,7 @@
"babel-eslint": "7.1.1",
"babel-jest": "18.0.0",
"babel-loader": "6.2.10",
+ "core-js": "2.4.1",
"babel-preset-react-app": "^2.0.1",
"case-sensitive-paths-webpack-plugin": "1.1.4",
"chalk": "1.1.3",
--- /create-react-app/test-2 ---------------------------------------------------------------------------------------
194.0 MiB [##########] /node_modules
188.0 KiB [ ] yarn.lock
60.0 KiB [ ] README.md
32.0 KiB [ ] /public
24.0 KiB [ ] /src
4.0 KiB [ ] package.json
4.0 KiB [ ] .gitignore
Also adding babel-runtime reduces the size even further:
--- a/packages/react-scripts/package.json
+++ b/packages/react-scripts/package.json
@@ -28,6 +28,8 @@
"babel-eslint": "7.1.1",
"babel-jest": "18.0.0",
"babel-loader": "6.2.10",
+ "babel-runtime": "6.20.0",
+ "core-js": "2.4.1",
"babel-preset-react-app": "^2.0.1",
"case-sensitive-paths-webpack-plugin": "1.1.4",
"chalk": "1.1.3",
--- /create-react-app/test-3 ---------------------------------------------------------------------------------------
142.3 MiB [##########] /node_modules
188.0 KiB [ ] yarn.lock
60.0 KiB [ ] README.md
32.0 KiB [ ] /public
24.0 KiB [ ] /src
4.0 KiB [ ] package.json
4.0 KiB [ ] .gitignore
Lastly if just babel-runtime (with a carat range) is defined then you get most of the benefits:
--- a/packages/react-scripts/package.json
+++ b/packages/react-scripts/package.json
@@ -28,6 +28,7 @@
"babel-eslint": "7.1.1",
"babel-jest": "18.0.0",
"babel-loader": "6.2.10",
+ "babel-runtime": "^6.20.0",
"babel-preset-react-app": "^2.0.1",
"case-sensitive-paths-webpack-plugin": "1.1.4",
"chalk": "1.1.3",
--- /Users/kimjona/code/create-react-app/test-4 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
148.7 MiB [##########] /node_modules
188.0 KiB [ ] yarn.lock
60.0 KiB [ ] README.md
32.0 KiB [ ] /public
24.0 KiB [ ] /src
4.0 KiB [ ] package.json
4.0 KiB [ ] .gitignore
From this is looks like the quickest solution would be add both From this is looks like the quickest solution would be add just babel-runtime and core-js to react-scripts.babel-runtime to react-scripts. What do you think @gaearon ?
Node version: v6.9.0
Yarn version: v0.18.1
Operating system: Mac OS X 10.11.6
jkimbo
on 18 Jan 2017
[email protected] contains core-js@^2.4.0 so dono if there are other babel-runtime versions as well? regenerator-runtime may help as well but probably not as much (also the other dep in babel-runtime).
Either way we will try to fix this soon for Babel 7.
hzoo
on 18 Jan 2017
From this is looks like the quickest solution would be add just babel-runtime to react-scripts. What do you think @gaearon ?
Sounds awesome, send a PR please.
gaearon
on 23 Jan 2017
jkimbo
on 24 Jan 2017
Fixed by #1441, will be out in 0.9.0.
gaearon
on 24 Jan 2017
Related issues
rdamian3
·
3Comments
JimmyLv
·
3Comments
xgqfrms-GitHub
·
3Comments
fson
·
3Comments
Evan-GK
·
3Comments
Most helpful comment
I don't mean to imply that I'm somehow more qualified to prioritize tasks than anyone else, especially @gaearon, but I fail to see why this issue is a big deal.
At the end of the day, disk space is cheap, this has no impact on build size or, if I understand correctly, build time. Its just feels to me like complaints about the size of node_modules are almost like saying "back in my day we used to walk uphill both ways to hand pick minified jQuery plugins and lovingly lay them down in the
vendordirectory."Certainly this duplication bug is a problem that yarn should strive to fix, but I don't really see how this matters enough to warrant temporary workarounds in CRA.