Metro: bundler chokes on large files

[thurber]

Do you want to request a feature or report a bug?
bug

What is the current behavior?
Trying to require json files that are large, or even lots of small ones, causes bundler to choke without really reporting anything (stuck at X%). By large, I mean only a few megabytes. By lots of small ones, I mean like 10 jsons of 800kb each (trying to break the large file into small chunks).

If the current behavior is a bug, please provide the steps to reproduce and a minimal repository on GitHub that we can yarn install and yarn test.
import bigJson from './big.json'

What is the expected behavior?
Bundler has no problem importing big json.

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.
metro 0.28.0, node 9.5.0, yarn 1.5.1, osx sierra

See #111 and #123 for other reports of the same problem, but issues were closed without resolution.

[rafeca]

Hey @thurber! This is usually caused by babel trying to parse the JSON (which are not even needed to be transformed), can you check if telling babel to ignore json files improves things?

To do so, you can follow the instructions in https://github.com/facebook/metro/issues/123#issuecomment-357751394

If this improves things, we can make metro auto-ignore JSON files from being transformed by babel

[thurber]

Hi @rafeca, thanks for your response. Indeed, ignoring the json files worked in metro v0.20.3 for dev builds only, but this workaround is no longer successful in metro v0.28.0, and never for production builds.

Edit: I should note that for my particular use case, this corresponds to react-native v0.51.0 (working in dev builds) and react-native 0.54.0 (never working). It's hard for me to grok exactly how react-native is configuring/using metro, and if that configuration contributes to the problem.

[rafeca]

Ok, then it means that even the babel parsing that metro is doing by itself (to wrap the JSON) is causing the OOM.

I'll have to make a change in metro, to avoid parsing JSON files with babel

[rafeca]

@thurber I've just landed a change that will make metro not pass any JSON file to babel. This should fix this issue

[thurber]

hi @rafeca, thanks for working on this, the transform is definitely faster, but alas the problem remains. I've created a repo that reproduces the issue here: https://github.com/thurber/bigFileBug

I couldn't get yarn to accept master branch for metro, but you'll see that the problem exists even for the large JS file.

The project was created just with react-native init and then generating and importing a large JS file and a large JSON file.

[rafeca]

Hi @thurber! we transform metro sourcecode before publishing to npm, so installing master from yarn won't work unfortunately.

Regarding the large JS file, unfortunately metro needs to parse it through babylon (which is the one that is causing the OOM) in order to detect potential require() calls on this file and discover new dependencies.

What you can do as a workaround is to increase the default memory allocation that nodejs uses (by e.g calling node --max-old-space-size=8192 node_modules/react-native/local-cli/cli.js start from your repo).

What's your use case for having a single >3MB JS file in the repo? (I guess that it's some precompiled library).

[thurber]

@rafeca, I am not sure how to confirm, but I suspect the problem is not during the babel transform, but at some later stage in the packaging ( I have logged out to console before and after transform step for each big file ). Node memory size does not appear to help. Out of curiosity, does my example repo finish building on your machine? Even with only large json file and your updates to not pass it through the transform, my metro build gets stuck at 99%. I'm not sure how to get metro to tell me what it's trying to do at that moment... :(

Use case is for including a three.js mesh buffer file in json format.

[rafeca]

@thurber I've cloned your repo and after a few tweaks I was able to build the bundle correctly with the JSON file by patching the master version of metro.

A few considerations though:

1. I had to comment out the import bigJS from './bigJS' from App.js (as mentioned above, only huge JSON files are supported for now).
2. To get the master version of metro working I had to first transform its source code and then manually copy the transformed code into the node_modules folder of your repo.
3. I had to manually remove the metro version installed in node_modules/react-native/node_modules (RN is depending in an older version of metro, so yarn/npm installs it in the nested node_modules).
4. Since the current master version of metro is not backwards compatible with RN 0.54 (due to a new config param) I had to manually do some patches there.

I'm going to try to make a somewhat more elegant patch to your repo and send a PR, but I think that the easiest thing for you is to wait until the next RN version is shipped with the actual patch

[thurber]

@rafeca I see, I missed step # 3 when trying this on my own ^^;

Thanks very much for your patience in this matter! No need to create a special patch for me, as I have the luxury of waiting for the next RN version.

[wesvance]

I think I'm also experiencing this issue. I'm trying to import a very large JSON file into my app, but it fails due to either timing out, or maxing out the available cache. I tried to ignore JSON in my babel config, but it has no effect.

As @rafeca said, is the best thing to do is to wait for the next RN version to ship? Are there any other work arounds?

[HikingDev]

Shouldnt this issue be fixed with the latest release?
I've updated to "react-native": "0.55.1" but my 17Mb Json file still crashes the Metro bundler.

[rafeca]

Hi @roesneb, unfortunately due to the different release cycles between metro and React Native, the fix has not reached RN stable yet. Good news is that the fix is going to be in the upcoming 0.56 version :)

[ddeath]

@rafeca and what about the support for large js files? When can we expect that to be working in RN?
On RN 0.57.1 the big js files are still not working. Any ETA on this?

Thank you