Flow: Multiflow change causes runaway memory consumption

[philikon]

We have a monorepo with a fairly large JS codebase spread over several dozen npm packages. We have a single .flowconfig file at the root of the monorepo. Flow checks the entire tree.

We're currently running Flow 0.37.4 which is working well. #3225 prevented us from upgrading to 0.38.0. Now this problem prevents us from upgrading to 0.39.0. Bisecting through the Flow source, I've determined that 3d8f7fc1a189ab8f1d09a784cc6a921ff5863df4 (cfe630ad5f4e84aaf83dad2a30761f9a2464e710 on the 0.39 branch) is responsible for the breakage.

Symptom: running flow check never determinates, consumes high amounts of CPU and increasing amounts of memory the longer it runs. Revisions before multiflow was introduced complete within several seconds.

Is there any debugging output I can generate that would let us figure out what Flow is stuck on? I'm suspecting it's going down some infinite loop or similar.

[philikon]

At @gabelevi's suggestion I tried finding out where Flow might be stuck in an infinite loop. Just so I had something to compare, I decided to first run Flow 0.37.4 with --verbose-indent, but sadly it segfaulted after a while:

$ flow check --verbose-indent 2>flow_verbose_stderr
Worker interrupted with signal: sigsegv
$ du -sh flow_verbose_stderr 
7.5G    flow_verbose_stderr

I haven't bothered running Flow 0.39 with --vebose-indent yet, though I suspect much the same symptom.

[philikon]

Good news! I managed to find the offending code in our repo and fix it in a way that Flow would no longer be stuck eating memory.

To reproduce, clone https://github.com/silklabs/silk and drop the following .flowconfig in the root:

[ignore]
.*/docs/build/.*

.*/sdk/interface/silk-wifi.js.flow
.*/sdk/interface/silk-sysutils.js.flow

.*/bsp-gonk/vendor/silk/silk-alog/.*
.*/bsp-gonk/vendor/silk/silk-properties/.*
.*/bsp-gonk/vendor/silk/silk-movie/.*
.*/bsp-gonk/vendor/silk/silk-capture/.*
.*/node-caffe/.*

[options]
unsafe.enable_getters_and_setters=true
module.system=haste
module.use_strict=true
module.file_ext=.js
module.file_ext=.json
esproposal.class_instance_fields=enable
esproposal.class_static_fields=enable

Run flow. There'll be some errors that are immaterial to this problem. Point is, Flow runs and completes.

To reproduce the nontermination problem, apply the following patch:

diff --git a/bsp-gonk/vendor/silk/silk-wifi/lib/index.js b/bsp-gonk/vendor/silk/silk-wifi/lib/index.js
index 3927a5f..1085c2c 100644
--- a/bsp-gonk/vendor/silk/silk-wifi/lib/index.js
+++ b/bsp-gonk/vendor/silk/silk-wifi/lib/index.js
@@ -457,7 +457,7 @@ class WpaMonitor extends EventEmitter {
   }
 }

-function wpaCli(...args: Array<string>): Promise<string> {
+function wpaCli(...args) {
   if (!monitor.ready()) {
     log.info('wpaCli: supplicant not ready, waiting');
     return util.timeout(1000).then(() => wpaCli(...args));
@@ -477,7 +477,7 @@ function wpaCli(...args: Array<string>): Promise<string> {
   });
 }

-function wpaCliExpectOk(...args: Array<string>): Promise<void> {
+function wpaCliExpectOk(...args) {
   return wpaCli(...args).then((ok) => {
     if (ok.match(/^OK/)) {
       return;

[philikon]

FWIW, occasionally I still observe non-determination, even with all rest arg types declared. A co-worker actually sees the non-determination problem every time on his machine.

Is there any progress on this? #3443 seems to provide a great minimal repro step. Anything else we can do to help zero in on the problem?

[0rvar]

I have this problem on Windows 10 from 0.39 to 0.47. 0.38 is the last version that does not infinite loop and consume all memory.

[Hizoul]

suffering from the same problem my system becomes completely unusable and that on a 12 core cpu x)

[barbalex]

This seems to be rather prominent in the issues. Maybe related issues found after a five minute search:

• https://github.com/facebook/flow/issues/5271
• https://github.com/facebook/flow/issues/4370
• https://github.com/facebook/flow/issues/4070
• https://github.com/facebook/flow/issues/4046
• https://github.com/facebook/flow/issues/3565
• https://github.com/facebook/flow/issues/3370
• https://github.com/facebook/flow/issues/2152

[JofArnold]

See my comment here for one way of preventing this: https://github.com/facebook/flow/issues/2152#issuecomment-366221060

[mrkev]

Closing since repro issue pointed out by @philikon (#3443) has been fixed. Other memory leaks can be discussed in different threads.