Flow: Flow 0.23.1 hanging forever in Linux on project codebase

[Macil]

I'm having an issue where since Flow 0.23.1 in linux specifically, flow check seems to hang forever when trying to type-check a private codebase with 100k flow-typed lines and its node_modules folder (some stuff in there does have flow-type info that's used). Flow 0.23.0 and previous versions were fine and usually took a minute at most.

I can't reproduce the issue with Flow 0.23.1 on OS X. I can reproduce the issue using the official Flow 0.23.1 linux build on Ubuntu 15.10 (64-bit, ran with 4gb of ram in VirtualBox) and on Debian 7 Wheezy (the official Flow build requires a version of libc newer than what's in Debian 7's repos, so I can't really recommend this). Flow 0.23.0 works fine on both.

If I run flow check --verbose, then I see it process many files and some type information about them as is normal, but it eventually halts. It doesn't halt in the same place always.

I've been trying to cut the codebase down to a minimal reproduction case, but it's slow-going as I have to wait up to a few minutes after each attempt to cut it down to see if the issue still exists, and I don't think it's a single file that's triggering the issue as my attempts at a binary search have failed. I figured I would report this first just in case anyone might have some troubleshooting tips, or might recognize what the problem is from this description. It doesn't seem there have been many commits between 0.23.0 and 0.23.1. I could do a git-bisect over Flow to find which one introduced the problem if that would be useful.

[inv8der]

I'm experiencing the same issue with [email protected]. We're performing type-checking during our CirceCI builds, running in an Ubuntu 14.04 container. Builds are timing out while running flow check --show-all-errors. When I run flow, it never gets past the parsing phase.

Launching Flow server for /home/ubuntu/gigwalk_apps_platform
Spawned flow server (child pid=17843)
Logs will go to /tmp/flow/zShomezSubuntuzSgigwalk_apps_platform.log
flow is still initializing; this can take some time. [parsing] \

Started experiencing this after upgrading from 0.22.1

[Macil]

So I've been trying to strip down a copy of the codebase to try to get the minimal reproduction case but haven't been very successful. I've been aggressively deleting or truncating files (so that Flow sees they have no @flow header and doesn't report an error when they're imported). I've run into a wall though now: if I remove or truncate many more files, Flow starts working again. There doesn't seem to be anything special with these individual files. It's especially weird that removing a zero-length file that nothing imports can cause Flow to work again! This seems to imply that I'm running into some sort of maximum file or memory limit.

Here's something interesting: I have 32745 files (.js, .jsx, and .json files are all that's left). Removing certain files, including a zero-length file that nothing refers to, makes Flow work (as in, not hang) again. That number of files is suspiciously close to 32768.

Increasing the open file limit with ulimit -n 40000 and increasing the stack size didn't help.

[balupton]

Started noticing this in my travis builds today after upgraded from 0.20

https://travis-ci.org/bevry/projectz/builds/127099577

[DanielHeath]

I've started getting this after upgrading to 0.24

[DanielHeath]

Attached is a backtrace & blocked task list from an instance of this happening in our CI environment:

https://gist.github.com/DanielHeath/5480367dac9760c592c2748b1bfcc28b

[samwgoldman]

I could do a git-bisect over Flow to find which one introduced the problem if that would be useful.

@AgentME That would be super helpful!

[balupton]

Another project affected by it: https://travis-ci.org/bevry/caterpillar/builds/127340723

In both cases, node 0.12 and node 4 hang, node 5 and node 6 work fine.

However, all node versions work locally on my OSX El Capitan machine.

[scast]

I am also observing this same behavior on my very simple codebase. It works fine locally on my Mac but trying to use flow to typecheck on Debian Jessie hangs forever.

If it helps, I am using flow 0.24.0 on Node 6. Also this started happening when I added a npm-shrinkwrap.json file to my codebase. Removing this file, fixes the problem.

[mehnomy]

Hello,
I'm having a very similar issue i have created a github issue in the Facebook CFT:
https://github.com/facebook/fbctf/issues/67

Just wondering whether anyone had any ideas of what may be causing the issue.

Kind Regards,
Byron Fisher

[gsingh93]

Our workaround was to put our src and node_modules folders next to each other (not nested), put the .flowconfig in src, and include the specific modules we needed: https://github.com/facebook/fbctf/blob/master/src/.flowconfig

[skevy]

This is happening for me as well. Flow gets stuck on the "local inference" phase -- GCE compute instance, 4 vCPUs, 8 GB of RAM, Ubuntu 16.04 (fresh install). Flow 25.

[skevy]

@samwgoldman I can confirm this issue does not occur for me in 0.23.0, but occurs in 0.23.1.

The change between those two versions was very minimal: https://github.com/facebook/flow/compare/baa38fface3f6f7f24125b10f11e46298bfdf723...c6dc55d5337c72af70a383b1879a9c3330605df1

This revelation, in addition to @scast's comment about the fact that things start failing when he added an "npm-shrinkwrap.json" file to his codebase, leads me to believe that there is some issue with the lack of strict parsing of JSON that causes flow to fail in some way (and for some reason not surface the error).

Does this help/does it trigger any ideas?

[shouze]

We have the same issue with flow 0.25. It tell us some memory limit happens when running on travis CI. Works localy on OSX El Capitan, very strange behaviour.

@skevy yes, you added some json file... and from 0.24 json parsing/support has been heavily revamped, I suspect that if we exclude json parsing then the failure will varnish. We're currently investigating in this direction for our project with failures.

[Macil]

Tried again with Flow 0.26.0 and I no longer see this issue. From the releases log, it looks like the issue might have been fixed in v0.24.2? I hadn't tried with it or 0.25.0.

• Fixed a bug where Flow might run out of memory in a repository with a lot of non-flow files

[shouze]

@AgentME I'm still experiencing the issue with 0.26

[Ragnaroek]

same here with versions 0.24.2, 0.25, 0.26.
blocks production.

[tryangul]

I am having the same issue with older versions -- 20.1 and 22.1

[shouze]

It's related to my app node_modules because if I delete the node_modules directory of my app then I don't reproduce this issue.
It's probably related to some specific module(s) but I can't figure which one yet.

[mroch]

i suspect it's related to the number of files rather than any specific file. there's a bug in the hashtable implementation that we (and hack) use where it loops forever when it fills up.

try adding

[options]
sharedmemory.hash_table_pow=19

to your flowconfig (0.26+) to double the size of the table. if that works, we'll at least know what the problem is.

[armandabric]

I've try to this option to my .flowconfig file but the option is not reconized: .flowconfig:17 Unsupported option specified! (sharedmemory.hash_table_pow)

My .flowconfig file:

[version]
0.26.0

[ignore]
.*/node_modules/fbjs
.*/node_modules/react-side-effect
build/*
.*/tests/.*
.*/__tests__/.*
.*/node_modules/.*/test/.*
.*/node_modules/.*/tests/.*

[options]
munge_underscores=true
esproposal.class_instance_fields=enable
esproposal.class_static_fields=enable
sharedmemory.hash_table_pow=19

[shouze]

@Spy-Seth I don't understand as sharedmemory.hash_table_pow is defined (added on 2016/05/24, 2 days prior to release 0.26.0).

[mroch]

my bad, it didn't make it into the 0.26 release branch: https://github.com/facebook/flow/blob/v0.26/src/common/flowConfig.ml

so if you're set up for it, you could try building from master. 0.27 will be released next week.

[shouze]

@mroch ok, thx we will give a try with master to bring you some feedback ASAP.

[stevoland]

The issue is resolved for me (in 0.26 at least) by ignoring json files:

[options]
module.file_ext=.js
module.file_ext=.jsx

[shouze]

@mroch ok I've tested with the master branch of flow with this docker image I've built and the result is slighly different. The flow process die silently & return an error code 15.

@stevoland yes, I suspect json files since the beginning. We have 2944 json files into our app's node_modules directory. I've seen that npm store a lot of big json files into node_modules/npm/.nyc_output/*.json. I will check if they're responsible au flow crashes.

[shouze]

@stevoland @mroch ok, in fact we had in our dev dependencies the npm-check-updates which have npm itself as a dependency. And npm bring many many pretty heavy json files.

[skevy]

Thanks for chiming in @mroch!

[alvaromb]

Same here. Hangs forever at flow is still initializing; this can take some time. [local inference] |.

[stevoland]

[options]
sharedmemory.hash_table_pow=19

works for me with 0.27.0

[balupton]

Updated from version 0.24.0 to v0.27.0 and problem is resolved for me.

[MoOx]

FYI, I am having a similar issue while trying to use Flow on Windows (0.27 and latest master branch)
(my project is brand new and don't have a lot of files, takes a second on my mac, I am setting flow for some windows dev)

• flow via docker toolbox (so OS>VM>docker container): horribly slow, can take a couple of minutes.
• flow via native windows bin (I builded it this morning, the build is ok), the check take around half a minute.
• flow via docker for Windows is not affected (but only works on Windows 10, and some devs have windows 7)

I moved flow stuff in src/ to avoid having node_modules, and boom speed issue is (almost) gone for flow via docker toolbox or via native bin. (FYI, I have .*/node_modules/.* ignored).

[Ragnaroek]

Problem still exists in 0.27 for us:

[16:32:27] 'flow-single' errored after 23 min
[16:32:27] Error: stderr maxBuffer exceeded.
at Socket. (child_process.js:697:13)
at Socket.emit (events.js:95:17)
at Socket. (_stream_readable.js:764:14)
at Socket.emit (events.js:92:17)
at emitReadable_ (_stream_readable.js:426:10)
at emitReadable (_stream_readable.js:422:5)
at readableAddChunk (_stream_readable.js:165:9)
at Socket.Readable.push (_stream_readable.js:127:10)
at Pipe.onread (net.js:528:21)

[Ragnaroek]

still in 0.28

[GuillaumeLeclerc]

Still in 0.29. Any workaround ?

[Ragnaroek]

Seems to be fixed in 0.30.0. Thanks!!

[jamiebuilds]

Is anyone still experiencing this in 0.30?

[GuillaumeLeclerc]

It's good here thanks !

[gnoff]

@thejameskyle

EDIT: Ignore the below I had an apparent older flow that was found in my path that didn't match the 0.30 version i thought i was running 0.20.1 or something.

on 0.30 / OSX, when i run flow i get into a loop of

Launching Flow server for ...
Spawned flow server (child pid=62840)
Logs will go to ...
flow is still initializing; this can take some time. [processing]
Launching Flow server for ...
...

when I run flow start i don't get the repeating behavior but the server never starts either and calls to flow stop error

flow check works fine

[rwilcox]

I no longer experience this in Flow 0.30

[samwgoldman]

Thanks for all the detailed debugging information, everyone! I'm gonna close this out because the issue seems to be resolved by bb67b737dc0c365dc1249f127e84b2cd36e7ab7a.

[yarax]

Reproduced in 0.34
flow is still initializing; this can take some time. [processing] \ hangs forever
.flowconfig

[libs]
../node_modules
[options]
sharedmemory.hash_table_pow=19

Folder contains just several js files

[mroch]

@yarax you probably don't want all of node_modules as libs. libs are interfaces like these. adding all of node_modules will probably spend an extremely long time loading all of the actual implementation files. i'm guessing it's not hanging forever, just taking a super long time.

try [include] instead of [libs].

there's probably more we could do to not actually load non-libs as libs, though. i'll file a separate issue for that.

[mroch]

2795

[yarax]

@mroch thanks!

[CMCDragonkai]

I get this like 3/4 times on a new create-react-app repository. Then after a while, it gets pass the flow is still initializing; this can take some time message, and then works. I noticed my CPU goes to 100% with antimalware service executable and then for some reason perhaps flow isn't fast enough and then the process terminates with some timeout message. This is on windows 10. Does this have something to do with scanning all the node_modules?

[jasonszhao]

I get the same result as OP in version 45 when I use all=true, and I also happen to get outrageously high memory usage (I only have 16 GB!). I'm using Create React App.

Here's my .flowconfig:

[include]
./client/src/
./lib/
./server/

[ignore]
.*/node_modules/.*
.*/.git/.*

[options]
module.file_ext=.js
module.file_ext=.jsx
sharedmemory.hash_table_pow=19
all=true

[sarkistlt]

the same issue, here my config:

[ignore]
.*/node_modules/.*
.*/dist/.*

[include]

[libs]

[options]
all=true
module.system=haste
esproposal.decorators=ignore
module.name_mapper='.*\.\(svg\|png\|jpg\|gif\|css\|scss\)$' -> 'stub'