Flow: Flow sometimes crashes on Prepack

Created on 2 May 2018 · 8Comments · Source: facebook/flow

Flow is an awesome type checker for JavaScript; we couldn't have implemented the Prepack partial optimizer without out.

However, Prepack is also a huge Flow-type annotated codebase, and has a large cycle between files. Flow's performance seems to degrade with our cycle size, so we are trying hard to keep our cycle size in check.

Part of how we are doing this is by running flow check --profile, and parsing the output, finding the longest cycle, and it's landblocking to increase the cycle size.

However, since a few days, this flow invocation sometimes fails --- in a non-reproducible manner, usually in our CI: https://circleci.com/gh/facebook/prepack/4184

Copy&pasting:

$ babel-node scripts/detect_bad_deps.js // a script that turns around and invokes: flow check --profile 
Error while processing Flow stderr, couldn't find cycle information:
   WALL TIME            CPU TIME        SECTION
--------------------------------------------------------
  8.011 (100.0%)     10.981 (100.0%)    <Total>
  8.008 (100.0%)     10.976 (100.0%)    Parsing
  0.003 (  0.0%)      0.004 (  0.0%)    <Unknown total>
Unhandled exception: WorkerController.Worker_failed_to_send_job(_)
Raised at file "src/core/lwt.ml", line 3008, characters 20-29
Called from file "src/unix/lwt_main.ml", line 42, characters 8-18
Called from file "src/commands/checkCommands.ml", line 106, characters 57-132
Called from file "src/flow.ml", line 84, characters 6-34

This happened with flow 0.69.0.

Crash

Source

NTillmann

😕3

Most helpful comment

Still exists in flow version 0.75.0.

Running flow check on Circle CI (box with 2CPU/4096MB) with NODE v9

bachstatter on 9 Jul 2018

👍2

All 8 comments

And it crashed again, this time with flow 0.71.0:

$ babel-node scripts/detect_bad_deps.js
Error while processing Flow stderr, couldn't find cycle information:
   WALL TIME            CPU TIME        SECTION
--------------------------------------------------------
  6.831 (100.0%)      2.100 (100.0%)    <Total>
  6.828 (100.0%)      2.096 ( 99.8%)    Parsing
  0.003 (  0.0%)      0.003 (  0.2%)    <Unknown total>
Unhandled exception: WorkerController.Worker_failed_to_send_job(_)
Raised at file "src/core/lwt.ml", line 3008, characters 20-29
Called from file "src/unix/lwt_main.ml", line 42, characters 8-18
Called from file "src/commands/checkCommands.ml", line 106, characters 57-132
Called from file "src/flow.ml", line 84, characters 6-34

Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18
Unix.Unix_error(Unix.EPIPE, "write", "")
Raised by primitive operation at file "unix.ml", line 257, characters 7-34
Called from file "hack/utils/marshal_tools.ml", line 144, characters 6-67
Called from file "hack/utils/marshal_tools.ml", line 157, characters 4-63
Called from file "hack/utils/sys/timeout.ml", line 28, characters 12-18

NTillmann on 9 May 2018

same for me, flow 0.72.0
guys what's the problem? how can u debug it?

cat /private/tmp/flow/zSUserszSalexey.kalachikzSdevelopmentzSnfl-web.log

2018-05-23 13:54:19.415] argv=/Users/alexey.kalachik/development/nfl-web/node_modules/flow-bin/flow-osx-v0.72.0/flow start --temp-dir /tmp/flow /Users/alexey.kalachik/development/nfl-web
[2018-05-23 13:54:19.421] Watching paths: 
        /Users/alexey.kalachik/development/nfl-web
        /Users/alexey.kalachik/development/nfl-web/src
[2018-05-23 13:54:19.422] Initializing Server (This might take some time)
[2018-05-23 13:54:19.422] executable=/Users/alexey.kalachik/development/nfl-web/node_modules/flow-bin/flow-osx-v0.72.0/flow
[2018-05-23 13:54:19.422] version=0.72.0
[2018-05-23 13:54:19.423] Parsing
[2018-05-23 13:59:59.458] File /Users/alexey.kalachik/development/nfl-web/node_modules/moment-duration-format/test/vendor/qunit.js is malformed
[2018-05-23 14:21:28.080] Building package heap
[2018-05-23 14:21:28.511] Loading libraries
[2018-05-23 14:21:30.199] Resolving dependencies
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Worker exited (code: 16)
Subprocess(51165): fail 16Subprocess(51166): fail 16Subprocess(51167): fail 16Subprocess(51168): fail 16Subprocess(51169): fail 16Subprocess(51172): fail 16Unhandled exception: WorkerController.Worker_failed(_)
Raised at file "src/core/lwt.ml", line 3008, characters 20-29
Called from file "src/unix/lwt_main.ml", line 42, characters 8-18
Called from file "src/server/server.ml", line 257, characters 6-54

cat /private/tmp/flow/zSUserszSalexey.kalachikzSdevelopmentzSnfl-web.monitor_log

May 23 13:54:19.385 [info] argv=/Users/alexey.kalachik/development/nfl-web/node_modules/flow-bin/flow-osx-v0.72.0/flow start --temp-dir /tmp/flow /Users/alexey.kalachik/development/nfl-web
May 23 13:54:19.385 [info] Creating a new Flow server
May 23 14:21:47.565 [error] Closing connection 'server #1' due to uncaught exception in read loop: End_of_file
Unable to obtain kernel buffer: Operation not permitted
usage: sudo dmesg
May 23 14:21:47.565 [error] backtrace:
May 23 14:21:47.565 [error] Raised at file "src/core/lwt_pqueue.ml", line 78, characters 15-30
May 23 14:21:47.565 [error] Called from file "src/core/lwt_pqueue.ml", line 86, characters 12-24
May 23 14:21:47.565 [error] 
May 23 14:21:47.628 [error] Flow server (pid 51164) exited with code Unknown_error (110)
May 23 14:21:47.628 [info] Monitor is exiting (Dying along with server)
May 23 14:21:47.628 [info] Broadcasting to threads and waiting 1 second for them to exit

P.S. also facing the same issue on 0.68.0 version

akalachik on 23 May 2018

@mroch @gabelevi @samwgoldman @avikchaudhuri @nmote sorry guys for tagging u directly, but as i can see - this issue reproducible not only for me and seems it's a bug

akalachik on 23 May 2018

👍2