Flow: Flow server does not start (0.106.0)

[lukeapage]

Flow version: 0.106.0

Flow crashes. 0.105.2 works fine. Logs have these errors:

Subprocess(1224): fail -1073741571Unhandled exception: Worker_failed (process_id = 1224): (Worker_quit WEXITED -1073741571)
Raised at file "src/procs/workerControllerLwt.ml", line 97, characters 10-73
Called from file "src/core/lwt.ml", line 2102, characters 23-28
Re-raised at file "src/procs/workerControllerLwt.ml", line 66, characters 6-1023
Re-raised at file "src/procs/workerControllerLwt.ml", line 65, characters 4-1023
Re-raised at file "src/procs/workerControllerLwt.ml", line 30, characters 0-1023
Re-raised at file "src/procs/multiWorkerLwt.ml", line 64, characters 8-448
Re-raised at file "src/common/lwt/lwtUtils.ml", line 25, characters 4-92
Re-raised at file "src/common/lwt/lwtUtils.ml", line 33, characters 2-99
Re-raised at file "src/procs/multiWorkerLwt.ml", line 75, characters 8-306
Re-raised at file "src/procs/multiWorkerLwt.ml", line 72, characters 4-1023
Re-raised at file "src/procs/multiWorkerLwt.ml", line 107, characters 0-241
Re-raised at file "src/services/inference/merge_service.ml", line 423, characters 2-689
Re-raised at file "src/services/inference/types_js.ml", line 574, characters 4-624
Re-raised at file "src/services/inference/types_js.ml", line 160, characters 6-137
Re-raised at file "hack/utils/core/exception.ml", line 31, characters 2-45
Called from file "src/core/lwt.ml", line 2102, characters 23-28
Re-raised at file "src/services/inference/types_js.ml", line 159, characters 18-277
Re-raised at file "src/services/inference/types_js.ml", line 159, characters 4-575
Re-raised at file "src/services/inference/types_js.ml", line 737, characters 4-681
Re-raised at file "src/services/inference/types_js.ml", line 727, characters 2-1023
Re-raised at file "src/services/inference/types_js.ml", line 2465, characters 4-934
Re-raised at file "src/common/transaction/transaction.ml", line 61, characters 6-17
Called from file "src/core/lwt.ml", line 2102, characters 23-28
Re-raised at file "src/common/transaction/transaction.ml", line 58, characters 4-102
Re-raised at file "src/common/transaction/transaction.ml", line 57, characters 2-183
Re-raised at file "src/server/server.ml", line 49, characters 2-476
Re-raised at file "src/server/server.ml", line 152, characters 4-156
Re-raised at file "src/common/profiling/profiling_js.ml", line 834, characters 4-350
Re-raised at file "src/common/profiling/profiling_js.ml", line 255, characters 4-118
Re-raised at file "src/common/profiling/profiling_js.ml", line 1035, characters 2-675
Re-raised at file "src/server/server.ml", line 173, characters 6-1023
Re-raised at file "src/server/server.ml", line 168, characters 4-1023
Re-raised at file "src/core/lwt.ml", line 2998, characters 20-29
Called from file "src/unix/lwt_main.ml", line 26, characters 8-18
Called from file "src/server/server.ml", line 220, characters 6-54

(328) merge_job THROWS: [1] <redacted filename>: Utils_js.Key_not_found("LeaderHeap", "<redacted filename>")
Raised at file "src/state/heaps/context/context_heaps.ml", line 205, characters 14-77
Called from file "src/services/inference/merge_service.ml", line 88, characters 25-80
Called from file "list.ml", line 117, characters 24-34
Called from file "src/services/inference/merge_service.ml", line 72, characters 4-1023
Called from file "src/services/inference/merge_service.ml", line 120, characters 4-48
Called from file "src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "src/services/inference/merge_service.ml", line 339, characters 10-1023


(328) merge_job THROWS: [1] <redacted filename>: Utils_js.Key_not_found("LeaderHeap", "<redacted filename>")
Raised at file "src/state/heaps/context/context_heaps.ml", line 205, characters 14-77
Called from file "src/services/inference/merge_service.ml", line 88, characters 25-80
Called from file "list.ml", line 117, characters 24-34
Called from file "src/services/inference/merge_service.ml", line 72, characters 4-1023
Called from file "src/services/inference/merge_service.ml", line 120, characters 4-48
Called from file "src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "src/services/inference/merge_service.ml", line 339, characters 10-1023


(328) merge_job THROWS: [1] <redacted filename>: Utils_js.Key_not_found("LeaderHeap", "<redacted filename>")
Raised at file "src/state/heaps/context/context_heaps.ml", line 205, characters 14-77
Called from file "src/services/inference/merge_service.ml", line 88, characters 25-80
Called from file "list.ml", line 117, characters 24-34
Called from file "src/services/inference/merge_service.ml", line 72, characters 4-1023
Called from file "src/services/inference/merge_service.ml", line 120, characters 4-48
Called from file "src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "src/services/inference/merge_service.ml", line 339, characters 10-1023


(328) merge_job THROWS: [1] <redacted filename>: Utils_js.Key_not_found("LeaderHeap", "<redacted filename>")
Raised at file "src/state/heaps/context/context_heaps.ml", line 205, characters 14-77
Called from file "src/services/inference/merge_service.ml", line 88, characters 25-80
Called from file "list.ml", line 117, characters 24-34
Called from file "src/services/inference/merge_service.ml", line 72, characters 4-1023
Called from file "src/services/inference/merge_service.ml", line 120, characters 4-48
Called from file "src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "src/services/inference/merge_service.ml", line 339, characters 10-1023


(328) merge_job THROWS: [1] <redacted filename>: Utils_js.Key_not_found("LeaderHeap", "<redacted filename>")
Raised at file "src/state/heaps/context/context_heaps.ml", line 205, characters 14-77
Called from file "src/services/inference/merge_service.ml", line 88, characters 25-80
Called from file "list.ml", line 117, characters 24-34
Called from file "src/services/inference/merge_service.ml", line 72, characters 4-1023
Called from file "src/services/inference/merge_service.ml", line 120, characters 4-48
Called from file "src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "src/services/inference/merge_service.ml", line 339, characters 10-1023


Worker slave 328 exception: Unix.Unix_error(Unix.EPIPE, "write", "")
Worker slave 328 Potential backtrace:
Raised by primitive operation at file "unix.ml", line 199, characters 7-34
Called from file "hack/utils/marshal_tools/marshal_tools.ml", line 82, characters 10-42
Called from file "hack/utils/marshal_tools/marshal_tools.ml", line 158, characters 6-67
Called from file "hack/utils/marshal_tools/marshal_tools.ml", line 171, characters 4-63
Called from file "hack/utils/core/measure.ml", line 244, characters 12-16
Called from file "hack/procs/worker.ml", line 97, characters 14-143
Called from file "hack/procs/worker.ml", line 138, characters 4-37

[goodmind]

@mroch

[lukeapage]

Just confirmed - none of those crash logs appear in 0.105.2

[goodmind]

Is it Windows-only?

[lukeapage]

This is on windows yes, I can probably test mac next week.

[lukeapage]

and its not intermittant, its every time

[mroch]

Can you upload the complete .log and .monitor_log as a gist? Usually this kind of log spew is the result of an earlier error causing the workers to exit.

[lukeapage]

~Looking at the errors about missing keys, the key is present but exported with export *~

// a.js
import * as constants from 'b';

constants.X;
// b.js
export * from 'c';

export const Y = 'y';

// c.js
export const X = 'x';

edit - other examples where the exception occurred are straight forward so this is likely a red herring.

[mroch]

Google says -1073741571 means stack overflow on Windows. cc @gabelevi

[lukeapage]

Heres the complete logs: https://gist.github.com/lukeapage/50162b1d2e7e49a431fb207302beb5dc

[gabelevi]

Yeah, it sounds like @mroch is correct. One of the merge jobs hit a stack overflow and therefore failed to merge its files. This triggers a cascade of Utils_js.Key_not_found("LeaderHeap") exceptions as the subsequent merge.

On Linux, I'd be able to look at the core dump (stack overflows usually segfault) and figure out what's happening. But I'm at a loss for how to debug this on Windows.

Plus merge isn't supposed to actually check the files. Hmmm....

Do you have a file that is extremely large? Maybe it has a ton of imports or exports?

The only big typing change in v0.106 that comes to mind is the change in how we spread objects (types and values). Maybe that's triggering more work in v0.106 than in v0.105? @jbrown215 - is that plausible?

[mroch]

But I'm at a loss for how to debug this on Windows.

it would be great if this also repros on linux/mac. :)

[jbrown215]

@gabelevi: I'd be very surprised if spreads were causing this. We don't do any new work with the new semantics, just different work with the same information we already had.

Also, this is probably not a types-first codebase, so the merge step probably still triggers a bunch of work here!

[gabelevi]

Also, this is probably not a types-first codebase, so the merge step probably still triggers a bunch of work here!

Oh duh. Yeah, you're right!

[mroch]

Have never done this sort of thing on Windows, but I wonder if this would work on ocaml binaries: https://blogs.msdn.microsoft.com/benjaminperkins/2017/06/28/capture-a-stackoverflowexception-and-make-a-dump-0xc00000fd/

[lukeapage]

I will try that. Will also try Mac to see if it also fails and if so if it gives a better error. Feel free to point me at any Mac specific info if I have to do something to get a stack trace.

[lukeapage]

I'm not having any luck trying to get procdump to give me a dump - it seems to run on a single process and flow is creating new processes like crazy during the merge process, each with a different pid.
I'll run on mac tomorrow and hopefully it will crash and give me more info.

[lukeapage]

@mroch @gabelevi
It happens on mac too.
We don't have massive files - the biggest is probably the lodash library definition.

I uploaded the mac logs: https://gist.github.com/lukeapage/35d015e4b114c998033a311f40c54b55

It doesn't look like any more information than on windows - maybe you know better?

I saw this message

Unable to obtain kernel buffer: Operation not permitted
usage: sudo dmesg

and I then re-ran flow under sudo - those lines disappear but I don't see any extra lines or any dump anywhere.

I'm happy to try something to give you a stack trace if you can tell me what to do or point me at a resource.

[lukeapage]

I've tried deleting files until it passes (albeit with errors for missing files) and it seems to be within our predominantly react code that it fails. I started adding files back in and there are multiples files that seem to cause the crash.

[lukeapage]

Okay, I managed to create a reproduction case

// @flow
import _ from 'lodash';

class A {
    handleScrollEnd = _.debounce(this.handleScrollEnd, 300);

    handleScrollEnd() {
    }
}

export default A;

[lukeapage]

heres a repro case without needing the flow lib defs for lodash

// @flow

function a<F>(b: F, n: number): F & { cancel: () => void } {
   return b;
}

class A {
    handleScrollEnd = a(this.handleScrollEnd, 300);

    handleScrollEnd() {
    }
}

export default A;

[mroch]

this is coming from code related to the new definite assignment linter. i'm surprised it's running at all without that turned on. cc @jbrown215

Worker slave 3221674 Potential backtrace:
Raised by primitive operation at file "flow/src/typing/context.ml", line 643, characters 8-30
Called from file "flow/src/typing/context.ml", line 635, characters 22-37
Called from file "flow/src/typing/context.ml" (inlined), line 631, characters 23-45
Called from file "flow/src/typing/flow_js.ml", line 163, characters 36-62
Called from file "flow/src/typing/merge_js.ml", line 250, characters 13-41
Called from file "list.ml", line 160, characters 12-15
[... list.ml repeated several hundred times ...]
Called from file "flow/src/typing/merge_js.ml", line 286, characters 48-61
Called from file "map.ml", line 295, characters 20-25
Called from file "flow/src/typing/merge_js.ml", line 305, characters 4-63
Called from file "list.ml", line 106, characters 12-15
Called from file "flow/src/typing/merge_js.ml", line 461, characters 2-35
Called from file "flow/src/services/inference/merge_service.ml", line 130, characters 44-345
Called from file "flow/src/services/inference/merge_service.ml", line 236, characters 6-46
Called from file "flow/src/services/inference/merge_service.ml", line 350, characters 20-66
Called from file "flow/src/services/inference/merge_service.ml", line 317, characters 22-26
Re-raised at file "flow/src/services/inference/merge_service.ml", line 320, characters 4-11
Called from file "flow/src/services/inference/merge_service.ml", line 339, characters 10-1023
Re-raised at file "hphp/hack/src/utils/core/exception.ml", line 30, characters 33-76
Called from file "flow/src/services/inference/merge_service.ml", line 375, characters 31-52
Called from file "list.ml", line 117, characters 24-34
Called from file "hphp/hack/src/procs/workerController.ml", line 195, characters 42-47
Called from file "hphp/hack/src/procs/worker.ml", line 157, characters 4-37

(updated with the bottom of the stack trace)

[mroch]

even smaller repro:

declare function a<F>(b: F): F & {};

class A {
  b = a(this.b);
}

I think I fixed it, by making this recursive call look for cycles:

https://github.com/facebook/flow/blob/d73947eb839bf591b3f5df817b4e27a6f700a9e9/src/typing/merge_js.ml#L256

[mroch]

@lukeapage can you confirm 0.106.2 fixes your real use case? thanks again for the repro!

[lukeapage]

Yes, thanks we are upgraded and no more issues with the release. It removed one suppression we had for valid code, which is nice.