Flow: Unhandled exception on flow check with saved state

[w01fgang]

Flow version: 0.128.0

Expected behavior

While using saved state

# .flowconfig
[options]
lazy_mode=fs
saved_state.fetcher=local

and files .flow.saved_state and .flow.saved_state_file_changes
created with flow save-state --root=. --out=.flow.saved_state
flow check should work without exceptions

Actual behavior

flow check
with an empty .flow.saved_state_file_changes

*** done committing modules ***
[2020-07-04 18:06:42.250] Committing NameHeap
[2020-07-04 18:06:42.250] Committing InfoHeap
[2020-07-04 18:06:42.261] Committing all_providers hashtable
Unhandled exception: Lwt.Resolution_loop.Canceled
Raised at file "map.ml", line 135, characters 10-25
Called from file "src/unix/lwt_engine.ml", line 302, characters 8-36

and with pages/index.js in .flow.saved_state_file_changes

*** done committing modules ***
[2020-07-04 18:08:59.561] Committing NameHeap
[2020-07-04 18:08:59.561] Committing InfoHeap
[2020-07-04 18:08:59.565] Committing all_providers hashtable
[2020-07-04 18:08:59.566] recheck 1 modified, 0 deleted files
[2020-07-04 18:08:59.566] modified files:
[2020-07-04 18:08:59.566] 1/1: /path_modified/pages/index.js
[2020-07-04 18:08:59.566] Parsing
[2020-07-04 18:08:59.568] recheck: old = 2581, del = 0, fresh = 0, unmod = 2581
*** old modules (changed and deleted files) 0 ***
*** new modules (new and changed files) 0 ***
*** done committing modules ***
[2020-07-04 18:08:59.578] Resolved requires are unchanged
[2020-07-04 18:08:59.578] Re-resolving directly dependent files
[2020-07-04 18:08:59.578] Resolved requires are unchanged
[2020-07-04 18:08:59.578] Recalculating dependency graph
[2020-07-04 18:08:59.579] to_merge: Focused: 0, Dependents: 0, Dependencies: 0
[2020-07-04 18:08:59.579] Calculating dependencies
[2020-07-04 18:08:59.579] Merging
[2020-07-04 18:08:59.579] Merge skipped 0 of 0 modules
[2020-07-04 18:08:59.579] Done
[2020-07-04 18:08:59.579] Checked set: Focused: 0, Dependents: 0, Dependencies: 0
[2020-07-04 18:08:59.579] Committing context heaps
[2020-07-04 18:08:59.579] Committing ResolvedRequiresHeap
[2020-07-04 18:08:59.579] Committing ResolvedRequiresHeap
[2020-07-04 18:08:59.579] Committing NameHeap
[2020-07-04 18:08:59.579] Committing InfoHeap
[2020-07-04 18:08:59.579] Committing all_providers hashtable
[2020-07-04 18:08:59.579] Committing parsing heaps
Unhandled exception: Lwt.Resolution_loop.Canceled
Raised at file "map.ml", line 135, characters 10-25
Called from file "src/unix/lwt_engine.ml", line 302, characters 8-36

• Link to Try-Flow or Github repo:
  Couldn't reproduce it on a small repo, this happens only on my private repo

[w01fgang]

spent a few hours on trying to reproduce it, and one second later after creating an issue, found the solution
just increased sharedmemory.hash_table_pow from 21 to 22

I still experience the same issue, and it happens randomly.
I remove state file, create it again, and it works, once I change few files flow is starting to crash.

[nmote]

There's not much point in using saved state with flow check. Saved state is intended to allow Flow to avoid parsing all the files in the whole codebase, but flow check will parse and check all of the files anyway. We should probably just remove the saved state option from flow check altogether.

With that said, I just fixed what may be the root cause of this issue in 31d76c2c5965b1e312a571b5b32d4dc67e49fbf4. That issue manifested differently, with Flow crashing on init from saved state under certain circumstances. Let me know if that fixes your problem when it is released with Flow v0.133.0

[w01fgang]

@nmote with the latest version I still have the same issue
Here is the log

[2020-09-04 12:50:51.193] lazy_mode=off
[2020-09-04 12:50:51.193] arch=classic
[2020-09-04 12:50:51.193] abstract_locations=off
[2020-09-04 12:50:51.193] max_workers=12
[2020-09-04 12:50:51.193] debug=false
[2020-09-04 12:50:51.199] Initializing Server (This might take some time)
[2020-09-04 12:50:51.199] executable=/project_root/node_modules/flow-bin/flow-linux64-v0.133.0/flow
[2020-09-04 12:50:51.199] version=0.133.0
[2020-09-04 12:50:51.207] Reading saved-state file at "/project_root/.flow.saved_state"
[2020-09-04 12:50:51.275] Decompressing saved-state data
[2020-09-04 12:50:51.283] Denormalizing saved-state data
[2020-09-04 12:50:51.284] Denormalizing the data for the parsed files
[2020-09-04 12:50:51.300] Denormalizing the data for the unparsed files
[2020-09-04 12:50:51.355] Finished loading saved-state
[2020-09-04 12:50:51.356] Saved state script reports 0 files changed & we care about 0 of them
[2020-09-04 12:50:51.356] Restoring heaps
[2020-09-04 12:50:51.405] Loading libraries
[2020-09-04 12:50:52.221] Resolving dependencies
[2020-09-04 12:50:52.449] Failed to load recheck stats from "/tmp/flow/zSprojectzSroot.recheck_stats". Reason: "File doesn't exist"
Unhandled exception: Lwt.Resolution_loop.Canceled
Raised at file "map.ml", line 135, characters 10-25
Called from file "src/unix/lwt_engine.ml", line 302, characters 8-36

[nmote]

Thanks for the update. Could you run export OCAMLRUNPARAM=b in your shell, and then reproduce the crash again? It may give us a better stack trace in the logs.

If not, is there anything more you could share about your repository that may help me reproduce?

[w01fgang]

@nmote here what I found

1. Save the state
2. remove one file
3. Run flow check

Parsing service failed to cat /path_to_repo/libs/loadExtensions.js, so skipping it. Exception: (Sys_error
  "/path_to_repo/libs/loadExtensions.js: No such file or directory")
Raised by primitive operation at file "stdlib.ml", line 390, characters 28-54
Called from file "stdlib.ml", line 398, characters 2-47
Called from file "src/parsing/parsing_service_js.ml", line 563, characters 13-34

Unhandled exception: Lwt.Resolution_loop.Canceled
Raised at file "map.ml", line 135, characters 10-25
Called from file "src/unix/lwt_engine.ml", line 302, characters 8-36

Just running flow works

[2020-09-07 11:13:43.449] lazy_mode=fs
[2020-09-07 11:13:43.449] arch=classic
[2020-09-07 11:13:43.449] abstract_locations=off
[2020-09-07 11:13:43.449] max_workers=4
[2020-09-07 11:13:43.449] debug=false
[2020-09-07 11:13:43.451] Initializing Server (This might take some time)
[2020-09-07 11:13:43.451] executable=/path_to_repo/node_modules/flow-bin/flow-linux64-v0.133.0/flow
[2020-09-07 11:13:43.451] version=0.133.0
[2020-09-07 11:13:43.451] Reading saved-state file at "/path_to_repo/.flow.saved_state"
[2020-09-07 11:13:43.498] Decompressing saved-state data
[2020-09-07 11:13:43.508] Denormalizing saved-state data
[2020-09-07 11:13:43.508] Denormalizing the data for the parsed files
[2020-09-07 11:13:43.529] Denormalizing the data for the unparsed files
[2020-09-07 11:13:43.606] Finished loading saved-state
[2020-09-07 11:13:43.607] Saved state script reports 0 files changed & we care about 0 of them
[2020-09-07 11:13:43.607] Restoring heaps
[2020-09-07 11:13:43.694] Loading libraries
[2020-09-07 11:13:44.691] Resolving dependencies
[2020-09-07 11:13:44.953] Parsing
[2020-09-07 11:13:44.954] Resolved requires are unchanged
[2020-09-07 11:13:44.954] Re-resolving directly dependent files
[2020-09-07 11:13:44.954] Resolved requires are unchanged
[2020-09-07 11:13:44.955] Recalculating dependency graph
[2020-09-07 11:13:44.956] Failed to load recheck stats from "/tmp/flow/zSpathzStozSrepozShi-dir.recheck_stats". Reason: "File doesn't exist"
[2020-09-07 11:13:44.957] Server is READY
[2020-09-07 11:13:44.957] Took 1.505888 seconds to initialize.
[2020-09-07 11:13:44.957] Running a serial workload
[2020-09-07 11:13:44.957] Request: status
[2020-09-07 11:13:44.957] Status: OK

[nmote]

I took a look at this and was able to reproduce. It's basically the same as the issue that I fixed in https://github.com/facebook/flow/commit/31d76c2c5965b1e312a571b5b32d4dc67e49fbf4, only it takes a different code path when flow check is invoked.

I think the best way to address this is to simply prevent Flow from initializing from saved state when flow check is invoked. The whole point of saved state is to get a lazy server up and running quickly without having to parse the whole codebase on startup. This way, it can be ready to begin handling rechecks as changes come in. However, when you run flow check, you are asking Flow to typecheck the entire codebase and then exit. Therefore, it has to parse the whole codebase anyway, so saved state is of no use.