Build: Provide option to fail fast when build hits first severe error

[smaifullerton-wk]

When a build hits a severe error using DDC, the compiler will continue trying to build, usually resulting in cascading build errors related to or resulting from the first severe error.

If we run a failing build in CI, this leads to 2+ minutes added to our CI time before it exits with a failure.

It'd be great if there was an option to fail fast upon the first severe build error (or even consider making it the default..?)

Thank you!

[jakemac53]

Definitely agree on the usability concerns here.

From an implementation standpoint it is difficult because of the lazy builds (essentially, we build top down so that we won't attempt to compile code that isn't imported by some app). This means we have already started running other builders etc at the point that we get the failure - and there is no easy mechanism to "cancel" those asynchronous tasks that are already scheduled.

We could possibly come up with something though. We already run each build method within a Zone - so we could potentially use those to short circuit.

[smaifullerton-wk]

Just FYI, we had a situation just occur where a build that produced a severe error in the first two minutes churned for an hour until it hit our CI timeout:

2019-08-16T14:11:56.671721822Z     [INFO] 58m 50s elapsed, 31008/31054 actions completed.
2019-08-16T14:11:57.771683978Z     [INFO] 58m 51s elapsed, 31008/31054 actions completed.
2019-08-16T14:11:58.871736325Z     [INFO] 58m 52s elapsed, 31008/31054 actions completed.
2019-08-16T14:11:59.971671416Z     [INFO] 58m 53s elapsed, 31008/31054 actions completed.
2019-08-16T14:12:01.071678544Z     [INFO] 58m 54s elapsed, 31008/31054 actions completed.
2019-08-16T14:12:01.873275541Z     [WARNING] No actions completed for 15.1s, waiting on:
2019-08-16T14:12:01.873305252Z       - build_web_compilers:ddc on package:
[closed source files]

[smaifullerton-wk]

@jakemac53 We're considering updating the dart_dev package to parse and fail the process upon [SEVERE] output during a build. We've noticed a couple of cases where warnings in some cases yield SEVERE output, so we know we'll have to shore these up.

Are there any other considerations or disadvantages that come to mind? Thank you!

[jakemac53]

The problem with killing the process is you are going to end up throwing away any work that was successful (it won't re-serialize the asset graph to disk, so it will be reverted back to the state before the last build). This could also potentially lead to a corrupted cache.

If it is consistently hanging though then you don't have much choice, so we need to figure out the source of the hang.

[smaifullerton-wk]

Ah yeah, that doesn't sound ideal, although in CI it's not a big deal since we do fresh builds (for now), and CI is where it's most problematic. I'll keep an eye out for other long running cases and if it's too frequent maybe we'd look into adding an opt-in, temporary, CI-only option. Appreciate the response.