Evalml: Spike: New Engine Design

[freddyaboulton]

The Engine api/implementation has been changing rapidly (particularly in #1814 and #1913) since #1838 was merged a little under two weeks ago!

Once the dust settles on the EngineBase and SequentialEngine implementations/api we should take our latest learnings from the ParallelEngine work (#1869) and see if we can improve upon our initial design.

Some questions to consider:

1. Engines are created within AutoML but need a reference to AutoML. Can we make the engines not need a reference to AutoML?
2. train_and_score_pipeline is pretty big. Can we break it up into manageable/reusable pieces?
3. Currently, each engine needs to implement its own "loop" for training/scoring a batch of pipelines. This makes it hard to reuse logic needed to train and score an individual pipeline across our engines, e.g. when to use the ensembling dataset, how to handle errors. It could be worth it to "invert" the design - Engine subclasses only need to implement how to fit and/or score a single pipeline and we have a single "loop" for how to fit and/or score a batch in the base class.
4. How do we handle early stopping/keyboard interrupt the same way in sequential and parallel engines?
5. How do we handle max_iterations? Currently the SequentialEngine receives a callback from the AutoMLSearch instance and the callback updates the state of the AutoMLSearch to determine whether the engine should continue processing pipelines. This doesn't seem possible with the pipelines evaluated in parallel.
6. How do we test the parallel engine? Running class instances that have had functions patched via unittest.mock have those patches "stripped" after they go through Dask.Client.map().

Acceptance Criteria:
Document that proposes changes to the engine design or justifies the current design for the medium-term.

[chukarsten]

Love this issue. I'm going to add a little bit more to it @freddyaboulton and attach it to an epic. thank you for writing this.