Evalml: Investigate why Fraud/Lead scoring sometimes exhibits worse performance

[chukarsten]

Following up on #2139 , it seems our notebook lead_scoring notebook sometimes shows contradictory modeling performance.

From #2139 :

Found that when we leave the data splitter as it was, the AUC for the LeadScoring-optimized search was 0.5 since it always guessed the same output. This is likely because balancing the data in this scenario leads to no learning done. If the data is balanced 1:4, as long as the gain outweighs the loss, we will always predict the same output. I changed the sampling to be smaller in order to highlight the point we wanted to convey.
This previous version is similar to our current performance
We note that we've had many iterations where the performance was very poor:
v13
v17
v19
v20

Closing Criteria: determine why this is happening and implement the long term solution going forward.

[dsherry]

@bchen1116 will #2269 fix this issue? Or is there more which needs to happen before we can close it?

[dsherry]

@bchen1116 and I spent some time digging into this.

Summary

• We discovered a cosmetic bug in automl search related to binary classification objective tuning. If the primary objective does not depend on classification threshold (e.g. AUC/log loss), but a secondary objective does depend on threshold, then the values for that second objective computed during automl will not be computed with a tuned threshold, and will therefore be artificially pessimistic about performance on that objective.
• This cosmetic bug could mean that whenever we set AUC or log loss as primary objective for automl, other objectives computed during automl are computed without a tuned threshold and are therefore worse than what is possible.
• API cleanup needed in BinaryClassificationPipeline.optimize_threshold: ideally we don't ask for y_pred_proba as an input, but rather compute that internally using the pipeline.
• Found minor bug in bin class threshold tuning plot: title is set to y axis instead

Next steps

• Figure out a fix for the additional objective scoring bug as described above.
• File issues for API cleanup and minor plotting bug
• Continue to look at threshold tuning graphs for lead scoring objective and see if we can fix the tutorial demo.

[dsherry]

Here's the notebook @bchen1116 and I were using lead scoring.ipynb.zip

[bchen1116]

@dsherry Doc results for weighing the objective scores here

[bchen1116]

This PR was filed in response to the doc above to close the issue for lead-scoring objective. Furthermore, this issue tracks updating AutoMLSearch to threshold tune using an additional objective when necessary, which will fix our demos doc for both Fraud and Lead-Scoring. Lastly, this issue was filed to fix the fraud objective, as we have found it selects the naive estimator as the best estimator given any weights.

These issues/PRs close out this spike.