Evalml: Suppress sklearn UndefinedMetric warning from stdout (F1 score)

[kmax12]

Through the course of training models, I get warnings like this

I am able to trigger it more specifically with a call like this

from evalml.objectives import F1

f1 = F1()
f1.score(y_predicted=[0, 0],
         y_true=[0, 1])

In terms of ways to handle. Here is what comes to mind

1. Completely silence during automl search process. replace score with nan or worst possible score for metric. maybe we then store the error messages somewhere in the results dictionary
2. Make a cleaner warning message that doesn't take up multiple lines

[dsherry]

Yeah, we should fix this. In general we shouldn't allow anything to get printed to stdout unless our code does so.

I wonder if this is related to #311.

I think we should do both of your suggestions: suppress this sklearn stdout output, and also write our own warning message if possible.

[dsherry]

@christopherbunn RE you mentioning this in the meeting, should this be marked as In Progress?

[dsherry]

@christopherbunn @jeremyliweishih I did some more reading on this, and I'd like the fix here to be to set zero_division=0.0 for all our precision and f1 objectives (binary and multiclass).

Explanation
Precision is n_true_pos / (n_true_pos + n_false_pos). So if the model never predicts a particular label at all on the data split in question, there won't be any true or false positives for that label, and therefore there'll be a division by 0. Similar is true of f1.

Argument
Let's assume we do a good job of balancing our classes (which we currently don't, but is a separate topic, #194 #457 ). If so, it's highly unlikely either the training or validation splits would contain few instances of a particular class. And if we can assume that, and the model is still making no predictions for a particular label, I'd argue it's a poor model, and therefore we should give it the lowest possible score, which for both precision and f1 is 0.

Sound good?

Example

In [38]: import numpy as np

In [39]: import sklearn.metrics

In [40]: y_true = np.array([0, 0, 0, 0, 1])

In [41]: y_pred = np.array([0, 0, 0, 0, 0])

In [42]: sklearn.metrics.precision_score(y_true, y_pred)
/Users/dylan.sherry/.pyenv/versions/3.8.2/envs/evalml/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
Out[42]: 0.0

In [43]: sklearn.metrics.precision_score(y_true, y_pred, zero_division=0.0)
Out[43]: 0.0

[jeremyliweishih]

@dsherry makes sense to me. I think that’s what we’re doing right now but also with the warning (since it defaults to “warn” which sets to 0 and also posts warnings)

[dsherry]

@jeremyliweishih yeah!

[dsherry]

Grabbing this from @christopherbunn so we can get it merged this week as he does battle with his finals :)