Evalml: Calculating for high variance when the mean of cv_scores is zero

[angela97lin]

Right now, our calculation for high variance is:

        cv_scores_std = cv_scores.std()
        cv_scores_mean = cv_scores.mean()
        if cv_scores_std != 0 and cv_scores_mean != 0:
            high_variance_cv = bool(abs(cv_scores_std / cv_scores_mean) > threshold)```

This was put in place to prevent a divide-by-zero warning / error. However, this impl is not ideal: if the mean is close to zero (but not zero), high variance will likely be true (dividing by small number).

In addition, it is possible for there to be high variance if the mean is zero but the standard deviation is very high. Right now, we default to False but we may want to look for a impl that is smarter in this situation.

[chukarsten]

@angela97lin can you clarify what led to this issue coming up? This doesn't seem like a common occurence.

First approach is to maybe add epsilon to prevent a divide by zero.

[angela97lin]

@chukarsten This came up in #2024, where I covered the cases to not emit a runtime warning, but in an impl that might not be ideal. I wonder if there's a better approach than dividing by the mean; the approach of adding an epsilon will avoid the RuntimeWarning but also mean that high variance will likely always be triggered, since dividing by a tiny-close-to-zero value will be ~inf.

[freddyaboulton]

@angela97lin Is this still a problem? I manually checked your [0.0, -1.0, 1.0] example and I don't get a divide by zero error:

import evalml
from evalml import AutoMLSearch
from evalml.utils import infer_feature_types
import numpy as np


X, y = evalml.demos.load_fraud(n_rows=1000)


automl = AutoMLSearch(X_train=X, y_train=y, problem_type='binary', objective='f1', max_batches=1)

automl.search()

assert not automl._check_for_high_variance(automl.get_pipeline(0), np.array([0, -1, 1]))

I think we only compute the coefficient of variation when both the std and mean are non-zero so we're protected from an exception.

We can debate if returning False for the high variance check is useful when the mean is zero but I think the original concern is the division by 0 which I don't think can happen.

[angela97lin]

@freddyaboulton Ah sorry, the order of the PRs is a bit confusing. You're right, #2024 handled the case where mean is zero so that we don't trigger the the divide by zero issue. What I wanted this PR issue to track was a better way to calculate / handle when the mean is zero, since right now we're defaulting to False. Will update the description :d

[freddyaboulton]

@angela97lin Thanks for explaining! The problem of dividing by zero only happens with the objectives that can take on negative values. For those objectives, I wonder if our use of the coefficient of variation is even valid?

I looked at the wiki, and the definition section mentions that it should only be calculated for data on a ratio scale, i.e. data measured on a scale with a "meaningful zero". I take that to mean that the data can't be negative. This stats stackexchange post seems to support that interpretation.

With that in mind, I guess I see four options:

1. If the mean is <=0, don't calculate the coefficient of variation. We would never raise the warning in this situation.
2. Add an epsilon to the denominator. I guess it depends on the value of epsilon but basically we would always raise the warning if the mean is zero.
3. Don't change anything. We would never raise the warning when the mean == 0.
4. Find a different measure of overfitting that accounts for objectives that can take on negative values.

What do you think?

[bchen1116]

@freddyaboulton @angela97lin Agreed with Freddy. I think it'd make sense to not run the current high variance check on objectives that can have negative values, since it seems like that is mathematically incorrect.

I'm not a huge fan of options 1-3, since those options don't really accurately handle instances where the data isn't on a ratio_scale, especially since I don't think we should try to use coefficient of variation for objectives that aren't on the ratio_scale. I would go for a variation of option 4.

I think it makes sense to have 2 separate checks. We can continue with the current high variance check for objectives that cannot be negative (accuracy, AUC, recall, etc), and for objectives that can take on negative values (R2, etc), we should find a separate measure for overfitting (this seems more robust, but I'm not sure if this would be a good measure for overfitting).

[dsherry]

Converting this to a spike. Goal: figure out our options, and work with the team to pick one.