Evalml: Data Check API

[kmax12]

https://alteryx.quip.com/GgtAAtfgbDKf/Data-Checks-API-Design

[kmax12]

detect outliers (and maybe optionally remove?)

[jeremyliweishih]

Blanket applying guard rails vs. model or problem specific guard rails?

[dsherry]

I'm repurposing this epic to refer to the Data Check API project (formerly known as the "Guard Rails API").

@angela97lin , @kmax12 and I just met to discuss what to do with this project. We decided:

• Change project scope
  
  
  
  • Design an API for specifying data checks, which result in errors/warnings
  
  
  • Taking action based on those errors/warnings is out of scope for now, and will likely be a project down the road
  
  
• Change the name from "Guard Rails" to "Data Check," because we feel that's more descriptive
• @angela97lin will archive the old guard rails design doc, and create a new one with the new scope

[dsherry]

@angela97lin I just finished going through the new design doc. Good stuff! It's looking awesome.

I left some comments and suggestions, but from my perspective, this is ready for implementation! Exciting 👏

I think once you've had a chance to review my comments, the next step should be to create an issue for each task in the implementation plan and attach them to this epic. Let's also clean up the existing issues currently in this epic. At first glance I think we can either move them to the icebox or close them.

[dsherry]

@angela97lin and I just reviewed the new design doc once more and ironed out a couple design details. Same as before: next step is to create issues and attach them to this epic. Very cool!

[angela97lin]

From #509:

We don't want people using "mean"/"median" as the impute_strategy for string types. It could be good to add some sort of data check to account for this?

[angela97lin]

From #504: move _check_multiclass to a data check :)

[dsherry]

@angela97lin and I just reviewed the design again and made some final tweaks. We updated the implementation plan (currently scoped at 4 weeks total) and filed issues attached to this epic.

This project is ready for implementation!

@angela97lin is going to start this project on Monday. Our target completion date is Monday, May 25th, just in time for the May 2020 release!

We also cleaned up the old issues on this epic. Some got booted out to the icebox as future features, and others (bugs which will be fixed by data checks) were marked as blocked on this epic.

[dsherry]

@angela97lin and I just discussed: we didn't include in the design any way to get the data check results to users who use automl search directly (in python).

We discussed two options for that:
1) Have AutoSearchBase.search return something like {'status': 'complete'}, then if data checks fail we can include {'status': 'error', 'data_checks': [..]}
2) Have AutoSearchBase.search save results internally and expose a latest_data_check_results getter (not intended as final name) to get them. Then raise an exception to clearly signal there was a problem.

We like the second option more, so we'll go with that. It feels more in-line with our existing design patterns of a) exceptions over error codes and b) having AutoSearchBase objects be containers for state rathern than one-time use objects.

Here's some code we mocked up while on the call, intended to go in AutoSearchBase.search just before the main while loop which runs the real search:

        data_check_results = checks.validate(X, y)
        # option 1
        if len(data_check_results) > 0:
            logger.error('Data checks found problems')
            return {'status': 'error', 'data_check_results': data_check_results}
        # option 2
        if len(data_check_results) > 0:
            logger.error('Data checks found problems')
            self._latest_data_check_results = data_check_results
            raise SearchException('One or more data checks failed. Look at latest_data_check_results')

        @property
        def latest_data_check_results(self):
            return self._latest_data_check_results

[dsherry]

@angela97lin : our current plan is to do the May release a week from today. Do you think we'll be able to get #709 and #370 merged in time for that? If not, how much time do you think is needed? We can easily punt #710 to the next release.

[dsherry]

@angela97lin merged most of this in the May release! The only issue remaining is #710. So, closing this epic 🎉