Evalml: Generate human-readable descriptions of a pipeline's behavior

[dsherry]

Building off our fraud demo:

import evalml
X, y = evalml.demos.load_fraud(n_rows=1000)
pipeline = evalml.pipelines.BinaryClassificationPipeline(
    ['Missing Value Imputer', 'One Hot Encoder', 'Random Forest Classifier'], parameters={})
pipeline.fit(X, y)
print(pipeline.explain())

could output "The target 'fraud' is heavily influenced by 'amount'" and somewhat influenced by 'provider' and 'card_id'.

We can bucket features into three categories: "heavily influenced", "influenced" and "somewhat influenced" (exact wording can change), delineated by the relative importance magnitude. In the example above "amount" has a permutation importance of around 0.5, whereas "provider" and "card_id" are closer to 0.1. TBD the exact algorithm to compute this.

I'd suggest building in some heuristics and parameters:

• importance_method: defaults to "permutation"
• total_importance_threshold, num_feature_threshold, min_importance_threshold: take the top num_feature_threshold features, drop until their summed importances are <= total_importance_threshold, and drop any features with importance < min_importance_threshold. My thought for defaults to start with would be 0.8, 10 and 0.05.

Ideally we can structure this in a way which makes localization easier, where we make it easy to swap out the template strings as long as they keep the same number of named fill-in fields.

I think a quick design doc would help us solidify our thoughts on this.

[rpeck]

I think this is a swell idea. However, the bigger challenge, if we don't have it already, is to do human-readable explanations for features generated by FeatureTools, especially aggregated ones. It was A Project (tm) in a previous life of mine to do this for complex FT-like engineered features.

Once we have complex, aggregated features the overall explanation that you have in this ticket will become really wordy and complex...

The target 'fraud' is heavily influenced by the ratio of 'transaction count per day' over 'transaction count per month', heavily influenced by the difference between 'transaction amount' and 'mean transaction amount for this customer for the last year', and influenced by 'distance between the centroid of the zipcode of the transaction location' and 'centroid of the zipcode of the customer's home'.

[thehomebrewnerd]

@rpeck Featuretools does have the ability to generate human-readable feature descriptions, but the description for complex features can still be a bit challenging to understand at times.

Here is some documentation FYI: https://featuretools.alteryx.com/en/stable/guides/feature_descriptions.html

There is default text that gets used in the absence of anything else, or users can add more descriptive text to make them better.