Cuml: [BUG] Considerable difference between the sklearn and cuml RF accuracy

[Salonijain27]

The difference in accuracy between sklearn and cuml RF varies in the range of 3-7% (3% difference obtained after hyper-parameter tuning) for the below example. The base code for the example below is the rf notebook present in the notebooks-contrib repo (https://github.com/rapidsai/notebooks-contrib/blob/branch-0.14/intermediate_notebooks/examples/rf_demo.ipynb) :

from cuml import RandomForestClassifier as cuRF
from sklearn.ensemble import RandomForestClassifier as sklRF
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
import cudf
import numpy as np
import pandas as pd
import os
from urllib.request import urlretrieve
import gzip

# ## Helper function to download and extract the Higgs dataset

def download_higgs(compressed_filepath, decompressed_filepath):
    higgs_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz'
    if not os.path.isfile(compressed_filepath):
        urlretrieve(higgs_url, compressed_filepath)
    if not os.path.isfile(decompressed_filepath):
        cf = gzip.GzipFile(compressed_filepath)
        with open(decompressed_filepath, 'wb') as df:
            df.write(cf.read())


def main():

    # ## Download Higgs data and read using cudf

    data_dir = 'raid/data/rfc/'
    if not os.path.exists(data_dir):
        print('creating rf data directory')
        os.system('mkdir raid/data/rfc')

    compressed_filepath = data_dir+'HIGGS.csv.gz' # Set this as path for gzipped Higgs data file, if you already have
    decompressed_filepath = data_dir+'HIGGS.csv' # Set this as path for decompressed Higgs data file, if you already have
    download_higgs(compressed_filepath, decompressed_filepath)

    col_names = ['label'] + ["col-{}".format(i) for i in range(2, 30)] # Assign column names
    dtypes_ls = ['int32'] + ['float32' for _ in range(2, 30)] # Assign dtypes to each column
    data = cudf.read_csv(decompressed_filepath, names=col_names, dtype=dtypes_ls)
    data.head().to_pandas()


    # ## Make train test splits

    X, y = data[data.columns.difference(['label'])].fillna(value=0).as_matrix(), data['label'].to_array()

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=500_000)
    print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

    # cuml Random Forest params

    cu_rf_params = {
        'n_estimators': 25,
        'max_depth': 25,
        'n_bins': 512,
        'seed' : 0,

    }

    # Train cuml RF
    cu_rf = cuRF(**cu_rf_params)
    cu_rf.fit(X_train, y_train)


    # sklearn Random Forest params

    skl_rf_params = {
        'n_estimators': 25,
        'max_depth': 25,
        'random_state' : 0,
    }

    # Train sklearn RF parallely
    skl_rf = sklRF(**skl_rf_params, n_jobs=20)
    skl_rf.fit(X_train, y_train)

    from sklearn.metrics import confusion_matrix
    cu_preds = cu_rf.predict(X_test)
    sk_preds = skl_rf.predict(X_test)
    cu_conf_mat = confusion_matrix(y_test, cu_preds)
    sk_conf_mat = confusion_matrix(y_test, sk_preds)
    print(" cuml con matri : ")
    print(cu_conf_mat)
    print(" sklearn con matri : ")
    print(sk_conf_mat)

    # ## Predict and compare cuml and sklearn RandomForestClassifier

    print("cuml RF Accuracy Score: ", accuracy_score(cu_rf.predict(X_test), y_test))
    print("sklearn RF Accuracy Score: ", accuracy_score(skl_rf.predict(X_test), y_test))

if __name__ == '__main__':
    main()

Output :

(7179904, 28) (7179904,) (500000, 28) (500000,)
/home/saloni/miniconda3/envs/float64-rf/bin/ipython:61: UserWarning: For reproducible results, n_streams==1 is recommended. If n_streams is > 1, results may vary due to stream/thread timing differences, even when random_seed is set
[W] [14:07:02.530014] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
[W] [14:11:31.697048] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
 cuml confusion matri :
[[147193  88201]
 [ 56040 208566]]
 sklearn confusion matri :
[[168559  66835]
 [ 61668 202938]]
[W] [14:11:57.015702] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
cuml RF Accuracy Score:  0.711518
sklearn RF Accuracy Score:  0.742994

[Salonijain27]

on reducing the number of trees to 1 and max_depth=1 we see a large difference in sklearn and cuml's confusion matrix and accuracy. For reference the dataset has 52% 1's and 48% 0's:

 cu_rf_params = {
         'n_estimators': 1,
         'max_depth': 1,
         'n_bins': 512,
         'max_features': 1.0,
         'seed' : 0,
     }

    skl_rf_params = {
        'n_estimators': 1,
        'max_depth': 1,
        'max_features': 1.0,
        'random_state' : 0,
    }

Output:

(10500000, 28) (10500000,) (500000, 28) (500000,)
/home/saloni/miniconda3/envs/duplication-reduc/bin/ipython:35: UserWarning: For reproducible results in Random Forest Classifier or for almost reproducible results in Random Forest Regressor, n_streams==1 is recommended. If n_streams is > 1, results may vary due to stream/thread timing differences, even when random_seed is set
[W] [08:32:01.751873] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
[W] [08:32:23.564072] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
 cuml con matri :
[[     0 234585]
 [     0 265415]]
 sklearn con matri :
[[ 91895 142690]
 [ 51969 213446]]
[W] [08:32:25.031102] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
cuml RF Accuracy Score:  0.53083
sklearn RF Accuracy Score:  0.610682

When the max_depth is increased to max_depth=10 the accuracy of both the models increases considerably but cuML's accuracy is lower than sklearns :

(10500000, 28) (10500000,) (500000, 28) (500000,)
/home/saloni/miniconda3/envs/duplication-reduc/bin/ipython:35: UserWarning: For reproducible results in Random Forestesults in Random Forest Regressor, n_streams==1 is recommended. If n_streams is > 1, results may vary due to stream/tom_seed is set
[W] [08:39:34.370599] Expected column ('F') major order, but got the opposite. Converting data, this will result in a
[W] [08:42:55.570608] Expected column ('F') major order, but got the opposite. Converting data, this will result in a
 cuml con matri :
[[156682  78541]
 [ 71548 193229]]
 sklearn con matri :
[[159556  75667]
 [ 72049 192728]]
[W] [08:42:57.083508] Expected column ('F') major order, but got the opposite. Converting data, this will result in a
cuml RF Accuracy Score:  0.699822
sklearn RF Accuracy Score:  0.704568

[robocopnixon]

2561 Is related

[JohnZed]

The 1-tree, depth=1 result is really interesting and should be helpful. If sklearn can get to 0.61 with that, there is clearly one feature that has a lot of content. Either we're picking the wrong feature or the wrong split for a good feature.

[teju85]

Tagging @vinaydes since he's currently ramping up on the RF code and also working to fix accuracy issues.

[hcho3]

I've reduced the example from #2561 as follows:

import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import RandomForestClassifier
from cuml.ensemble import RandomForestClassifier as cuml_RandomForestClassifier

from sklearn.model_selection import cross_val_score

# Preprocessed data
X = np.load('data/loans_X.npy')
y = np.load('data/loans_y.npy')
X_test = np.load('data/loans_X_test.npy')
y_test = np.load('data/loans_y_test.npy')

params = {
    'n_estimators': 1,
    'max_depth': 1,
    'max_features': 1.0,
    'bootstrap': False
}

n_bins = 512

skl_clf = RandomForestClassifier(n_jobs=-1, **params)
skl_clf.fit(X, y)
skl_train_accuracy = skl_clf.score(X, y)
skl_test_accuracy = skl_clf.score(X_test, y_test)
print(f'sklearn: Training accuracy = {skl_train_accuracy}, Test accuracy = {skl_test_accuracy}')

cuml_clf = cuml_RandomForestClassifier(n_bins=n_bins, **params)
cuml_clf.fit(X, y)
cuml_train_accuracy = cuml_clf.score(X, y)
cuml_test_accuracy = cuml_clf.score(X_test, y_test)
print(f'cuml: Training accuracy = {cuml_train_accuracy}, Test accuracy = {cuml_test_accuracy}')

This minimal example fits a single tree stump (depth 1) using the whole training dataset without any sampling. All features are used in all splits.

I ran the script 10 times and the results are now deterministic:

sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5555555820465088, Test accuracy = 0.0

[hcho3]

The notebook from #2561 takes the last 20000 data points as the test set (pd_df.tail(20000)) and the rest as the training set. It doesn’t shuffle the data prior to the split. I suspect the lack of shuffling might have introduced an artificial difference between the training and test sets.

When I tried 10-fold cross-validation, I get

sklearn: CV accuracy = 0.5961166666666667 (std=0.003261395028337438)
cuml: CV accuracy = 0.5555555522441864 (std=0.0034107610216626157)

Still a noticeable difference but now we don’t get a 0 accuracy for cuML. The gap in training accuracy is still there, so I will be focusing on the gap in the training accuracy.

[vinaydes]

I think I spotted at least one of the reasons for the discrepancy. The depth of the tree is 0-based in sklearn and it is 1-based in cuML. What it means is when max_depth is set to 1, sklearn trained model has three nodes, one root and two leaf nodes. However cuML model contains one root and one leaf node. To get similar models from both, the max_depth parameter needs to be more by 1 in case of cuML. I modified above posted code by @hcho3 to accommodate this:

import numpy as np
import matplotlib.pyplot as plt

from sklearn.ensemble import RandomForestClassifier
from cuml.ensemble import RandomForestClassifier as cuml_RandomForestClassifier

from sklearn.model_selection import cross_val_score

# Preprocessed data
X = np.load('data/loans_X.npy')
y = np.load('data/loans_y.npy')
X_test = np.load('data/loans_X_test.npy')
y_test = np.load('data/loans_y_test.npy')

params = {
    'n_estimators': 1,
    'max_features': 1.0,
    'bootstrap': False
}

n_bins = 512
max_depth = 8

skl_clf = RandomForestClassifier(n_jobs=-1, max_depth=max_depth, **params)
skl_clf.fit(X, y)
skl_train_accuracy = skl_clf.score(X, y)
skl_test_accuracy = skl_clf.score(X_test, y_test)
print(f'sklearn: Training accuracy = {skl_train_accuracy}, Test accuracy = {skl_test_accuracy}')

cuml_clf = cuml_RandomForestClassifier(n_bins=n_bins, max_depth=max_depth+1, **params)
cuml_clf.fit(X, y)
cuml_train_accuracy = cuml_clf.score(X, y)
cuml_test_accuracy = cuml_clf.score(X_test, y_test)
print(f'cuml: Training accuracy = {cuml_train_accuracy}, Test accuracy = {cuml_test_accuracy}')

I got following result with this code:

depth = 1
sklearn: Training accuracy = 0.5961166666666666, Test accuracy = 0.7918
cuml: Training accuracy = 0.5961166620254517, Test accuracy = 0.7918000221252441

depth = 2
sklearn: Training accuracy = 0.6000111111111112, Test accuracy = 0.8201
cuml: Training accuracy = 0.6000111103057861, Test accuracy = 0.8201000094413757

depth = 3
sklearn: Training accuracy = 0.6409666666666667, Test accuracy = 0.4536
cuml: Training accuracy = 0.6409666538238525, Test accuracy = 0.4535999894142151

depth = 4
sklearn: Training accuracy = 0.6518222222222222, Test accuracy = 0.45665
cuml: Training accuracy = 0.6518222093582153, Test accuracy = 0.45669999718666077

Up to depth 4, the models and accuracy are very close to each other. However for further depths the cuML testing accuracy started dropping

depth = 5
sklearn: Training accuracy = 0.6656722222222222, Test accuracy = 0.50165
cuml: Training accuracy = 0.665672242641449, Test accuracy = 0.4603999853134155

depth = 6
sklearn: Training accuracy = 0.6787611111111111, Test accuracy = 0.5725
cuml: Training accuracy = 0.6787777543067932, Test accuracy = 0.5057500004768372

depth = 7
sklearn: Training accuracy = 0.7041055555555555, Test accuracy = 0.54715
cuml: Training accuracy = 0.7040888667106628, Test accuracy = 0.48510000109672546

depth = 8
sklearn: Training accuracy = 0.72555, Test accuracy = 0.38175
cuml: Training accuracy = 0.7254166603088379, Test accuracy = 0.24729999899864197

So the issue is only partially solved at the moment and needs further analysis.

[hcho3]

@vinaydes The gap in the testing accuracy is explained by how the test set was chosen. The notebook from #2561 selects the last 20000 rows of the data as the test set, and the test set happens to be "unlucky" choice such that cuML model does not perform well. The distribution of data points in the test set is very skewed:

| | Training set | Test set |
|--|--|--|
| Good loan (label = 1) | 100000 | 0 |
| Bad loan (label = 0) | 80000 | 20000 |

From this information alone, it is difficult to argue that the test set and the training set came from the same data distribution. Since the test set was not representative, the gap in the testing accuracy between cuML and scikit-learn does not accurately tell us whether cuML is actually worse than scikit-learn.

We want to estimate the generalization ability of a model without getting hit by a bad choice of the test set. To do this, we perform a statistical procedure known as cross-validation. The idea is that we try many possible ways of splitting the data into the training / test sets. For example, here are the steps for 10-fold cross validation:

1. Shuffle the data. This prevents the situation we had above, where all the bad loans got concentrated to a particular region of the table.
2. Split the data into 10 equal folds.
3. For each k from 0 to 9: repeat:
   
   
   
   • Designate the k-th fold as the test set, and designate the remaining 9 folds as the training set. Now fit the RF model using the training set. Then evaluate the model with the test set.
   
   
4. The previous step gave us 10 distinct measurements of the test accuracy. Now average the 10 measurements to get the cross-validated (CV) accuracy.

The idea is to try multiple possible test sets, so that "unlucky" choices of the test set get even out with "lucky" choices of the test set, and we can accurately gauge how well cuML RF performs compared to scikit-learn RF.

[hcho3]

@vinaydes I salvaged #2561 to improve the testing procedure. See my previous comment https://github.com/rapidsai/cuml/issues/2518#issuecomment-664697240.

Takeaways

• cuML achieves competitive cross-validated (CV) accuracy compared with scikit-learn, if the number of bins is set to 512.
• On the other hand, the default value of bins (8) causes cuML perform significantly worse.
• Setting number of bins to 512 makes cuML quite slow. With 8 bins, cuML RF is lightning fast. Let's see if we can find a happy medium between 8 and 512.

EDIT. See comment https://github.com/rapidsai/cuml/issues/2518#issuecomment-664750973

Results with 10-fold cross-validation

A single tree, no bootstrap sampling, using all features in splits

In terms of CV accuracy, cuML RF performs competitively with scikit-learn's RF, when using 512 bins. We report the mean CV accuracy as well as the standard deviation, since we collect accuracy over 10 choices of test sets. Log:

max_depth = 1, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 1, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 1, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 2, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 2, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 2, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 3, sklearn: CV accuracy = 0.8154620217548267 (std=0.001131272441770603)
max_depth = 3, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 3, n_bins = 512, cuml: CV accuracy = 0.8154620110988617 (std=0.0011312712844164514)
max_depth = 4, sklearn: CV accuracy = 0.8201405791918741 (std=0.0011592923512966393)
max_depth = 4, n_bins = 8, cuml: CV accuracy = 0.7992170929908753 (std=0.0010849865963664116)
max_depth = 4, n_bins = 512, cuml: CV accuracy = 0.820138281583786 (std=0.0011612556362821025)
max_depth = 5, sklearn: CV accuracy = 0.8214392832503957 (std=0.0012219652208284289)
max_depth = 5, n_bins = 8, cuml: CV accuracy = 0.8010297536849975 (std=0.0012023143996219528)
max_depth = 5, n_bins = 512, cuml: CV accuracy = 0.8214346826076507 (std=0.001223494073740236)
max_depth = 6, sklearn: CV accuracy = 0.825032390875369 (std=0.0010324289056678664)
max_depth = 6, n_bins = 8, cuml: CV accuracy = 0.8025708615779876 (std=0.00130990066761653)
max_depth = 6, n_bins = 512, cuml: CV accuracy = 0.8250270128250122 (std=0.0010330292690912599)
max_depth = 7, sklearn: CV accuracy = 0.8252870685948622 (std=0.001139214569830148)
max_depth = 7, n_bins = 8, cuml: CV accuracy = 0.8030978679656983 (std=0.0014123356813169786)
max_depth = 7, n_bins = 512, cuml: CV accuracy = 0.8253438413143158 (std=0.0011066300174844262)
max_depth = 8, sklearn: CV accuracy = 0.827621360944768 (std=0.0010276522491667046)
max_depth = 8, n_bins = 8, cuml: CV accuracy = 0.8036992728710175 (std=0.0013723448164754228)
max_depth = 8, n_bins = 512, cuml: CV accuracy = 0.8275431156158447 (std=0.0010271040431088488)

For max_depth = 8, cuML gives a slightly lower CV accuracy (0.82754) than scikit-learn (0.82762), but the difference is statistically insignificant (p = 0.8667 >> 0.05) given the standard deviation.

100 trees, with bootstrap sampling, using a subset of features in splits

This plot is using the hyperparameters

params = {
    'n_estimators': 100,
    'max_features': 'auto',
    'bootstrap': True
}

to fit 100-tree RF with bootstrap sampling. Log:

max_depth = 1, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 1, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 1, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 2, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 2, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 2, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 3, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 3, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 3, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 4, sklearn: CV accuracy = 0.7992838328120482 (std=0.0011294740445770785)
max_depth = 4, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 4, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 5, sklearn: CV accuracy = 0.7994641019920469 (std=0.0011285428028819774)
max_depth = 5, n_bins = 8, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 5, n_bins = 512, cuml: CV accuracy = 0.7992838323116302 (std=0.0011294712288540552)
max_depth = 6, sklearn: CV accuracy = 0.8014984563636478 (std=0.0020044352134864873)
max_depth = 6, n_bins = 8, cuml: CV accuracy = 0.7993076145648956 (std=0.0011267189606830993)
max_depth = 6, n_bins = 512, cuml: CV accuracy = 0.7997824490070343 (std=0.0011252340532052576)
max_depth = 7, sklearn: CV accuracy = 0.803785190212901 (std=0.0014393990094500617)
max_depth = 7, n_bins = 8, cuml: CV accuracy = 0.7995369791984558 (std=0.0011128446755883374)
max_depth = 7, n_bins = 512, cuml: CV accuracy = 0.8016871631145477 (std=0.0012400964621939077)
max_depth = 8, sklearn: CV accuracy = 0.8061777830961955 (std=0.0008289406350271656)
max_depth = 8, n_bins = 8, cuml: CV accuracy = 0.8003800213336945 (std=0.0011987357355839414)
max_depth = 8, n_bins = 512, cuml: CV accuracy = 0.8055610358715057 (std=0.0012534725053860625)

For max_depth = 8, cuML gives a slightly lower CV accuracy (0.80556) than scikit-learn (0.80618), but the difference is statistically insignificant (p = 0.2107 >> 0.05) given the standard deviation.

Code

preprocess.py: Process the mortgage CSV dataset into NumPy arrays:

CLICK ME

import cudf
import cupy
import numpy as np
import pandas as pd

from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder

from cuml.ensemble import RandomForestClassifier as cuml_RandomForestClassifier
from cuml.metrics.accuracy import accuracy_score
from cuml.preprocessing import LabelEncoder as cuml_LabelEncoder

# Download from https://www.kaggle.com/wendykan/lending-club-loan-data
DATAPATH = 'data/loan.csv'

columns_to_use = [
    'loan_amnt',
    'funded_amnt',
    'funded_amnt_inv',
    'term',
    'int_rate',
    'installment',
    'grade',
    'sub_grade',
    'emp_title',
    'emp_length',
    'home_ownership',
    'annual_inc',
    'verification_status',
    'issue_d',
    'loan_status',
    'pymnt_plan',
    'purpose',
    'title',
    'zip_code',
    'addr_state',
    'dti',
    'delinq_2yrs',
    'earliest_cr_line',
    'inq_last_6mths',
    'open_acc',
    'pub_rec',
    'revol_bal',
    'revol_util',
    'total_acc',
    'initial_list_status',
    'out_prncp',
    'out_prncp_inv',
    'total_pymnt',
    'total_pymnt_inv',
    'total_rec_prncp',
    'total_rec_int',
    'total_rec_late_fee',
    'recoveries',
    'collection_recovery_fee',
    'last_pymnt_d',
    'last_pymnt_amnt',
    'next_pymnt_d',
    'last_credit_pull_d',
    'collections_12_mths_ex_med',
    'policy_code',
    'application_type',
    'verification_status_joint',
    'acc_now_delinq',
    'tot_coll_amt',
    'tot_cur_bal',
    'open_acc_6m',
    'open_act_il',
    'open_il_12m',
    'open_il_24m',
    'total_bal_il',
    'open_rv_12m',
    'open_rv_24m',
    'max_bal_bc',
    'all_util',
    'total_rev_hi_lim',
    'inq_fi',
    'total_cu_tl',
    'inq_last_12m',
    'acc_open_past_24mths',
    'avg_cur_bal',
    'chargeoff_within_12_mths',
    'delinq_amnt',
    'mo_sin_old_rev_tl_op',
    'mo_sin_rcnt_rev_tl_op',
    'mo_sin_rcnt_tl',
    'mort_acc',
    'num_accts_ever_120_pd',
    'num_actv_bc_tl',
    'num_actv_rev_tl',
    'num_bc_sats',
    'num_bc_tl',
    'num_il_tl',
    'num_op_rev_tl',
    'num_rev_accts',
    'num_rev_tl_bal_gt_0',
    'num_sats',
    'num_tl_30dpd',
    'num_tl_90g_dpd_24m',
    'num_tl_op_past_12m',
    'pct_tl_nvr_dlq',
    'pub_rec_bankruptcies',
    'tax_liens',
    'tot_hi_cred_lim',
    'total_bal_ex_mort',
    'total_bc_limit',
    'total_il_high_credit_limit',
    'sec_app_earliest_cr_line',
    'hardship_flag',
    'disbursement_method',
    'debt_settlement_flag',
]

DONT_INCLUDE_TRAINING = ["loan_status", # NEVER INCLUDE THIS!
                         "total_pymnt",
                         "total_pymnt_inv",
                         "total_rec_prncp",
                         "total_rec_int",
                         "total_rec_late_fee",

                         "recoveries",
                         "collection_recovery_fee",

                         "last_pymnt_d",
                         "last_pymnt_amnt",
                         "next_pymnt_d",
                         "policy_code",
                         "acc_now_delinq",
                         "delinq_amnt",
                         "hardship_flag",
                         "debt_settlement_flag"
                         ]

pd_df = pd.read_csv(DATAPATH, usecols=columns_to_use, low_memory=False)
pd_df = pd_df.loc[pd_df['loan_status'].isin(["Charged Off", "Fully Paid"])]
pd_df.update(pd_df.select_dtypes(include=[np.number]).fillna(0))
pd_df.update(pd_df.select_dtypes(include=[np.dtype('O')]).fillna('0'))
pd_text_columns = set(pd_df.columns[pd_df.dtypes == np.dtype('O')])

pd_df['loan_status'] = pd_df['loan_status'].replace({'Charged Off': 0, 'Fully Paid': 1})

for column in pd_text_columns:
    pd_df[column] = LabelEncoder().fit_transform(pd_df[column])

X = pd_df.drop(DONT_INCLUDE_TRAINING, axis=1).to_numpy(dtype=np.float32)
y = pd_df["loan_status"].to_numpy(dtype=np.float32)

print(X.shape, y.shape)
np.save('data/loans_X.npy', X)
np.save('data/loans_y.npy', y)

run_experiment.py: Perform 10-fold cross validation

import numpy as np
import json

from sklearn.ensemble import RandomForestClassifier
from cuml.common import logger
from cuml.ensemble import RandomForestClassifier as cuml_RandomForestClassifier

from sklearn.model_selection import cross_val_score, KFold

# Preprocessed data
X = np.load('data/loans_X.npy')
y = np.load('data/loans_y.npy')

params = {
    'n_estimators': 1,
    'max_features': 1.0,
    'bootstrap': False
}

skl_cv_mean = []
skl_cv_std = []
cuml_cv_mean = {}
cuml_cv_std = {}
cuml_cv_mean[8] = []
cuml_cv_std[8] = []
cuml_cv_mean[512] = []
cuml_cv_std[512] = []

cv_fold = KFold(n_splits=10, shuffle=True, random_state=2020)

for max_depth in range(1, 9):
    skl_clf = RandomForestClassifier(n_jobs=-1, max_depth=max_depth, **params)
    skl_cv_acc = cross_val_score(skl_clf, X, y, cv=cv_fold)
    print(f'max_depth = {max_depth}, sklearn: CV accuracy = {skl_cv_acc.mean()} (std={skl_cv_acc.std()})')
    skl_cv_mean.append(skl_cv_acc.mean())
    skl_cv_std.append(skl_cv_acc.std())
    del skl_clf

    for n_bins in [8, 512]:
        cuml_clf = cuml_RandomForestClassifier(n_bins=n_bins, max_depth=max_depth + 1, **params)
        cuml_cv_acc = cross_val_score(cuml_clf, X, y, cv=cv_fold)
        print(f'max_depth = {max_depth}, n_bins = {n_bins}, cuml: CV accuracy = {cuml_cv_acc.mean()} (std={cuml_cv_acc.std()})')
        cuml_cv_mean[n_bins].append(cuml_cv_acc.mean())
        cuml_cv_std[n_bins].append(cuml_cv_acc.std())
        del cuml_clf

obj = {
    'skl_cv_mean': skl_cv_mean,
    'skl_cv_std': skl_cv_std,
    'cuml_cv_mean': cuml_cv_mean,
    'cuml_cv_std': cuml_cv_std
}

with open('record.json', 'w') as f:
    json.dump(obj, f)

plot.py: Generate plots

import json
import numpy as np
import matplotlib.pyplot as plt

with open('dump.json', 'r') as f:
    record = json.load(f)

bar_width = 0.25

r1 = np.arange(len(record['skl_cv_mean']))
r2 = r1 + bar_width
r3 = r2 + bar_width

kwargs = {'width': bar_width, 'edgecolor': 'white', 'ecolor': 'black', 'capsize': 10}

plt.figure()
plt.bar(r1, record['skl_cv_mean'], yerr=record['skl_cv_std'], label='skl', **kwargs)
plt.bar(r2, record['cuml_cv_mean']['8'], yerr=record['cuml_cv_std']['8'],
        label='cuml (n_bins=8)', **kwargs)
plt.bar(r3, record['cuml_cv_mean']['512'], yerr=record['cuml_cv_std']['512'],
        label='cuml (n_bins=512)', **kwargs)
plt.xticks(r1 + bar_width, np.arange(1, 9))
plt.xlabel('Max tree depth')
plt.ylabel('CV accuracy')
plt.ylim([0.75, 0.85])
plt.legend(loc='best')
plt.tight_layout()
plt.savefig('skl_vs_cuml.png', dpi=150)

[hcho3]

Update: when I increased max_depth to 53, the CV accuracy is now worse.

Single tree (n_estimators': 1, 'max_features': 1.0, 'bootstrap': False)

max_depth = 53, sklearn: CV accuracy = 0.7539611249273823 (std=0.001020878693017557)
max_depth = 53, n_bins = 8, cuml: CV accuracy = 0.7114697873592377 (std=0.0013110998742815142)
max_depth = 53, n_bins = 512, cuml: CV accuracy = 0.7521515309810638 (std=0.0011663341752869891)

The difference is statistically significant (p = 0.0017 << 0.05).

100-tree RF (n_estimators': 100, 'max_features': 'auto', 'bootstrap': True)

max_depth = 53, sklearn: CV accuracy = 0.8264492284835798 (std=0.0010266766758390317)
max_depth = 53, n_bins = 8, cuml: CV accuracy = 0.8024834156036377 (std=0.0011626662742013204)
max_depth = 53, n_bins = 512, cuml: CV accuracy = 0.8099511563777924 (std=0.000996424137152377)

The difference is statistically significant (p < 0.0001 << 0.05).

[robocopnixon]

I think it's interesting that 2% seems to be such a commonly occurring gap between SKL and cuML; others have reported similarly to me in totally different applications.

[JohnZed]

Thank you, Philip! I think we should be open to increasing the default n_bins if it's really helpful.

[hcho3]

Follow-up to https://github.com/rapidsai/cuml/issues/2518#issuecomment-664716386. I ran more exhaustive benchmark with more combination of tree depths.

Setting max_depth to 10 or more causes cuML to fall behind scikit-learn.

Single tree

Click to enlarge:

100 trees

Click to enlarge:

[hcho3]

Since the full mortgage dataset is quite big (1303607 rows), I reduced it by taking 1% of the rows randomly:

np.random.seed(2020)
nrow = y.shape[0]
idx = np.sort(np.random.choice(nrow, nrow // 100, replace=False))
X = X[idx, :]
y = y[idx]
np.save('loans_X_sample.npy', X)
np.save('loans_y_sample.npy', y)

loans_X_sample.npy (13036 rows * 79 columns)
loans_y_sample.npy (13036 rows)

I ran 10-fold cross-validation with the sampled data, and I am observing significant different between cuML and sklearn for max_depth >= 11. (To remove effect of bootstrap sampling, I fit a single tree with n_estimators': 1, 'max_features': 1.0, 'bootstrap': False.)

[hcho3]

Hi everyone: I found a possible cause of the accuracy drop: bootstrapping. See #2895 for more details.

[JohnZed]

Improvements to the new backend show significant progress, but we'll keep this open in next release until we get the last few tests done.