Cuml: [BUG] LogisticRegression suffers from accuracy loss when penalty is enabled

[teju85]

Describe the bug
When the penalty term is enabled, it appears that the resulting accuracy of the cuML model is worse when compared to the equivalent one from scikit-learn.

Steps/Code to reproduce bug
@tfeher was able to reproduce this internally. Tagging him to add a reproducer when he's ready.

Also tagging @yasmina-altair as she too has been facing similar issues.

Expected behavior
Accuracy and/or resulting model from cuML should be as close as possible to scikit-learn's.

Environment details (please complete the following information):

• Environment location: any
• Linux Distro/Architecture: any
• GPU Model/Driver: any
• CUDA: [10.2]
• Method of cuDF & cuML install: issue is seen even on our latest nightly.

[teju85]

@yasmina-altair, please do provide us a way to reproduce your findings when you are ready.

[tfeher]

Here is a reproducer:

import numpy as np
import sklearn.datasets
import sklearn.linear_model
import cuml

X, y = sklearn.datasets.make_classification(
   n_samples=10000, n_features=964, n_informative=62, n_redundant=0,
   n_classes=2, n_clusters_per_class=4, shuffle=False, random_state=42)
y = y.astype(np.float64)

params = {'penalty':'l2', 'C':0.1, 'tol':1e-4, 'fit_intercept': True, 'max_iter':2000, 'verbose':False}
cls = cuml.linear_model.LogisticRegression(**params)
cls.fit(X, y)
print('cuML score train: {0:7.4f}'.format(cls.score(X, y)))

skl_cls = sklearn.linear_model.LogisticRegression(**params)
skl_cls.fit(X, y)
print('skl  score train: {0:7.4f}'.format(skl_cls.score(X, y)))

Which produces output:

cuML score train:  0.7370
skl  score train:  0.7739

The difference in the score can be much larger, depending on the dataset and the number of samples.

[tfeher]

The actual problem is caused by how the regularization strength term is interpreted. cuML's implementation has an additional n_samples factor when the regularization is applied. With the current version of cuML, one needs to set C_cuml = n_samples * C_skl to get the same results.

So after the following changes we get the same results:

cu_params = params.copy()
cu_params['C'] = params['C'] * X.shape[0]
cls = cuml.linear_model.LogisticRegression(**cu_params)

We will fix the handling of C parameter in cuML, so that it is interpreted the same way as in scikit-learn.

[JohnZed]

Nice analysis, @tfeher!

[tfeher]

Thanks to @yjk21 for finding the root cause!

[yasmina-altair]

Hi,
seems that you found the source of the issue. As a reference, here's another example that I just created in colab:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression as sk_log_reg
from sklearn.metrics import roc_auc_score
from cuml.linear_model import LogisticRegression as cu_log_reg

X, y = load_breast_cancer(return_X_y=True)

sk_clf = sk_log_reg()
sk_clf.fit(X, y)

cu_clf = cu_log_reg()
cu_clf.fit(X, y.astype(float))


print("accuracy sklearn: ", sk_clf.score(X, y))
print("accuracy cuml: ", cu_clf.score(X, y))

print("AUC sklearn: ", roc_auc_score(y, sk_clf.predict(X)))
print("AUC cuml: ", roc_auc_score(y, cu_clf.predict(X)))
----------------------------------------------------------------------------------------------------------------------------------------------
accuracy sklearn:  0.9472759226713533
accuracy cuml:  0.6274164915084839
AUC sklearn:  0.9397825167802971
AUC cuml:  0.5

AUC is 0.5 for cuml LogisticRegression, there is no class separation, the model is always predicting one class. When I run:

cu_clf.predict(X)

the output is an array of ones.

[tfeher]

The regularization strength is corrected, and the fix is available in the nightly images. Using this the first problem disappears.

The problem reported by @yasmina-altair still exists. It is not related to the penalty parameter (the example does not apply any penalty factors). The problem is related to the scaling of the input data, the problem disappears if apply a standard scaler:

X = sklearn.preprocessing.StandardScaler().fit_transform(X)

I will investigate this further.

[yjk21]

Hey @tfeher
The problem seems to be that log.reg. in sklearn applies L2 regularization with C=1 by default: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
But as discussed offline, this seems not be the reason for the problem.

[tfeher]

@yasmina-altair The numerical problems that can occur with unscaled data will be fixed by #2543 . After that is merged, the reproducer that you gave should work correctly. (The default values for regularization parameters are in agreement with Sklearn).

Thanks @yjk21 for the helpful discussions.

[yasmina-altair]

awesome, thank you for the update!

[tfeher]

The nightly images now contain the fix for both the problems encountered in this issue.