Handson-ml: Chapter 3, RandomForest scores

[JoeLeonard212]

[ageron]

Hi @JoeLeonard212 ,
Could you please describe what the issue is?
Thanks!

[JoeLeonard212]

Hi! It's not a technical issue, I just didnot understand how do you obtain scores from probas to plot the ROC curve in this line:
y_scores_forest = y_probas_forest[:, 1] # score = proba of positive class

Thank you @ageron

[ageron]

Hi @JoeLeonard212 ,
To plot a ROC curve, you need a label and a score for each instance. The score can be the estimated probability, the log probability, or any other value as long as it is a strictly increasing function of the estimated probability.
In short, you can use the estimated probabilities directly as the thresholds to plot the ROC curve, that's why I wrote "#score = proba of positive class".

I hope this helps.

[ageron]

Just to be clear, if you use an SVM classifier (sklearn.svm.SVC), it typically does not estimate probabilities or even log probabilities. It just outputs a score for each class and each instance, that's it. Higher score means higher confidence, and that's all we need to use these scores to plot a ROC curve. If you set probability=True when creating the SVC, it will learn how to map these scores to probabilities, and therefore you will have a predict_proba() method and a predict_log_proba() method after training. You can use the scores, or the probas, or even the log probas, and you should get exactly the same ROC curve.

[JoeLeonard212]

Hi @JoeLeonard212 ,
"...or any other value as long as it is a strictly increasing function of the estimated probability."
"You can use the scores, or the probas, or even the log probas, and you should get exactly the same ROC curve."

Ok got it now, the previous sentence in the book confused me: "But to plot a ROC curve, you need scores, not probabilities. A simple solution is to use the positive class's probabilities as the score." That sounded a bit contradictory but now is clear, thank you for your work and your reply!

[ageron]

Ah thanks, you're right, that's a mistake in the book, sorry for the confusion: probabilities work fine. I'll clarify that in the 2nd edition.