Recommenders: [ASK] Unusually low precision and recall for KNN Top-N Recommenders

[geriskenderi]

Description

I'm trying to implement user and item based knn collaborative filtering using the combination of cornac and reco_utils. I managed to build the models just fine but when evaluating the rankings the precision score is extremely low. Here's a screenshot to help understand:

Here's the code I'm using to build the knn models and run the evaluation:

K = 80

# KNN models
user_knn_cosine = cornac.models.UserKNN(k=K, similarity="cosine", name="UserKNN-Cosine")
user_knn_pearson = cornac.models.UserKNN(
    k=K, similarity="pearson", name="UserKNN-Pearson"
)
user_knn_amp = cornac.models.UserKNN(
    k=K, similarity="cosine", amplify=2.0, name="UserKNN-Amplified"
)
user_knn_idf = cornac.models.UserKNN(
    k=K, similarity="cosine", weighting="idf", name="UserKNN-IDF"
)
user_knn_bm25 = cornac.models.UserKNN(
    k=K, similarity="cosine", weighting="bm25", name="UserKNN-BM25"
)

uknn_models = {
    "cosine": user_knn_cosine,
    "pearson": user_knn_pearson,
    "amp": user_knn_amp,
    "idf": user_knn_idf,
    "bm25": user_knn_bm25
}

# @5
print("Results @5")
k = 5
for model in uknn_models:
    print(model, "...\n")
    current_model = uknn_models[model]
    current_model.fit(train_set)
    # Predict all unseen entries
    all_predictions = predict_ranking(current_model, train, usercol='userID', itemcol='itemID', remove_seen=True)
    print(all_predictions.head())
    eval_map = map_at_k(test, all_predictions, col_prediction='prediction', k=k)
    eval_ndcg = ndcg_at_k(test, all_predictions, col_prediction='prediction', k=k)
    eval_precision = precision_at_k(test, all_predictions, col_prediction='prediction', k=k)
    eval_recall = recall_at_k(test, all_predictions, col_prediction='prediction', k=k)

    print("NDCG:\t%f" % eval_ndcg, "Precision@K:\t%f" % eval_precision, "Recall@K:\t%f" % eval_recall, sep='\n')
    print()

The train test split was done by using python_random_split

Anyone else has ran into this situation before? Any suggestions or tips?

[gramhagen]

@tqtg any thoughts here?

[tqtg]

@geriskenderi did you manage to try with different settings of nearest neighbors?

[geriskenderi]

@tqtg yep I tried with 40, 60, 80 which should be pretty standard numbers for KNN algos with Movielens 100K

[tqtg]

The implementations of KNN methods in Cornac follow closely the ones described in [1]. The emphasis is on predicting rating, thus performance is measured using RMSE/MAE.

The main difference as compared to implementations in other libraries lies in the definition of nearest neighbors.

• Cornac follows the definition of nearest neighbors who also rate the predicted item, which is described in the original paper [2]. Thus, the nearest neighbors are re-identified for each prediction. This implementation focuses more on estimating rating values.
• In some other implementations, the list of nearest neighbors for each user is pre-computed and will not be changed during prediction. This implementation might be more consistent in terms of ranking as the predictions for a particular user are always inferred from the same set of nearest neighbors.

Have said that I think the objectives of the two implementations are quite different. More experiments are required to fully understand the difference in terms of performance.

If you are interested, feel free to refer to our implementation in Cornac and have deeper discussions.

References
[1] Aggarwal, C. C. (2016). Recommender systems (Vol. 1). Cham: Springer International Publishing.
[2] Breese, J. S., Heckerman, D., & Kadie, C. (2013). Empirical analysis of predictive algorithms for collaborative filtering. arXiv preprint arXiv:1301.7363.

[geriskenderi]

Yes absolutely, User and Item based KNN focus on rating prediction. What I was trying to do was transform them into Top-K recommenders. As you said more experiments are needed to understand the difference. I will try different settings. Thanks for the help.

[gramhagen]

thanks, feel free to open a new issue if there is more discussion needed

[MauritiusKlein]

Thanks for the issue and the reply!
I was searching for an answer to this question too.
As I compared different algorithms I realized that Item/User KNN especially had very bad precision and recall values. However as your precision recall metric returns reasonable values for other approaches like e.g. BPR, I was sure that it was some kind of issue related to the KNN implementation.
This are my Prec/Recall Results with some different cornac algorithms on the MovieLens 100K.
(Ignore wALS, that is another implementation)
I was hoping to get results similar to the ones from https://www.librec.net/release/v1.3/example.html