Recommenders: [ASK] NCF results are less than the original paper

[Beheshtian]

Description

Hi
The NCF results are less than the original paper.
I tested 1M MovieLens dataset and the HR and NDCG results are lower than the results in the paper.
What is the problem?
tnx

Other Comments

recommenders/notebooks/02_model/ncf_deep_dive.ipynb

[micaleel]

For starters, that notebook (i.e. recommenders/notebooks/02_model/ncf_deep_dive.ipynb) uses a chrono split with 75% of the dataset used for training. Whereas, the paper used a leave-one-out protocol.

It would help to use the right hyper-parameters to get the results reported in the paper.

[Beheshtian]

For starters, that notebook (i.e. recommenders/notebooks/02_model/ncf_deep_dive.ipynb) uses a chrono split with 75% of the dataset used for training. Whereas, the paper used a leave-one-out protocol.

It would help to use the right hyper-parameters to get the results reported in the paper.

thanks for your kindly response micaleel.
but in the section (3.4.3 "Leave-one-out" Evaluation) the leave-one-out protocol has been used and the HR ratio for MovieLens 1M is 0.52 but in the paper the HR is 0.68 .
I also set the hyper-parameters according to the paper.
I hope you'll be in touch again soon

[loomlike]

@micaleel @Beheshtian Thank you for the good discussion, and totally agree that we should refer the known hyper-parameters if possible. But at the same time, we should be careful to use the known hyper-parameters to the same train/test split of the source of the hyper-parameters so that our test set does not include any validation samples the original source used to find the best hyper-parameters.

And, often, finding and using the exact same splits and hyper-parameters to produce the same results in the papers are very difficult. Here is an interesting paper about reproducibility, which won the best paper award in RecSys 2019:

... we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort.

Another way to find good hyper-parameters is to utilize automated tools. We have good examples of them under notebooks/04_model_select_and_optimize.

[Beheshtian]

@micaleel @Beheshtian Thank you for the good discussion, and totally agree that we should refer the known hyper-parameters if possible. But at the same time, we should be careful to use the known hyper-parameters to the same train/test split of the source of the hyper-parameters so that our test set does not include any validation samples the original source used to find the best hyper-parameters.

And, often, finding and using the exact same splits and hyper-parameters to produce the same results in the papers are very difficult. Here is an interesting paper about reproducibility, which won the best paper award in RecSys 2019:

... we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort.

Another way to find good hyper-parameters is to utilize automated tools. We have good examples of them under notebooks/04_model_select_and_optimize.

Thank you for your kindly response loomlike.
So can I refer to the Microsoft implementation results of NCF model in my paper?
I hope you'll be in touch again soon.

[yueguoguo]

Note, to fairly compare with other algorithms, the "leave-one-out" protocol is adjusted in the deep dive notebook where NCF is illustrated.

See the notes in the notebook,

We truncated the ranked list at 10 for both metrics. As such, the HR intuitively measures whether the test item is present on the top-10 list, and the NDCG accounts for the position of the hit by assigning higher scores to hits at top ranks.
This means there are TOP_K other than 1 recommendation items for evaluation. This is different from the settings used in the paper.

# Other than 1, TOP_K items are used for calculating the hit ratio in the testing data
k = TOP_K

ndcgs = []
hit_ratio = []

for b in data.test_loader():
    user_input, item_input, labels = b
    output = model.predict(user_input, item_input, is_list=True)

    output = np.squeeze(output)
    rank = sum(output >= output[0])
    if rank <= k:
        ndcgs.append(1 / np.log(rank + 1))
        hit_ratio.append(1)
    else:
        ndcgs.append(0)
        hit_ratio.append(0)

Note 1: In exact leave-one-out evaluation protocol, we select only one of the latest items interacted with a user as test data for each user. But in this notebook, to compare with other algorithms, we select latest 25% dataset as test data. So this is an artificial "leave-one-out" evaluation only showing how to use test_loader and how to calculate metrics like the original paper. You can reproduce the real leave-one-out evaluation by changing the way of splitting data.

This corresponds to the comment by @micaleel that the stratified data split is performed chronologically with a pre-defined split ratio, with which 25% of the total items that a user has interacted with are used for testing. I.e.,

# The python_chrono_split function splits data chronologically with stratification on user (in this case) by default.
train, test = python_chrono_split(df, 0.75)

This is different from the original paper, where only the latest one item that a user has interacted with is used for testing.

Hope this helps clarify.

[Beheshtian]

Note, to fairly compare with other algorithms, the "leave-one-out" protocol is adjusted in the deep dive notebook where NCF is illustrated.

See the notes in the notebook,

We truncated the ranked list at 10 for both metrics. As such, the HR intuitively measures whether the test item is present on the top-10 list, and the NDCG accounts for the position of the hit by assigning higher scores to hits at top ranks.
This means there are TOP_K other than 1 recommendation items for evaluation. This is different from the settings used in the paper.

# Other than 1, TOP_K items are used for calculating the hit ratio in the testing data
k = TOP_K

ndcgs = []
hit_ratio = []

for b in data.test_loader():
    user_input, item_input, labels = b
    output = model.predict(user_input, item_input, is_list=True)

    output = np.squeeze(output)
    rank = sum(output >= output[0])
    if rank <= k:
        ndcgs.append(1 / np.log(rank + 1))
        hit_ratio.append(1)
    else:
        ndcgs.append(0)
        hit_ratio.append(0)

Note 1: In exact leave-one-out evaluation protocol, we select only one of the latest items interacted with a user as test data for each user. But in this notebook, to compare with other algorithms, we select latest 25% dataset as test data. So this is an artificial "leave-one-out" evaluation only showing how to use test_loader and how to calculate metrics like the original paper. You can reproduce the real leave-one-out evaluation by changing the way of splitting data.

This corresponds to the comment by @micaleel that the stratified data split is performed chronologically with a pre-defined split ratio, with which 25% of the total items that a user has interacted with are used for testing. I.e.,

# The python_chrono_split function splits data chronologically with stratification on user (in this case) by default.
train, test = python_chrono_split(df, 0.75)

This is different from the original paper, where only the latest one item that a user has interacted with is used for testing.

Hope this helps clarify.

Thank you for your kindly response yueguoguo.
I don't understand the below sentence:
"This means there are TOP_K other than 1 recommendation items for evaluation. This is different from the settings used in the paper."
So do you use 100 negative items and calculate HR@10 and NDCG@10 with 11 items in the leave-one-out evaluation (because top_k is 10) in the ncf_deep_dive.ipynb and the settings used in the paper include 99 negative items and used 10 items for evaluation and the results are also correspond to HR@10 and NDCG@10 ?
I hope you'll be in touch again soon.

[yueguoguo]

@Beheshtian from Figure 7 in the paper, I think it sweeps negative samples per positive sample from 1 to 10.

Note in the NCF deep dive notebook, the main objective IS NOT to replicate the results shown in the paper. The purpose is really to help readers understand the algorithm, the evaluation methods, etc. And the evaluation results can be different from the paper because of different evaluation protocols, model parameters, etc.