Recommenders: [ASK] MAP_AT_K cross check

[seancaodo]

Description

Hi,

I was doing some calculations for MAP_AT_K off SAR example: https://github.com/Microsoft/Recommenders/blob/master/notebooks/00_quick_start/sar_movielens.ipynb
and noticed the difference between MAP and Precision@K: 0.105815 and 0.326617.

The Precision@K is re-creatable at 0.326617

### Precision@K

new_df = pd.merge(top_k, test, how='left', on=['UserId','MovieId'])
from dplython import (DplyFrame, X, select, sift, head, arrange, mutate, group_by, summarize, DelayFunction)
new_df = DplyFrame(new_df)
where = DelayFunction(np.where)
new_df = new_df >> mutate(Relevant = where(X.Rating > 0, 1, 0))
dd2 = new_df >> group_by(X.UserId) >> summarize(mm = X.Relevant.sum()/10)
dd2['mm'].mean()

### End of Precision@K

But MAP might open for a discussion:

### MAP 1

new_df = pd.merge(top_k, test, how='left', on=['UserId','MovieId'])
from dplython import (DplyFrame, X, select, sift, head, arrange, mutate, group_by, summarize, DelayFunction)
new_df = DplyFrame(new_df)
where = DelayFunction(np.where)
new_df = new_df >> mutate(Relevant = where(X.Rating > 0, 1, 0))
new_df = new_df >> arrange(X.UserId, -X.prediction)
new_df['ForCount'] = 1
new_df['cumsum'] = new_df.groupby('UserId')['Relevant'].transform(pd.Series.cumsum)
new_df['cumsum2'] = new_df.groupby('UserId')['ForCount'].transform(pd.Series.cumsum)
new_df['prec'] = new_df['cumsum'] / new_df['cumsum2']
new_df['prec'] = new_df['prec'] * new_df['Relevant']
dd2 = new_df >> group_by(X.UserId) >> summarize(ss = X.prec.sum())
dd2['prec2'] = dd2['ss'] / 10
dd2['prec2'].mean()

### End of MAP 1

The above lines had MAP calculated as 0.2271

Instead of dividing the precision rates by 10, if we divide them by total Movies rated by user in test set like below:

######## MAP 2

new_df = pd.merge(top_k, test, how='left', on=['UserId','MovieId'])
from dplython import (DplyFrame, X, select, sift, head, arrange, mutate, group_by, summarize, DelayFunction)
new_df = DplyFrame(new_df)
where = DelayFunction(np.where)
new_df = new_df >> mutate(Relevant = where(X.Rating > 0, 1, 0))
new_df = new_df >> arrange(X.UserId, -X.prediction)
new_df['ForCount'] = 1
new_df['cumsum'] = new_df.groupby('UserId')['Relevant'].transform(pd.Series.cumsum)
new_df['cumsum2'] = new_df.groupby('UserId')['ForCount'].transform(pd.Series.cumsum)
new_df['prec'] = new_df['cumsum'] / new_df['cumsum2']
new_df['prec'] = new_df['prec'] * new_df['Relevant']
dd2 = new_df >> group_by(X.UserId) >> summarize(ss = X.prec.sum())

dd3 = test >> group_by(X.UserId) >> mutate(nn = 1) >> summarize(m = X.nn.sum())
dd4 = pd.merge(dd2, dd3, how='left', on = ['UserId'])
dd4['prec2'] = dd4['ss'] / dd4['m']
dd4['prec2'].mean()

########## End of MAP2

Then the MAP rate is the same as given in the example: 0.1058

Question is why do we divide by total rated Movies in test set only? For UserId = 1, this from test set is 69 but in total he/she rated 272 movies. Why do we divide by 69 and not 272?

Reference: I couldn't get to your advised reference https://people.cs.umass.edu/~jpjiang/cs646/03_eval_basics.pdf so tried a different source here:
http://sdsawtelle.github.io/blog/output/mean-average-precision-MAP-for-recommender-systems.html
which seems to agree with one from Stanfor Uni at:
http://web.stanford.edu/class/cs276/handouts/EvaluationNew-handout-6-per.pdf

I may miss out some important details too.

Thanks,

In which platform does it happen?

CPU

How do we replicate the issue?

I have pasted the codes using Dplython module.

Expected behavior (i.e. solution)

Other Comments

[miguelgfierro]

thanks @seancaodo, we will take a look

[yueguoguo]

@seancaodo Thanks for the ask and discussion. Greatly appreciate it. 👍

1. Following your code, the MAP calculated is 0.2212. I think the discrepancy comes from dd2['prec2'] = dd2['ss'] / 10 where your calculation is based on a division by k, whilst our implementation is based on a division by the number of ground truth items in the testing data. So, when I changed line 588 in this from df_sum_all["map"] = df_sum_all.apply(lambda x: (x.precision / x.actual), axis=1) to df_sum_all["map"] = df_sum_all.apply(lambda x: (x.precision / k), axis=1) it gave me 0.2212.
2. Our implementation referenced the one in PySpark - we try to make sure the two versions of Python evaluator and PySpark evaluator produce matched results. See here for details. In this implementation, it uses the actual number of ground truth items for normalization. If you take a look a the illustration of MAP calculation on the slides it is following the same normalization method.
3. The link to the reference was not valid. We will fix it.
4. The comment that notes the MAP function is referenced from Spark documentation as well. To some extent, the ranking is penalized by the position of the item in the recommendation list, and that is why when it is calculated, the total number of hits is divided by the ranking position of the item. Agree that AP can inflates the overall MAP, and that is why ranking matters in the calculation.

[seancaodo]

@yueguoguo Hello, I just figured it out too. We were posting in parallel 👍. Thanks very much for the fast response.

I think we can close the threat. A minor question left for me to think is why we divide by number of ground truth for each user as number split by test, train sets. If we scale up or down the size of these two sets, the MAP rates will differ I suppose. It's minor I think as we mainly use these for comparisons between algorithms anyway.

Welldone too, thank you guys very much for sharing this great piece of work.