Recommenders: [ASK] How to create test submission for MIND competition

[ngcferreira]

Description

I've been trying to use the nrms_MIND notebook to create test submission by doing the following:

test_news_file = os.path.join(data_path, 'test', r'news.tsv')
test_behaviors_file = os.path.join(data_path, 'test', r'behaviors.tsv')
group_impr_indexes, group_labels, group_preds = model.run_fast_eval(test_news_file, test_behaviors_file)

with open(os.path.join(data_path, 'prediction.txt'), 'w') as f:
    for impr_index, preds in tqdm(zip(group_impr_indexes, group_preds)):
        impr_index += 1
        pred_rank = (np.argsort(np.argsort(preds)[::-1]) + 1).tolist()
        pred_rank = '[' + ','.join([str(i) for i in pred_rank]) + ']'
        f.write(' '.join([str(impr_index), pred_rank])+ '\n')

f = zipfile.ZipFile(os.path.join(data_path, 'prediction.zip'), 'w', zipfile.ZIP_DEFLATED)
f.write(os.path.join(data_path, 'prediction.txt'), arcname='prediction.txt')
f.close()

The problem is that when I submit those to the competition, I get a weird error which doesn't tell much on how I can solve this problem. The error is the following:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. /tmp/codalab/tmpDabQNd/run/program/evaluate.py:17: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 61 but corresponding boolean dimension is 7 false_score = score[label==0] /tmp/codalab/tmpDabQNd/run/program/evaluate.py:18: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 61 but corresponding boolean dimension is 7 positive_score = score[label==1] Traceback (most recent call last): File "/tmp/codalab/tmpDabQNd/run/program/evaluate.py", line 125, in auc, mrr, ndcg, ndcg10,mrrs = scoring(truth_file, submission_answer_file) File "/tmp/codalab/tmpDabQNd/run/program/evaluate.py", line 91, in scoring mrr = mrr_score(y_true,y_score) File "/tmp/codalab/tmpDabQNd/run/program/evaluate.py", line 36, in mrr_score y_true = np.take(y_true, order) File "/opt/conda/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 124, in take return take(indices, axis, out, mode) IndexError: index 19 is out of bounds for size 7

What am I doing wrong and how can I fix that?

Thank you,
Nuno

[miguelgfierro]

@leavingseason FYI

[ngcferreira]

@Leavingseason @miguelgfierro Any help would be really appreciated. Would like to submit my predictions to the competition but it always fails with the above error :(

[wuch15]

Hi @ngcferreira, I have downloaded your submission. I find that its size is very small (only 300+kb), while a valid submission should be much larger. Your prediction file only contains 7,538 lines, while the entire behavior.tsv file contains 2,370,727 lines. Thus, you may need to check your code to see whether the prediction for each line in the behavior.tsv file is included. Thanks.

[ngcferreira]

@wuch15 Thank you for your help. Unfortunately I went on vacations before I've seen your answer so I couldn't submit my results anymore.