Datasets: 🐛 [Metrics] ROUGE is non-deterministic

[astariul]

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to this Colab notebook for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?

[thomwolf]

Hi, can you give a full self-contained example to reproduce this behavior?

[lhoestq]

Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

[lhoestq]

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to this Colab notebook for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

Why ROUGE is not deterministic ?

This is because of rouge's BootstrapAggregator that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using

score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_agregator=False)

Or you can set numpy's random seed if you still want to use the aggregator.

[lhoestq]

Maybe we can set all the random seeds of numpy/torch etc. while running metric.compute ?

[thomwolf]

We should probably indeed!

[lhoestq]

Now if you re-run the notebook, the two printed results are the same @colanim

['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']

However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:

rouge = nlp.load_metric('rouge', seed=42)