Fairseq: Reproducing RoBERTa STS-B results

[mitchellg]

Thanks for your great work! Quick request: could you please add how to replicate RoBERTa STS-B results? There are good instructions for fine-tuning the model but I didn't see any for running the prediction task itself. Thanks!

[ngoyal2707]

You can use something like following code-snippet for inference on STS-B.

from fairseq.models.roberta import RobertaModel
from scipy.stats import pearsonr

roberta = RobertaModel.from_pretrained(
    'checkpoints/',
    checkpoint_file='checkpoint_best.pt',
    data_name_or_path='STS-B-bin'
)

roberta.cuda()
roberta.eval()
gold, pred = [], []
with open('glue_data/STS-B/dev.tsv') as fin:
    fin.readline()
    for index, line in enumerate(fin):
        tokens = line.strip().split('\t')
        sent1, sent2, target = tokens[7], tokens[8], float(tokens[9])
        tokens = roberta.encode(sent1, sent2)
        features = roberta.extract_features(tokens)
        predictions = 5.0 * roberta.model.classification_heads['sentence_classification_head'](features)
        gold.append(target)
        pred.append(predictions.item())

print('| Pearson: ', pearsonr(gold, pred))

I will add above to readme, thanks for pointing it out.

[mitchellg]

Thanks so much! Unfortunately I wasn't able to reproduce your STS-B score. The value of predictions.item() was always ~2.84, changing only very slightly for each example. I followed your finetuning instructions for STS-B and didn't change the above code for inference. Any thoughts on what might be going on?

[ngoyal2707]

hmm can you please share your training logs? specially what loss you are getting?
Also can you please make sure you see "loaded checkpoint ..." file in the training logs?

[mitchellg]

You're right, I wasn't loading the checkpoint. Fixed that and now I'm able to reproduce STS-B. Thanks again!

[absin1]

Where do we get the checkpoint file?

[YJYJLee]

predictions = 5.0 * roberta.model.classification_heads'sentence_classification_head'

Why do we need to multiply 5.0 to the output of the model?

[sandeep-adireddi]

why we need data_name_or_path as STS-bin