Parlai: `BiEncoderRankerAgent` parameters

[bkj]

The convai2 future work slides suggest that the BERT bi_encoder_ranker should be able to get to hits@1 of about 0.79 on the PersonaChat dataset.

I looked back through the commits and found an old testing.sh script, which suggests these parameters:

# Train the bi-encoder
# 2 GPU machines: batchsize 32 num-epoch 0.1 warmup_updates 200. 25% accuracy in 5min (+ eval)
# 8 GPU machines: batchsize 128 num-epoch 3 warmup_updates 4000. 79% accuracy in 40mn
python -u parlai/scripts/train_model.py -pyt convai2 \
  -m parlai.agents.bert_ranker.biencoder_ranker:BiEncoderRankerAgent \
  --batchsize 32 --dict-file ./dictionary --model-file ./my_biencoder  \
  --pretrained-bert-path $MODELFILE --bert-vocabulary-path $VOCABFILE \
  --eval-batchsize 8 --learningrate 5e-5  --log_every_n_secs 10 \
  --shuffle true --type-optimization all_encoder_layers \
  --data-parallel true --history-size 5 --label-truncate 300 \
  --text-truncate 300 --num-epochs 0.1 \
  --lr-scheduler fixed --lr-scheduler-patience 1 --lr-scheduler-decay 0.35 \
  -veps 1.0 -vme 2000  --warmup_updates 200
# Should yield
# test:{'exs': 7801, 'accuracy': 0.25, 'f1': 0.3272, 'hits@1': 0.25,
#       'hits@5': 0.634, 'hits@10': 0.845, 'hits@100': 1.0, 'bleu': 0.2502,
#       'lr': 5e-05, 'num_updates': 411, 'examples': 7801, 'loss': 2388.0,
#       'mean_loss': 0.3062, 'mean_rank': 5.269}

but I'm having a little trouble using those parameters to reproduce the 0.79 number.

I ran the 2 GPU settings and got significantly better results:

valid:{'exs': 7801, 'accuracy': 0.4716, 'f1': 0.5274, 'hits@1': 0.472, 'hits@5': 0.818, 'hits@10': 0.936, 'hits@100': 1.0, 'bleu': 0.4719, 'lr': 5e-05, 'num_updates': 411, 'examples': 7801, 'loss': 1791.0, 'mean_loss': 0.2296, 'mean_rank': 3.27, 'train_accuracy': 0}
test:{'exs': 7801, 'accuracy': 0.4716, 'f1': 0.5274, 'hits@1': 0.472, 'hits@5': 0.818, 'hits@10': 0.936, 'hits@100': 1.0, 'bleu': 0.4719, 'lr': 5e-05, 'num_updates': 411, 'examples': 7801, 'loss': 1791.0, 'mean_loss': 0.2296, 'mean_rank': 3.27, 'train_accuracy': 0}

And then ran the 8 GPU settings (though I only used 6 GPUs), and got significantly worse results:

valid:{'exs': 7801, 'accuracy': 0.6657, 'f1': 0.7024, 'hits@1': 0.666, 'hits@5': 0.921, 'hits@10': 0.979, 'hits@100': 1.0, 'bleu': 0.6658, 'lr': 5e-05, 'num_updates': 2054, 'examples': 7801, 'loss': 1076.0, 'mean_loss': 0.1379, 'mean_rank': 2.086, 'train_accuracy': 0}
test:{'exs': 7801, 'accuracy': 0.6657, 'f1': 0.7024, 'hits@1': 0.666, 'hits@5': 0.921, 'hits@10': 0.979, 'hits@100': 1.0, 'bleu': 0.6658, 'lr': 5e-05, 'num_updates': 2054, 'examples': 7801, 'loss': 1076.0, 'mean_loss': 0.1379, 'mean_rank': 2.086, 'train_accuracy': 0}

Any ideas what might be going on and how to improve the results? I understand it's risky to go digging through git history 😀 but AFAICT all of the parameter guidance has been removed..

Thanks!
~ Ben

Note: I had opened a similar issue a while ago, but found the old testing.sh recently.

[stephenroller]

https://arxiv.org/pdf/1905.01969.pdf

Section 5.2 names some of the most important hyperparameters, but the authors should respond soonish.

[stephenroller]

Also I notice the script you reference says —num-epochs 0.1. You should remove that and let it run to convergence.

[bkj]

OK thanks! I hadn't seen that paper, so I'll look there.

For my 6 GPU experiment, I changed the parameters according to

# 8 GPU machines: batchsize 128 num-epoch 3 warmup_updates 4000. 79% accuracy in 40mn

so --num-epochs = 3 -- but it's possible 3 wasn't enough to converge.

[klshuster]

Our best results with the Bi-encoder ranker had --batchsize 512, --warmup-updates 100, and --history-size 20, though the rest of those parameters should be good (you could change num_epochs to make it higher, but it's not entirely necessary). Additionally, we skipped the linear layer here to obtain our highest result on the validation set.

[bkj]

OK cool. Assembling the old testing.sh, the paper and your comments, this is my best guess at the parameters:

python -u parlai/scripts/train_model.py 
    --task convai2 
    --model bert_ranker/bi_encoder_ranker 
    --dict-file ./outpath/dictionary 
    --model-file ./outpath/my_biencoder  
    --log_every_n_secs 10 
    --type-optimization all_encoder_layers 
    --shuffle true 
    --data-parallel true 
    --batchsize 512                 # Section 5.2
    --eval-batchsize 8              # testing.sh
    --learningrate 5e-5             # Section 5.2
    --history-size 20               # this issue
    --label-truncate 72             # Section 5.1
    --text-truncate 360             # Section 5.1
    --num-epochs 3                  # testing.sh
    --lr-scheduler reduceonplateau  # Section 5.2
    --lr-scheduler-patience 1       # Section 5.2 (Is this right)
    --lr-scheduler-decay 0.4        # Section 5.2
    --validation-every-n-epochs 0.5 # Section 5.2
    --warmup_updates 100            # Section 5.2
    --fp16 true                     # Section 5.2

I'll give this a shot on an 8xV100 box, and report back here w/ the results.

Are there plans to release the code/parameters for the polyencoder experiments?

Thanks!

[stephenroller]

I'd suggest leaving off --num-epochs altogether, so let it just run.

Also I think you need this, which dramatically changes the semantics:

-cands batch -ecands inline

[bkj]

I think those are the default values, per
https://github.com/facebookresearch/ParlAI/blob/master/parlai/agents/bert_ranker/helpers.py#L65

[bkj]

I ran the above command on a p3.16xlarge for 6 epochs (took ~1.5 hours). I reduced the batch size from 512 to 256, because I was getting lots of warnings.

The validation results are

$ ~/software/parlai/experiments/v0$ cat my_model.trainstats | jq -rc '.valid_reports | .[]'
{"exs":7801,"accuracy":0.7047,"f1":0.7366,"hits@1":0.705,"hits@5":0.945,"hits@10":0.987,"hits@100":1,"bleu":0.7044,"lr":5e-05,"num_updates":257,"examples":7801,"loss":32.28,"mean_loss":0.004138,"mean_rank":1.873,"mrr":0.8063,"train_time":537.2963297367096}
{"exs":7801,"accuracy":0.749,"f1":0.7767,"hits@1":0.749,"hits@5":0.956,"hits@10":0.989,"hits@100":1,"bleu":0.7491,"lr":5e-05,"num_updates":514,"examples":7801,"loss":27.85,"mean_loss":0.00357,"mean_rank":1.704,"mrr":0.8385,"train_time":1025.2623586654663}
{"exs":7801,"accuracy":0.7608,"f1":0.7861,"hits@1":0.761,"hits@5":0.96,"hits@10":0.991,"hits@100":1,"bleu":0.7608,"lr":5e-05,"num_updates":771,"examples":7801,"loss":26.52,"mean_loss":0.003399,"mean_rank":1.645,"mrr":0.8472,"train_time":1529.0462341308594}
{"exs":7801,"accuracy":0.7734,"f1":0.7978,"hits@1":0.773,"hits@5":0.965,"hits@10":0.991,"hits@100":1,"bleu":0.7732,"lr":5e-05,"num_updates":1028,"examples":7801,"loss":25.73,"mean_loss":0.003298,"mean_rank":1.608,"mrr":0.8552,"train_time":2034.2776310443878}
{"exs":7801,"accuracy":0.7762,"f1":0.8005,"hits@1":0.776,"hits@5":0.965,"hits@10":0.992,"hits@100":1,"bleu":0.7763,"lr":5e-05,"num_updates":1285,"examples":7801,"loss":25.3,"mean_loss":0.003243,"mean_rank":1.595,"mrr":0.8579,"train_time":2539.547839164734}
{"exs":7801,"accuracy":0.7737,"f1":0.7985,"hits@1":0.774,"hits@5":0.965,"hits@10":0.991,"hits@100":1,"bleu":0.7738,"lr":5e-05,"num_updates":1542,"examples":7801,"loss":26.08,"mean_loss":0.003344,"mean_rank":1.607,"mrr":0.8561,"train_time":3045.046471118927}
{"exs":7801,"accuracy":0.7717,"f1":0.7962,"hits@1":0.772,"hits@5":0.964,"hits@10":0.992,"hits@100":1,"bleu":0.7717,"lr":5e-05,"num_updates":1799,"examples":7801,"loss":26.97,"mean_loss":0.003457,"mean_rank":1.615,"mrr":0.8543,"train_time":3529.8368356227875}
{"exs":7801,"accuracy":0.785,"f1":0.8082,"hits@1":0.785,"hits@5":0.97,"hits@10":0.993,"hits@100":1,"bleu":0.785,"lr":2e-05,"num_updates":2056,"examples":7801,"loss":25.55,"mean_loss":0.003275,"mean_rank":1.541,"mrr":0.8649,"train_time":4015.5114421844482}
{"exs":7801,"accuracy":0.7855,"f1":0.8088,"hits@1":0.786,"hits@5":0.969,"hits@10":0.992,"hits@100":1,"bleu":0.7855,"lr":2e-05,"num_updates":2313,"examples":7801,"loss":26.32,"mean_loss":0.003374,"mean_rank":1.556,"mrr":0.8645,"train_time":4519.821675300598}
{"exs":7801,"accuracy":0.7882,"f1":0.8109,"hits@1":0.788,"hits@5":0.971,"hits@10":0.993,"hits@100":1,"bleu":0.7882,"lr":8e-06,"num_updates":2570,"examples":7801,"loss":26.17,"mean_loss":0.003354,"mean_rank":1.547,"mrr":0.8659,"train_time":5025.581673622131}
{"exs":7801,"accuracy":0.7903,"f1":0.8127,"hits@1":0.79,"hits@5":0.971,"hits@10":0.993,"hits@100":1,"bleu":0.7903,"lr":8e-06,"num_updates":2827,"examples":7801,"loss":26.37,"mean_loss":0.00338,"mean_rank":1.54,"mrr":0.8678,"train_time":5531.12650847435}

so we top out around 0.79.

That's consistent w/ the results in the testing.sh file, but ~4% lower than the batchsize=256 results reported in the paper.

Next I'll try skipping the linear layer that @klshuster mentioned above. Beyond that, any other thoughts on where the extra 4% in the paper might come from?

[stephenroller]

I recall someone saying skipping that linear layer was pretty important, so that sounds like a good bet.

[bkj]

Removed the linear layer and set --lr-scheduler-patience 0, and get this performance after 5.5 epochs / 1.5 hours on the p3.16xlarge:

{"exs":7801,"accuracy":0.7611,"f1":0.7859,"hits@1":0.761,"hits@5":0.958,"hits@10":0.989,"hits@100":1,"bleu":0.7611,"lr":5e-05,"num_updates":257,"examples":7801,"loss":26.3,"mean_loss":0.003371,"mean_rank":1.676,"mrr":0.8456,"train_time":538.8439116477966}
{"exs":7801,"accuracy":0.795,"f1":0.8175,"hits@1":0.795,"hits@5":0.971,"hits@10":0.993,"hits@100":1,"bleu":0.795,"lr":5e-05,"num_updates":514,"examples":7801,"loss":22.53,"mean_loss":0.002888,"mean_rank":1.535,"mrr":0.8704,"train_time":1026.8353519439697}
{"exs":7801,"accuracy":0.7993,"f1":0.8212,"hits@1":0.799,"hits@5":0.968,"hits@10":0.992,"hits@100":1,"bleu":0.7991,"lr":5e-05,"num_updates":771,"examples":7801,"loss":21.83,"mean_loss":0.002798,"mean_rank":1.526,"mrr":0.8727,"train_time":1531.4706733226776}
{"exs":7801,"accuracy":0.814,"f1":0.8339,"hits@1":0.814,"hits@5":0.971,"hits@10":0.993,"hits@100":1,"bleu":0.8138,"lr":5e-05,"num_updates":1028,"examples":7801,"loss":21.14,"mean_loss":0.00271,"mean_rank":1.493,"mrr":0.8816,"train_time":2035.4154632091522}
{"exs":7801,"accuracy":0.8163,"f1":0.8365,"hits@1":0.816,"hits@5":0.972,"hits@10":0.994,"hits@100":1,"bleu":0.8161,"lr":5e-05,"num_updates":1285,"examples":7801,"loss":21.03,"mean_loss":0.002695,"mean_rank":1.472,"mrr":0.884,"train_time":2538.34698843956}
{"exs":7801,"accuracy":0.8125,"f1":0.8331,"hits@1":0.812,"hits@5":0.972,"hits@10":0.993,"hits@100":1,"bleu":0.8123,"lr":5e-05,"num_updates":1542,"examples":7801,"loss":22.4,"mean_loss":0.002871,"mean_rank":1.489,"mrr":0.8809,"train_time":3041.653540611267}
{"exs":7801,"accuracy":0.8223,"f1":0.842,"hits@1":0.822,"hits@5":0.975,"hits@10":0.993,"hits@100":1,"bleu":0.8223,"lr":2e-05,"num_updates":1799,"examples":7801,"loss":21.53,"mean_loss":0.00276,"mean_rank":1.462,"mrr":0.8874,"train_time":3524.147078037262}
{"exs":7801,"accuracy":0.8252,"f1":0.8447,"hits@1":0.825,"hits@5":0.977,"hits@10":0.995,"hits@100":1,"bleu":0.825,"lr":8e-06,"num_updates":2056,"examples":7801,"loss":20.9,"mean_loss":0.002679,"mean_rank":1.434,"mrr":0.8903,"train_time":4024.2351970672607}
{"exs":7801,"accuracy":0.8255,"f1":0.8449,"hits@1":0.826,"hits@5":0.976,"hits@10":0.995,"hits@100":1,"bleu":0.8254,"lr":8e-06,"num_updates":2313,"examples":7801,"loss":21.33,"mean_loss":0.002735,"mean_rank":1.435,"mrr":0.8902,"train_time":4528.047632694244}
{"exs":7801,"accuracy":0.828,"f1":0.847,"hits@1":0.828,"hits@5":0.977,"hits@10":0.994,"hits@100":1,"bleu":0.8278,"lr":3.2e-06,"num_updates":2570,"examples":7801,"loss":21.3,"mean_loss":0.00273,"mean_rank":1.43,"mrr":0.8919,"train_time":5033.247322559357}
{"exs":7801,"accuracy":0.8289,"f1":0.8479,"hits@1":0.829,"hits@5":0.977,"hits@10":0.995,"hits@100":1,"bleu":0.8287,"lr":1.28e-06,"num_updates":2827,"examples":7801,"loss":21.37,"mean_loss":0.00274,"mean_rank":1.429,"mrr":0.8923,"train_time":5536.921379566193}

So maximum accuracy of 0.8289, which is within the margin of error of 0.830 reported in the paper for batchsize=256. Thanks for everyone's help!