Farm: Albert fine-tuning on SQUAD 2.0

[ckobus]

Hi,

I am trying to fine-tune an Albert large v2 model on SQUAD 2.0 dataset; but so far, I do not succeed to make it converge (the loss doesn't decrease).
I saw others have the same issues (cf https://github.com/huggingface/transformers/issues/2104).
I tried with Albert large v1 and the training went OK even though at the end, I did not get the same results as Bert large.
Did anyone succeed using Albert large v2 (and even larger model versions) on SQUAD 2.0? Which hyperparameters did you use?

Thanks for the help!

[tholor]

Hi @ckobus ,

You are not the only one encountering these issues. With v2 models there seems to be still a few bugs (see also https://github.com/huggingface/transformers/issues/2337). With v1, results are better but still not very stable (judging from a few of our own runs and discussions with other researchers like @BramVanroy). I hope to find some time soon to do more experiments on this. For now, I would recommend using roberta models. We got quite some nice results with a roberta-base:

  "exact": 78.49743114629833,
  "f1": 81.73092721240889

with params

    batch_size = 50
    n_epochs = 3
    evaluate_every = 2000
    base_LM_model = "roberta-base"
    max_seq_len = 384
    learning_rate = 3e-5
    warmup_proportion = 0.2

[BramVanroy]

I can indeed confirm that Albert has rarely been stable in my experiments. Fiddling with the optimizer (e.g. using LAMB, LR 3e-5) helps slightly but still. I would also recommend RoBERTa.

[ckobus]

Thank you @tholor and @BramVanroy for the answers and the advise!!