Transformers: Is there a way to evaluate GPT-2 model during fine-tuning process for accuracy and fluency?
❓ Questions & Help
I'm trying to evaluate GPT-2 model during fine tuning process, and I'm able to calculate the loss at each epoch, but do not know how accuracy can be calculated or how to give a score to the model. Would like to get some suggestions as help.
Details
A link to original question on Stack Overflow: https://stackoverflow.com/questions/60483956/how-to-perform-accuracy-testing-on-text-generation-task
D-i-l-r-u-k-s-h-i
>All comments
A common way of evaluating LMs is to measure their Perplexity.
Say you want to finetune GPT2 on your dataset D.
Define train, val and test datasets (maybe something around 75%, 10%, 15%.
Measure the perplexity on train and val after each epoch. Compare train and eval curves for overfitting.
There are a ton of other evaluation measures that might be better for your task - Google will be your best friend :-)
patrickvonplaten
on 19 Mar 2020
Related issues
lemonhu
·
3Comments
chuanmingliu
·
3Comments
fabiocapsouza
·
3Comments
adigoryl
·
3Comments
siddsach
·
3Comments
Most helpful comment
A common way of evaluating LMs is to measure their Perplexity.
Say you want to finetune GPT2 on your dataset D.
Define train, val and test datasets (maybe something around 75%, 10%, 15%.
Measure the perplexity on train and val after each epoch. Compare train and eval curves for overfitting.
There are a ton of other evaluation measures that might be better for your task - Google will be your best friend :-)