Transformers: Finetuning GPT-2

[sadhikamalladi]

It looks like there used to be a script run_lm_finetuning.py that has been replaced by run_language_modeling.py. It's unclear to me how to use this script to run finetuning. I want to finetune GPT-2 on a variety of downstream tasks, and would love some help!

[Genius1237]

You can use run_language_modeling.py for this purpose. Just set --model_type to gpt2, set --model_name_or_path to the gpt2 model checkpoint you want (gpt2) and set --train_data_file to your dataset and you should be ready to go.

[sadhikamalladi]

Thanks! The issue is that I want to use the pre-trained version of GPT-2. I remember run_lm_finetuning had a few lines of code where it would download and load that pre-trained model.

[Genius1237]

You can specify gpt2 in --model_name_or_path. That corresponds to one of the pre-trained checkpoints that it'll download and use. The other possible pre-trained models that you can specify there are gpt2-medium,gpt2-large,gpt2-xl and distilgpt2.

[sadhikamalladi]

If specifying gpt2 there downloads the checkpoints then how do you train from scratch? I've been specifying that parameter and it seems like it is training from scratch (starting perplexity ~1000).

[Genius1237]

I believe that if you want to train from scratch, you'll have to point that to a folder with a config file (with the parameters of the model) and no pytorch_model.bin checkpoint file in that folder.

[Mse709]

@Genius1237 Hi there,
I am finetuning the 124M model based on my dataset (almost 2mb) and I am using Colab by Max Wolf. I am wondering if there is a way that I can generate texts based on my trained model + internet context (not only metadata). I wanna generate some notes regarding current issues (such as COVID-19) based on my trained model.
Could you help me with that, please? thanks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.