Transformers: Finetune GPT2

[rabeehk]

Hi
According to pytorch-transformers/docs/source/index.rst
There was a run_gpt2.py example which also shows how to finetune GPT2 on the training data.
I was wondernig if you could add this example back, and proving sample script to finetune GPT2.
thanks.
Best regards,
Rabeeh

[LysandreJik]

Hi Rabeeh,

We are currently working on an updated example on fine-tuning generative models, especially GPT-2. The example should be up later this week, keep an eye out!

[letsgoduke]

Any update on when this example will be available? Thanks!

[aiscientist]

Hope this issue won't be closed until the example is done.

[LysandreJik]

The script is being worked on over at https://github.com/huggingface/pytorch-transformers/pull/987 (see relevant file here). It works for GPT/GPT-2 but it isn't ready for BERT/RoBERTa so we're not releasing it yet.

It shows how to fine-tune GPT-2 using causal language modeling on WikiText-2.

[ruidongtd]

Any update on when this example will be available? Thanks!
The link of "see relevant file here" is 404

[thomwolf]

Oh yes, the script is out.

It was renamed run_lm_fintuning.py you can find it in the examples folder: https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_lm_finetuning.py

You can use it to fintune GPT, GPT-2, BERT or RoBERTa on your dataset.

Here is an example on how to run it: https://huggingface.co/pytorch-transformers/examples.html#causal-lm-fine-tuning-on-gpt-gpt-2-masked-lm-fine-tuning-on-bert-roberta

[Henry-E]

Silly question but how do you know which gpt-2 model is being trained? Does it default to the largest one available. I couldn't find any indication of which size model is being used in the fine tuning script.

[thomwolf]

Hi Henry,
Default to the small one.
You can select the size with the model_name_or_path argument. Just put in
the argument the relevant shortcut name for the model as listed here.

On Wed, 6 Nov 2019 at 12:35, Henry-E [e-mail] wrote:

Silly question but how do you know which gpt-2 model is being trained?
Does it default to the largest one available. I could find any indication
of which size model is being used in the fine tuning script.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/huggingface/transformers/issues/969?email_source=notifications&email_token=ABYDIHNYJ3YQTDE6P6HPCOTQSKTYRA5CNFSM4IJMWQW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDGHRYA#issuecomment-550271200,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYDIHM7FQGTLI5UPLWJHSTQSKTYRANCNFSM4IJMWQWQ
.

[Henry-E]

Ah got it, thanks!

[texturejc]

run_lm_fintuning.py is no longer available in the examples folder when you clone the transformers repo. Is there a reason for this? It was available a couple of months ago.

[julien-c]

It’s named run_language_modeling.py now

[texturejc]

Great, thanks!

[iliemihai]

This may sound silly also, but will run_lm_fintuning.py be able to finetune microsoft/DialoGPT model on a custom dataset? Thank you

[LysandreJik]

Yes, but it's named run_language_modeling.py now.