Transformers: Fine tuning GPT-2 for LM objective function

[bakszero]

I'm looking for finetuning GPT-2 parameters for a custom piece of text, so that the weights are tuned for this piece of text, building from the initial model.

The script here does it for the original tensorflow implementation: https://github.com/nshepperd/gpt-2 , could you please give me suggestions on how to do this finetuning from the Pytorch version and subsequently use it for text generation? It'd be of great help, thanks!

[phtully]

Seems like this is now possible with last week's merged PR, but I'm curious to see what the core devs say about this as well (btw, keep up the great work!)

[fabrahman]

I have the same question :)
I have tried the codes for BERT finetuning which is in lm-finetuning folder but looking for the same script for gpt-2.

Thanks

[thomwolf]

Yes fine-tuning GPT-2 is fixed with #597 indeed.
I'll see if I can add an example but basically changing gpt to gpt-2 in the gpt example should be pretty much fine.

[fabrahman]

@thomwolf Thanks for the great work.
just wondering in order to do unsupervised LM fine-tuning (not classification) on a new dataset, should we just modify run_openai_gpt.py or is there an existing script for that?

[thomwolf]

No existing script for that but you can start from run_openai indeed and use just the OpenAIGPTLMHeadModel.
If you want to supply another example, happy to welcome a PR

[navneet-nmk]

I am still confused as to how to use the run_openai_gpt.py to finetune gpt2 model. A short example would be helpful

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.