Transformers: GPT2 774M weights released!

[moinnadeem]

🚀 Feature

Hi! OpenAI released the 774M weights in GPT2, is it possible to integrate this into pytorch-transformers?

https://twitter.com/OpenAI/status/1163843803884601344

Also, sorry for the obnoxiously quick ask! Thanks for all the great work you do for the community.

Thanks!

[AlexEne]

I did the following:

1. Run download_model.py 774 from here
2. Create a file named config.json with the following contents (Might be correct but I am not super sure):

{
    "vocab_size": 50257,
    "n_ctx": 1024,
    "n_embd": 1280,
    "n_head": 20,
    "n_layer": 36,
    "n_positions": 1024,
    "embd_pdrop":0.1,
    "attn_pdrop": 0.1,
    "resid_pdrop": 0.1,
    "layer_norm_epsilon": 1e-5,
    "initializer_range": 0.02
}

1. Clone this repo

2. Run python .\pytorch-transformers\pytorch_transformers\convert_gpt2_checkpoint_to_pytorch.py --gpt2_checkpoint_path models/774M --pytorch_dump_folder_path ./ --gpt2_config_file config.json

3. Use it with

config = GPT2Config.from_pretrained("config.json")
model = GPT2LMHeadModel.from_pretrained("pytorch_model.bin", config=config)

1. Realize there's no way you can fine-tune this your PC's GPU you need to rent something with more memory.

[thomwolf]

We've added it on master.
You can install from source and use the shortcut name gpt2-large to use it (but beware, it's big!)

[zacharymacleod]

Question: Will the gpt2-large be added to Write With Transformer? I've been eagerly looking forward to that since the moment the 774M was released!

[LysandreJik]

@zacharymacleod Glad you asked! We're definitely planning on adding it in the near future :)

[moinnadeem]

Seems to me as if this has been addressed via #1064 . Closing the feature request now!