Transformers: How to train from scratch

[anandhperumal]

I would like to train the model from scratch.
How can I drop the trained weight? using the same architecture for Gpt2

[LysandreJik]

If you want to randomly initialize a model simply initialize it via its constructor rather than from the from_pretrained method:

from transformers import GPT2Config, GPT2Model

config = GPT2Config()  # define your configuration here
model = GPT2Model(config)  # Initialize your model from your config

[anandhperumal]

@LysandreJik Thanks for the input.
I did something like this

    config = GPT2Config(vocab_size)
    model = GPT2Model(config)

Apart from vocab size, I'm keeping everything else to default value how do I make sure that it doesn't have any pre-trained value?

[rlouf]

The values are only loaded if your instantiate the model by calling ˋGPT2Model.from_pretrained`, so you’re fine 🙂

[anandhperumal]

@rlouf Thanks