Transformers: Do Hugging Face GPT-2 Transformer Models Automatically Does the Absolute Position Embedding for Users?

[h56cho]

Hello,

According to Hugging Face GPT2DoubleHeadsModel documentation (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel)

"Indices of input sequence tokens in the vocabulary. 
GPT-2 is a model with absolute position embeddings"

So does this mean that, when we implement any Hugging Face GPT-2 Models (GPT2DoubleHeadsModel,GPT2LMHeadsModel, etc.) via the model( ) statement, the 'absolute position embedding' is _automatically_ done for the user, so that the user actually does not need to specify anything in the model( ) statement to ensure the absolute position embedding?

If the answer is 'yes', then why do we have an option of specifying position_ids in the model( ) statement?

Thank you,

[LysandreJik]

Indeed, as you can see from the source code here, when no position ids are passed, they are created as absolute position embeddings.

You could have trained a model with a GPT-2 architecture that was using an other type of position embeddings, in which case passing your specific embeddings would be necessary. I'm sure several other use-cases would make sure of specific position embeddings.

[h56cho]

Ooohh, ok,

so to clarify, absolute position embedding _is automatically done_ by the model( ) statement, but if we want to use our custom position embedding (i.e. other than the absolute position embedding), we can use the position_ids option inside the model( ) statement......is what I said above correct?

Thank you,

[LysandreJik]

Yes, that is correct!

[h56cho]

Thank you :) !