Transformers: config.attention_head_size for structured pruning out-of-box

[dsindex]

🚀 Feature request

Motivation

for structured pruning like fastformers, https://github.com/microsoft/fastformers#pruning-models ,

we should modify the source code of transformers for `attention_head_size.

for example,

1. configuration_bert.py
   https://github.com/microsoft/fastformers/blob/main/src/transformers/configuration_bert.py#L128

2. modeling_bert.py
   https://github.com/microsoft/fastformers/blob/main/src/transformers/modeling_bert.py#L192
   https://github.com/microsoft/fastformers/blob/main/src/transformers/modeling_bert.py#L263

is it possible to set attention_head_size from outside(config.json) ?

Your contribution

[ykim362]

@dsindex FYI, I am working on creating a PR including this feature for the effort of #8083 .

[dsindex]

@ykim362 great! closing issue here.