Transformers: AutoModel fails to load FlauBERT with `output_hidden_states`

[LoicGrobol]

MWE:

import transformers

model = transformers.AutoModel.from_pretrained("flaubert-base-cased", output_hidden_states=True)

Tested on rev 5a6b138 fails with

Traceback (most recent call last):
  File "mwe.py", line 3, in <module>
    model = transformers.AutoModel.from_pretrained("flaubert-base-cased", output_hidden_states=True)
  File "<redacted>/transformers/modeling_auto.py", line 377, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "<redacted>/transformers/modeling_utils.py", line 463, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'output_hidden_states'

This works when loading directly from transformers.FlaubertModel.

[LysandreJik]

Hi! the output_hidden_states should be specified in the configuration when loading from AutoModel classes. Doing the following is necessary to instantiate a class with hidden states:

import transformers

config = transformers.AutoConfig.from_pretrained("flaubert-base-cased", output_hidden_states=True)
model = transformers.AutoModel.from_pretrained("flaubert-base-cased", config=config)

However, your issue showed me there was a bug with the loading of FlauBERT models with AutoModels, which I patched in https://github.com/huggingface/transformers/commit/ff6f1492e8296f511682fd56fcf62be0854723a2.
Please install from source to have the fix: pip install git+https://github.com/huggingface/transformers, I'll push a pypi patch for this soon.

[Evpok]

Oh, okay, thanks. From what I understood of AutoModel doc, I thought all **kwargs in AutoModel.from_pretrained were passed to the config.

[LysandreJik]

Indeed, the documentation seems misleading in that regard. I'm updating it.

[Evpok]

AutoTokenizer seems to have the same problem as the one you fixed in AutoModel

transformers.AutoTokenizer.from_pretrained("flaubert-base-uncased")

results in

OSError: Model name 'flaubert-base-uncased' was not found in tokenizers model name list (xlm-mlm-en-2048, xlm-mlm-ende-1024, xlm-mlm-enfr-1024, xlm-mlm-enro-1024, xlm-mlm-tlm-xnli15-1024, xlm-mlm-xnli15-1024, xlm-clm-enfr-1024, xlm-clm-ende-1024, xlm-mlm-17-1280, xlm-mlm-100-1280)

[LysandreJik]

Indeed it does, thanks @Evpok !

[LysandreJik]

Should have been patched and tested with 1e82cd8.

[Evpok]

Thanks for the quick response ♥

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[KamyarGhajar]

I think this change should be bolder in documentations somehow as I had this problem too.