Transformers: Adding mbart-large-cc25

[maksym-del]

🌟 New model addition

Multilingual BART model implemented in fairseq introduced by FAIR

Model description

This issue is to request adding mBART model existing as a part of fairseq lib.

Link to the fairseq description of the model
Link to the mBART paper

Multilingually pretrained BART checkpoint.

The model code follows the original BART model code which is already a part of transformers repo. However, it introduces a couple more features like multilingual denoising and translation from pretrained BART.

Open source status

• [x] _the model implementation is available: (give details)_
  Link to the PR adding mBART to the fairseq
  This PR shows the main pieces that were added to the fairseq to make mBART work considering BART which is already existing in the codebase. However, a few additional mBART commits were added afterward.

• [x] _the model weights are available: (give details)_
  Link to the weights

• [x] _who are the authors: (mention them, if possible by @gh-username)_
  Facebook AI Research (@MultiPath)

[sshleifer]

This is a Work in progress but still a few weeks out :)

[ratthachat]

Hi @sshleifer , additional (perhaps bug, or document bug) related to this issue:

This model page suggests that we can load mBart-cc25 :
https://huggingface.co/facebook/mbart-large-cc25

However, using the instructed command with the newest HuggingFace 2.8.0 :
model = AutoModel.from_pretrained("facebook/mbart-large-cc25")

is failed :

AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-4-c034f52e2196> in <module>
     11 '''
     12 
---> 13 model =  AutoModel.from_pretrained("facebook/mbart-large-cc25")
     14 tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")

/opt/conda/lib/python3.7/site-packages/transformers/modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    421         for config_class, model_class in MODEL_MAPPING.items():
    422             if isinstance(config, config_class):
--> 423                 return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    424         raise ValueError(
    425             "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"

/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    625             except Exception:
    626                 raise OSError(
--> 627                     "Unable to load weights from pytorch checkpoint file. "
    628                     "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "
    629                 )

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 

[sshleifer]

Yes, the docs are wrong/aspirational at the moment. Will fix today!

[sshleifer]

Fixed the docs. That model is currently not supported, but it's on my roadmap to add it in the coming weeks.

[pchankh]

sshleifer, wonder if the mbart-large-cc25 have been added? We are looking to use mbart for a multilingual text classification problem. Thanks for the great work.
Patrick

[sshleifer]

Hopefully this weekend!

[sshleifer]

What languages are you trying to support?
We have 1,000+ models in the MarianMTModel family, 11 of which are multi-lingual.

[sshleifer]

We are blocked for the moment on https://github.com/pytorch/fairseq/issues/2258,
if anybody has any ideas how to fix that it would be much appreciated!