Transformers: ETA on TFEncoderDecoderModel and is BERTShare from https://arxiv.org/pdf/1907.12461.pdf planned?

[anicolson]

🚀 Feature request

Is there a plan for BERTShare from https://arxiv.org/pdf/1907.12461.pdf to be an option for the EncoderDecoderModel?

Also, I can see that an TFEncoderDecoderModel is on the 'ToDo' list for the EncoderDecoder Framework. Any chance of an expected time of completion of this would be greatly appreciated.

Motivation

Having an easy to use seq2seq model integrated into hugging face (with TensorFlow) would help my research immensely. Also, models like BERTShare are much more parameter efficient.

Your contribution

I am happy to help in any form. Not sure where help is needed tbh.

[LysandreJik]

I think we can keep this open, this looks like a fun project. Pinging @patrickvonplaten to let him know!

[patrickvonplaten]

The models of https://arxiv.org/pdf/1907.12461.pdf are already added. You can check them out here (they are not called shared, but are shared indeed): https://huggingface.co/models?search=google%2Froberta2roberta

Also, I'll be releasing an in-detail notebook about these models on Monday, so stay tuned :-)

No ETA on TFEncoderDecoder models, but it's definitely on the roadmap :-)

[anicolson]

The models of https://arxiv.org/pdf/1907.12461.pdf are already added. You can check them out here (they are not called shared, but are shared indeed): https://huggingface.co/models?search=google%2Froberta2roberta

Also, I'll be releasing an in-detail notebook about these models on Monday, so stay tuned :-)

No ETA on TFEncoderDecoder models, but it's definitely on the roadmap :-)

Thanks, I am switching from TF to PyTorch :)