Transformers: "Reformer: The Efficient Transformer" looks awesome. I'd love to see it in the library.

[Alan-Lee123]

🌟New model addition

Model description

Efficient Transformer with locality-sensitive hashing and reversible layers
https://openreview.net/forum?id=rkgNKkHtvB

Open Source status

• [. ] the model implementation is available: (give details)
  There is an implementation from google
  https://github.com/google/trax/blob/master/trax/models/research/reformer.py
• [ ] the model weights are available: (give details)
• [. ] who are the authors: (mention them)
  Nikita Kitaev, Lukasz Kaiser, Anselm Levskaya

Additional context

[zbloss]

I have started to refactor the original source code in Pytorch if you'd like to help I'd greatly appreciate it! https://github.com/zbloss/reformer

[lucidrains]

I have a working implementation at https://github.com/lucidrains/reformer-pytorch !

[tmabraham]

Any update on adding this to the library?

[bratao]

They published in their blog about it https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html?m=1

[tmabraham]

Hence my interest in a huggingface implementation :)

[digital-thinking]

Looking forward to see this model in the transformers lib :)

[cfoster0]

Don't think we should rush this one. The reformer paper is pretty tricky to implement in a clean way, plus there aren't any pre-trained models that use it yet. Just one person's opinion, though.

[alexvaca0]

The implementation by @lucidrains seems to work https://github.com/lucidrains/reformer-pytorch ; it'd be cool if it was included in the transformers library. It seems strange to me that no pretrained Reformer has been uploaded since the paper was released, any ideas why? is it possible that it doesn't work in practice as stated by the authors in the paper? Anyone who has trained a Reformer on their own and have tried it to solve a real problem?
Thank you very much in advance

[LuoweiZhou]

Same here, curious to know why. Thank you!

[flozi00]

https://github.com/google/trax/blob/master/trax/models/reformer/machine_translation.ipynb

There should be an pretrained model now.
Would be very happy to see Reformer model in this project.

[MrRace]

+1

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[julien-c]

Closed by @patrickvonplaten

[JulesGM]

has this been done ?