Transformers: ProphetNet

[aretius]

🌟 New model addition

ProphetNet

Model description

ProphetNet introduces a novel self-supervised objective named future n-gram prediction and the proposed n stream self-attention mechanism. Instead of the optimization of one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations

Open source status

• [X] the model implementation is available: (give details)
  https://github.com/microsoft/ProphetNet
• [X] the model weights are available: (give details)
  Weights for both small and large pre-trained dataset version of models are available
  https://github.com/microsoft/ProphetNet#pre-trained-models
• [X] who are the authors: (mention them, if possible by @gh-username)
  Yu Yan @yuyan2do , Weizhen Qi @weizhen

[qiweizhen]

@aretius Thank you for mentioning ProphetNet. ProphetNet for huggingface is sheduled as you suggested.

[alexgaskell10]

@qiweizhen this sounds great, I would love to give it a go. Any planned date for delivering this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.