Transformers: BART for Pre-Training
❓ Questions & Help
How can I run BART pre-training?
I have data to pre-training(Masked LM)
swashiro
All 8 comments
This should help: https://github.com/huggingface/transformers/issues/5096#issuecomment-645860271
patrickvonplaten
on 26 Aug 2020
@sshleifer - think this is the 3rd issue about Bart pre-training -> maybe it would be a good idea to release a small notebook at some point.
patrickvonplaten
on 26 Aug 2020
@patil-suraj you took a stab at this at some point? this may have been optimistic :(
sshleifer
on 26 Aug 2020
Yes, I was trying to port fairseq dataset here, same for t5, I'll try to focus more on it when I'm done with current PRs, should strat with a notebook as Patrick said, then try to include it in examples/
patil-suraj
on 26 Aug 2020
@patrickvonplaten Does that mean I can train with Masked-input, input(label) and Decoder-input?
swashiro
on 27 Aug 2020
yes, this should be possible
patrickvonplaten
on 1 Sep 2020
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 1 Nov 2020
@patil-suraj any news on the pretraining script for Bart?
dhruvramani
on 6 Dec 2020
Related issues
lcswillems
·
3Comments
HansBambel
·
3Comments
fabiocapsouza
·
3Comments
zhezhaoa
·
3Comments
fyubang
·
3Comments
Most helpful comment
Yes, I was trying to port fairseq dataset here, same for t5, I'll try to focus more on it when I'm done with current PRs, should strat with a notebook as Patrick said, then try to include it in examples/