Fairseq: Implement Reducing Transformer Depth on Demand With Structred Dropout

[gvskalyan]

Please consider implementing https://arxiv.org/pdf/1909.11556.pdf with a README.md in the examples to reproduce the results. @huihuifan

[huihuifan]

We are actively working on this and will release code and commands for reproduction very soon, then open source models soon afterwards. Thanks for your interest!

[gvskalyan]

https://github.com/pytorch/fairseq/commit/dabbef467692ef4ffb7de8a01235876bd7320a93 solves this

[gvskalyan]

Acoording to this ran --model-overrides "{'decoder_layers_to_keep':'0,2,4,6'}" for a transformer-base model while evaluating, but ran into below error.
RuntimeError: Error(s) in loading state_dict for TransformerModel:
Missing key(s) in state_dict:

Also tried resuming training from layer-dropped out model, still the same error occurs @huihuifan .
Thanks.

[huihuifan]

hi @gvskalyan, does the error message say which keys in the state dict are missing?

[gvskalyan]

Sorry, it is evident that I have trained the transformer base, but am overriding only decoder layers but not encoder layers. Also it works only for same layers on both sides. Thanks.

[huihuifan]

thanks for raising this error, I will look into it on current master branch and get back to you.