Transformers: Add optimal model size and stopping time feature

[TevenLeScao]

🚀 Feature request

The calculator blog post presented an automated way to find scaling laws with model size and compute budget on language modeling tasks. Adding it to the library would help save on training costs by picking an optimal model size and training time.

Motivation

Estimating how big of a model to use and how long to train for is more of an art than a science. An automated tool to perform that task would allow researchers and practitioners to concentrate on the the high-level parts of their projects as opposed to parameter tweaking.

Your contribution

I can submit a PR with my existing work, probably integrating it within Trainer and/or knocknock.

[lopuhin]

Great stuff, thank you! The energy estimates look 1000 worse than reality though, V100 running for 12 h should not consume 5432 kWh I think, else we'd be all dead. 5.4 kWh looks more reasonable.

[TevenLeScao]

Great stuff, thank you! The energy estimates look 1000 worse than reality though, V100 running for 12 h should not consume 5432 kWh I think, else we'd be all dead. 5.4 kWh looks more reasonable.

Ah yes - I remembered having a doubt on that, I checked again the library we used to estimate those and there might have been a unit conversion error, I'll fix that ASAP tomorrow!

Edit: it's fixed, thank you @lopuhin !

[BramVanroy]

This is already looking very promising! Good stuff.

When clicking the "initialize in transformers" button, the code block should probably not center-align the code, but left align instead. That makes the code a lot more readable.

[TevenLeScao]

This is already looking very promising! Good stuff.

When clicking the "initialize in transformers" button, the code block should probably not center-align the code, but left align instead. That makes the code a lot more readable.

Yeah that was a bit of an aesthetic choice to not break the flow of the web page, it definitely wouldn't be like this in a tool rather than a demo!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[julien-c]

unstale, what's the status on this @TevenLeScao? Should we close?

[TevenLeScao]

@julien-c we had originally decided not to go forward with this, but I started working on it amongst the discussions about the scale of GPT-3. I didn't get to finish it before leaving for holidays two weeks ago, but the PR will be ready this week.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.