Fairseq: Weights and Biases support

[Guitaricet]

🚀 Feature Request

Add WandB support for logging.

Motivation

• WandB is cloud-based - you can access your logs without a direct ssh connection, just using the website
• WandB interface is better suited for many experiments, it also can log model weights, configs, and source code
• WandB is prettier

Pitch

Tensorboard is slow a buggy - sometimes I wait about a minute for it to load all the logs, sometimes it just doesn't show them.
There is no reason not to allow fairseq users to use wandb.
It is also very simple to add a new progress bar wrapper (and I already did that).

Alternatives

Logger extension interface (plug-in) which can allow users to use any logging method.

Additional context

I already coded this feature for myself. Currently, I am using it for my experiments and it looks pretty good. You can see PR draft #1781 for more information.
I'll be happy if you can help me to understand why Windows build is failing, I did not write any platform-specific code and I have no windows computer to test it.

[Guitaricet]

Anybody? =(

[CSP197]

@Guitaricet - I can help with this and I have a Windows computer for testing =)

[Guitaricet]

Awesome! You can look into this ci log. Tests are failing for windows-latest, 3.6. Everything is looking good for Ubuntu and MacOS.

Also, what kind of tests does wandb integration require (if any)? I did not find any tests for tensorboard.

[Guitaricet]

@CSP197 thank you for offering help, but I just noticed that this test is failing on all PRs, including documentation improvements. So this is not a problem of this one.

[CSP197]

@Guitaricet - Yeah, I think its a permissions issue with the Windows system.
I can run the tests successfully on my local machine...

Ran 116 tests in 166.650s OK (skipped=17)

...but run into the same issues when running on the Github Actions workflow.

[Guitaricet]

@CSP197 thank you!
Now I hope somebody from fairseq will find this feature useful.

[Guitaricet]

PR is ready for review

[Guitaricet]

@myleott sorry that I am asking you directly, but could you please comment if you want this feature in fairseq and accept the PR if so.

[BadrYoubiIdrissi]

Would love to see this feature added :) Although maybe exposing a logging extension interface might be more appropriate. It would avoid core fairseq from having many cutom logging progress bars.

[Guitaricet]

@BadrYoubiIdrissi I improved my implementation a bit - now it also logs the model architecture, gradient norms, and the logger output before the training. The fork is also updated. Hope you'll find it useful.

[rmrao]

Would be really nice to see this merged!

[myleott]

Merged in 1a709b2a401ac8bd6d805c8a6a5f4d7f03b923ff