Transformers: More flexible wandb support for Trainer

[Guitaricet]

🚀 Feature request

A. Make it possible to initialize wandb outside Trainer class.
B. Add use_wandb argument to the Trainer arguments.

Motivation

A. Currently, wandb inside Trainer configuration if very limited. There are only three environment variables WANDB_WATCH, WANDB_PROJECT, and WANDB_DISABLED. (And WANDB_DISABLED does not work properly in some cases).

Making it possible to initialize wandb outside Trainer will allow us to:

1. Add custom fields to wandb.config
2. Add tags, notes and just to make configuration as flexible as possible
3. Upload files
4. Use wandb.log more safely outside the Transformers code

It will also make the interaction with wandb more clear.

B. It is a much more clear interface then an env variable. The question here is which option should have the priority?

Your contribution

There's a picture in my mind about how to do this and not to destroy backward compatibility. I can make a PR, but maybe need some minor help on writing tests.

[misrasaurabh1]

Yeah I agree with the motivations here. From my experience I like to initialize wandb as soon as the main script starts which has benefits like

• Capturing all the console logs, so if something failed before the execution reached the trainer class one could debug the issue through Wandb. This is specially useful for doing training in a kubernetes environment where the logs are not very easily available after a crash.
• Instantiating at the beginning also allows us to capture all the CLI arguments properly before they might have been modified. This is a problem right now because Wandb only captures the training_args leaving out model_args or data args. This is important for replicating a training run in the future.

[borisdayma]

Doesn't it work already when initializing wandb outside?

I believe that wandb.init does not create a new run if one is already running so all the functions should be available anywhere in the script.

[Guitaricet]

When looking over the code, I was pretty sure it will initialize a new run. However, I checked and everything (mentioned in the issue) works smoothly.

[julien-c]

Ok, thanks for checking @Guitaricet