Icevision: WandB integration

[oke-aditya]

🚀 Feature

How to integrate Weights and Biases into Mantisshrimp.

Let us define our objectives,

1. What needs to be implemented. What needs to be added.

PyTorch Lightning supports a lot of logging tools natively. Do we need to edit our code or we just make use of what lightning has?
Also, do we need to edit our source code or do a demonstration/tutorial of how to use these loggers?

Related to #84 #83

[lgvaz]

We don't need to edit our code at all, support for wandb should be given by the training engine.

Luckily both fastai and lightning already supports it =)

[oke-aditya]

Can we create a checklist of tasks to be done ?

[lgvaz]

• [ ] Understand all wandb capabilities, it can be used for more than just benchmarking
• [ ] Can we use it for visualising model predictions? Where is the model failing at, where it's good at, etc
• [ ] Hyperparameters tuning: #84
• [ ] Compare different architectures (wait for #149)
• [ ] Compare the difference in training when using a pretrained model on imagenet vs coco
• [ ] Are there default useful logs we can provide for ours models?

---

Resources:
https://app.wandb.ai/stacey/deep-drive/reports/The-View-from-the-Driver's-Seat--Vmlldzo1MTg5NQ

[oke-aditya]

I will update as I make progress.

[ai-fast-track]

Related to the resource mentioned here above: I watched these 2 interesting (short) video tutorials: Highly recommended.

Exploring the W&B Dashboard
Welcome to Weights & Biases - Introduction Walkthrough (2020)

[ai-fast-track]

The way I see it, we could create a separate tutorial (notebook) dedicated to Wandb Integration where we show how to use wandb with mantis. You could showcase both fastai2 and Lightning integration using the Pets dataset.

The notebook could be a stripped version of the 01-getting _started notebook. We can start by simply using the wand default config for both fastai2 and Lightning, and then iterate from there.

For the incubator, we will most likely build our own wandb config to highlight the stuff that would have more impact for that competition.

[oke-aditya]

Great. I will list down the tasks which we should do. We can make a project open to everyone to build and run so that people can experiment with it.

• [ ] Basic Wandb tutorial. This should build upon quickstart concepts. (Both Fastai and Lightning)

• [ ] Wandb Hyperparameter Tuning with sweeps: - A Tutorial of how to tune hyperparameters of a model. (Lightning Preferrable)

• [ ] Wandb Benchmarking on a Dataset: - Let's take Pets. It's more diverse. We need to benchmark different models, with different Hyperparameters. We need to create a nice project report for it too. (Lightning Preferrable). This should also include
  visualizing model predictions. This can compare much stuff like Transfer learning vs from scratch training, ImageNet weights vs COCO weights. Different Backbones, Different Architectures, Different Hyperparameters. different Optimizers, Having Fastai vs Lightning too. This is a huge one so should be the last tutorial.

Also, we need to pinpoint some resources for the user to learn about wandb. It is not our task to make users learn wandb, but to show how good it is with our package.

I guess these 3 tutorials would be comprehensive. Do share your thoughts.

[oke-aditya]

Please have look at this. @ai-fast-track @lgvaz @paras-jain .
Let me know what improvements are needed. It is silightly complex to set up and do the job. Also, we need to make .py files as wandb does not support jupyter very well. So how does that go in our docs ?

[paras-jain]

Please revise the adapters, since RCNNLightningAdapter is now deprecated.

[oke-aditya]

Oh yes sure. @paras-jain you too worked on this. My repo is just for testing stuff, can you raise a PR with stuff you tried in a seperate folder ?
When we will create tutorials maybe we will put in docs.

[paras-jain]

As discussed I got unclear results, but I'll raise a PR tomorrow.

[tugot17]

If RCNNLightningAdapter is deprecated, then what is a current "canonical" approach to using Lightning?

[oke-aditya]

We should use.
faster_rcnn.lightning.ModelAdapter
This is the small change. Just replace the RCNNLightningAdapter with this.

After this change it works. I have verified. Do let me know if you need help. 😀

[tugot17]

From where should I import faster_rcnn.lightning.ModelAdapter?

from where import ModelAdapter

class LightModel(ModelAdapter):
    def configure_optimizers(self):
        opt = SGD(self.parameters(), 2e-4, momentum=0.9)
        return opt

[tugot17]

I solved it using from mantisshrimp.models.rcnn.lightning.model_adapter import ModelAdapter

[lgvaz]

Another option is:

from mantisshrimp.models.rcnn import faster_rcnn

faster_rcnn.lightning.ModelAdapter

[ai-fast-track]

Since Fastai2 has already a good integration with Wnadb through the WandbCallback callback. I think we should give a try.

https://dev.fast.ai/callback.wandb

Here is the code snippet:

import wandb
from fastai2.callback.wandb import *

# start logging a wandb run
wandb.init()  # optional -> wandb.init(project='my_project')

# just add WandbCallback to your learner
learn.fit(..., cbs=WandbCallback())

Check out Boris Dayma's blog post: Compare & monitor fastai2 models

[lgvaz]

Closing this in favor of #84

WandB is already integrate into fastai and lightning, we just need a tutorial on how to use it