Icevision: Integrate Pytorch Image Models as Hub Module

[oke-aditya]

🚀 Feature

Many SOTA implementations in Pytorch are available in this well maintained repo.
I guess we could have them as hub module.
It can tightly be integrated later.
https://github.com/rwightman/pytorch-image-models

[oke-aditya]

• We will use ross models, but not make package dependent on it. Nor we will provide fixes to his code / take any part of his code as our liability.

Proposal 1: -

• Make us as hub to connect to his models

Dream API

model = hub.models.hrnet() # Should redirect to here https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/hrnet.py

Models corresponds to rwightman's models. So we use his models, but not make it mandatory codebase. Also errors will resonate to his codebase. So we do not take that responsibilty.

[oke-aditya]

• What we would take care is making them usable as backbones. That's the only code that we take responsibilty to maintain.
• These should provide a more flexible codebase.

[lgvaz]

Related to #122 , should these models be inside hub? Or should we create another abstraction?

What are these models? Only backbones or also completely new algorithms?

Let's say we decide to integrate YOLO, how would that look like?

[oke-aditya]

These are simply CNNs, provided by Ross, he also provides a few optimizers, layers too. We could take them too as a part.
Maybe we can rename them. They are definetely not bakcbones, we would need to add code to make them useful as backbones.

[oke-aditya]

What Ross provides is set of CNNs, some layers, some optimizers (as of now). We need to add these to our hub.

These CNNs -> Backbone conversion is code to be handled by us, as this would make it compatible with mantisshrimp.

User should access these models as backbones through our code.

[lgvaz]

To fully understand how to properly do this, we need to specify how the integration of a custom "algorithm" would be like.

Sure, having new backbones and layers to experiment with torchvision MaskRCNN would be nice, but nothing changes much.

This is why I'm asking, how would we integrate YOLO? Or maybe, how to integrate efficientdet from Ross?

[oke-aditya]

Its tricky, coz we want to take responsibility of only maintaing integration of custom algorithms. But not EfficientDet in whole. I am Unclear at this point. Need your thoughts.

[oke-aditya]

We have to do something very similar to what we did to torchvision models.

[lgvaz]

A good first step would be experimenting with these libraries first, to understand how they work.

Let's try Ross efficientdet and a version of Yolo, maybe this or this

[oke-aditya]

Taking inspiration from Microsoft who are trying similar thing. Here. See how they embedded somebedoy else's library into their repo.

[oke-aditya]

Simply add this cnns in a folder and make utils on top of that to make it compatible with Fastai and Lightning. Same can be done for YOLOv5 and even EfficentDet.

[ai-fast-track]

Like @oke-aditya mentioned, here above, the Microsoft approach is an interesting one: Freeze the external (model) repo, and connect it to Mantisshrimp with very minimal changes. Freezing the repo (not the model) avoid the problem of an ever changing repo that triggers error in our package because of the eventual misalignment between our package and their new versions.

[lgvaz]

Yep, this is what we are already doing with DETR

[oke-aditya]

I guess folders is better coz the source repository changes very fast and can cause code break. Also making folders will allow us to do a minor edit in their code. It might be overhead to update the folder but, we can slowly keep pushing these changes (with tests it will be easy). This will allow us to edit models and add customizations. (That's what microsoft intended maybe).

[lgvaz]

A github submodule is very similar to having a folder, if you take a look at detr, it actually points to a fork of the original source code, where I modified some of the stuff.

As any git repo, we have the power to choose when to update it, so it does not break out of nowhere.

We also don't want the folder to really be a part of our library, just a dependency.

[oke-aditya]

But it does not get cloned. We do not want their entire repo from people. We just need parts of it. Also while patching we might not wish to patch the entire repo. So some manual labour here is affordable.

[oke-aditya]

Models which are pure PyTorch code will not create dependency. It would just be addition.

[oke-aditya]

pip install timm does the job 👍 . I guess we will add this to our requirements.txt (not extra) and work on Integrating these stuff as backbones.

[oke-aditya]

Solution to this.

• [x] 1. Work out backbones to integrate timm.
• [ ] 2. Create a nice Tutorial to show how tightly we are integrated with it.
• [ ] 3. Add these to docs.

[lgvaz]

This is outdated, we can open a new issue about integrating timm to backbones if necessary