Vision: Add ProxylessNAS models

[Lyken17]

ProxylessNAS is a method proposed in https://arxiv.org/abs/1812.00332, which aims to _directly_ and _efficiently_ search CNN on target task (CIFAR/ImageNet) and hardware (CPU/GPU/Mobile).

Because the direct specialization, ProxylessNAS is 1.8x times faster than MobilenetV2 (at ResNet50 level accuracy) and also has better accuracy / speed trade-off on different hardware

The code/models are already released at github. I would like to create a pull request to contribute this model to torchvision. Is there any point I need to take care?

[fmassa]

Hi,

Thanks for raising this issue.

More recently, there has been MobileNetV3 that has been implemented. Do you think there are advantages of ProxylessNAS over MobileNetV3?

[Lyken17]

@fmassa

1. Performance: Compared with MobilenetV3, proxylessnas's accuracy is still competitive when tricks like SE / Swish / special training schedule are not used.
   MobilnetV3 (75.2) / Proxyless-CPU (75.3) / Proxyless-GPU (75.1) / Proxyless-Mobile (74.6)
2. Hardware Specialization: MobilnetV3 does not consider underlie hardware, but it is known that each platform has different preferences, like GPU is better at parallel computing and CPU has higher frequency. MobilenetV3 tends to build a general CNN for all platforms, but the figure below shows the customization can bring 20~35% speed boost without loss on accuracy.

[Lyken17]

@fmassa Any suggestions for making a PR?

[fmassa]

Hi @Lyken17

Sorry for the delay in replying, I was on holidays for the last couple of weeks.

I think it might be better for now to add those models to TorchHub https://pytorch.org/hub .
TorchHub has been built exactly for that, and I think would be a great fit for now for ProxylessNAS.

The training code for Proxyless seem to be different than what we have in torchvision/references, which we try to enforce as much as possible.

Thoughts?

[zhanghang1989]

ProxylessNAS is a method proposed in https://arxiv.org/abs/1812.00332, which aims to _directly_ and _efficiently_ search CNN on target task (CIFAR/ImageNet) and hardware (CPU/GPU/Mobile).

Because the direct specialization, ProxylessNAS is 1.8x times faster than MobilenetV2 (at ResNet50 level accuracy) and also has better accuracy / speed trade-off on different hardware

The code/models are already released at github. I would like to create a pull request to contribute this model to torchvision. Is there any point I need to take care?

Any plan to release the search pipeline for ProxlessNAS?

[Lyken17]

@zhanghang1989 Sure, we love open source. The searching code is now released on the same git repo.

@fmassa Sorry for coming back late, was rushing for ICLR in last few weeks .

As for the training settings, we are using

1. The standard data preprocessing (color jitter, random crop, random flip)
2. Learning rate warmup for first 5 epochs (to better fit distributed training, can be removed if on a single machine)
3. Learning rate cosine scheduling instead (different torchvision/references using stepLR)
4. Optional label smoothing for better performance.

The key difference is we are using cosine LR schedule instead of the multi-stage strategy. As mentioned in MobilenetV2 paper, they decay the learning rate by 0.97 every 2.4 epochs [1]. This schedule is specially designed for MBConv modules and hurts other models' performance (e.g., ResNet). Therefore, we choose a more general cosine schedule. It (1) performs as good as StepLR on tradition models like VGG and ResNet (2) fully re-produces the accuracy of MobilenetV2. I think it is a fair setting. Let me know if you have any comments.

[1] https://arxiv.org/pdf/1807.11626.pdf MnasNet: Platform-Aware Neural Architecture Search for Mobile
[2] https://arxiv.org/pdf/1812.01187.pdf Bag of Tricks for Image Classification with Convolutional Neural Networks

[fmassa]

@Lyken17 sounds good, thanks a lot for the information!

Can you first send a PR to torchhub https://github.com/pytorch/hub adding your models? I think it would be a better fit for now, and then we can reconsider adding ProxylessNAS into torchvision in a few months.

[gyeongin]

Hi @Lyken17, do you have any plan for releasing the trained CIFAR-10 models as well? It looks like the repository (https://github.com/mit-han-lab/ProxylessNAS) only contains the ImageNet models.

[Lyken17]

@fmassa How does the hubconf.py look? I have tested it locally and can load without error.

model = torch.hub.load('mit-han-lab/ProxylessNAS', 'proxyless_cpu')

and the PR is https://github.com/pytorch/hub/pull/61. Let me know if you have any suggestions.

PS: hub is indeed a cool feature. Love it!

[Lyken17]

Hi @gyeongin , we have no plan to clean the CIFAR codebase and release it. If you are interested in searching on CIFAR, you can have a look at Path-level ENAS, from which our CIFAR experiment settings are adapted from.

For pretrained models, we have shared it on Google Drive.

[fmassa]

ProxylessNAS is now available via TorchHub, and can be found in https://pytorch.org/hub/pytorch_vision_proxylessnas/

Closing this issue for now, will revisit it in 6 months.

Thanks for the discussion!