Vision: NASNet Model

[ahkarami]

Recently the Google Brain Team released a fantastic CNN model, NASNet, in TF-slim, which achieved the state-of-the-art Top-1 Accuracy on ImageNet by 82.7 %. I want to know that the PyTorch team has any plan for implement or porting this model into the PyTorch Offifcial Models (_i.e.,_ torchvision models)?

[alykhantejani]

Hi @ahkarami,

In torchvision we would like to have models that have been trained in pytorch using pytorch + torchvision so that they are also reproducible by the community. If somebody from the community would like to train these networks, we would be more than willing to accept a PR + the weights

[Cadene]

@alykhantejani and InceptionV3 is an exception :P

[soumith]

@Cadene indeed it's an exception because we could never train Inceptionv3 to the same accuracy as google's paper in my few attempts.

@Cadene by any chance do you have a NASNet definition already? I can kick-off training if so...

[Cadene]

Not already, but I am currently working on it. I’ll let you know asap

[aussetg]

@Cadene, how far are you? I was going to start porting Tensorflow Slim's model definition as the paper alone is... light in crucial details and parameters to say the least. Don't want to uselessly duplicate efforts :)

[Cadene]

@aussetg I can't focus on it too much because of some deadlines, but I am definitely working on it.
To give you an idea, I loaded the model on TensorBoard and I am currently porting the first two blocks (CellStem0 and CellStem1).

[Cadene]

@soumith @aussetg I did not port the pretrained parameters yet to validate the model, but at least you can run a forward + backward pass on this version.
https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/nasnet.py

[aussetg]

@Cadene You implemented the separable convolution using Conv2d(in=out=groups) + pointwise, while it won't work on the release version wouldn't it be better to make use of https://github.com/pytorch/pytorch/pull/3057 ? i.e merge Conv2d(in, in, in) + pointwise(n) with conv2d(in, in*n, in)

[Cadene]

@aussetg It makes sense. I just added a global variable to switch between the two implementations. https://github.com/Cadene/pretrained-models.pytorch/commit/99deb0e503ed70021142baa04cce02264d25c31a

[soumith]

new release coming this week, so feel free to switch to that suggestion by @aussetg

[Cadene]

NASNet-A-Large has been successfully ported to pytorch! https://github.com/Cadene/pretrained-models.pytorch

However, the forward pass is a bit slow. During the evaluation:

• 3 seconds per batch of 50 images with the old version of pytorch on a GTX 1070
• 1.5 seconds per batch with the last version of pytorch from master (cudnn v7 and cuda 8)

[fmassa]

@Cadene awesome! Do you have an idea on how fast it runs on TensorFlow?
I would expect it to be slower in PyTorch because of the padding difference in TF, but I'm curious to know how much slower it is.

[Kulikovpavel]

Can you add NASNet medium?
According to this, it is best accuracy/complexity network

[fmassa]

Sure, NasNet (medium) sounds like a good addition as well

[Cadene]

@Kulikovpavel Hi, did you see any available pretrained model for NASNet medium somewhere? Did not find any in tensorflow/slim directory :/