Keras-retinanet: efficientnet backbone

[lukasschmit]

I am looking into optimizing a retinanet model by using efficientnet. Would it be possible to use the pretrained models available from Google's TPU repository directly?

I know that #233 mentions that a model needs to have 5 stages, although from the paper it seems like efficientnet has 9 stages for its B0 model (correct me if I'm wrong).

[hgaiser]

The current structure of RetinaNet assumes, indeed, that there are 5 stages. I haven't read efficientnet, but if it has 9 stages, it will be challenging to get it to work together with retinanet. Once you have it integrated within retinanet I would assume that the pretrained models from Google can be used, perhaps with a small conversion in between.

[evolu8]

I'm very interested in this. Is it really TPU dependant though? I've not looked but I was hoping EfficientNet would be runnable on GPU with minor or no mods. No?

[lukasschmit]

@evolu8 Google's TPU implementation supports GPU execution from what I can tell from the code. Seems like you can just pass --use_tpu False to the command line.

But if not, there are several Keras implementations of EfficientNet cropping up -- this one seems pretty solid.

[evolu8]

Ah, thanks! Sorry, I misread your post, thinking it implied TPU support was needed. My bad. :)

[evolu8]

Could seem a little hacky, but might ignoring the even blocks work? It just wants features, right?

[lukasschmit]

That is pretty hacky but could work. How about just the last 5?

[evolu8]

Might lose signal on pixel level textures. I'm favouring a spread across the network. But that is pure intuition.

[mosheliv]

Would be very interested in this as well. I have used efficientnets in a project (classification) and they perform beautifully (i used https://github.com/qubvel/efficientnet). They converge very quickly and perform much better then even seresnext.

[ajoseph12]

Has anybody tried efficientnets on medical images (greyscale : 8-bit or 16-bit). I have tried swapping the backbone for EfficientNet B3 (trained on imagenet), but the results are far from promising, I have better luck with VGG-16.

[hgaiser]

I finally read up on EfficientNet and it looks very promising. I don't have time to create a backbone for it, but it would be very interesting to have it as an addition to keras-retinanet.

[evolu8]

Has anybody tried efficientnets on medical images (greyscale : 8-bit or 16-bit). I have tried swapping the backbone for EfficientNet B3 (trained on imagenet), but the results are far from promising, I have better luck with VGG-16.

Not yet, but that is my intention. What was your experience? Want to take it off thread? ([email protected])

[lvaleriu]

I have implemented EfficientNet as a backbone and started training on Coco. I'm available to share some info about this experiment on Keras-Retinanet Slack channel: https://kerasteam.slack.com/messages/C7P6JEQDU/

[mosheliv]

This is a very exclusive slack...

On Thu, Jul 4, 2019, 10:34 Valeriu Lacatusu [e-mail]
wrote:

I have implemented EfficientNet as a backbone and started training on
Coco. I'm available to share some info about this experiment on
Keras-Retinanet Slack channel:
https://kerasteam.slack.com/messages/C7P6JEQDU/

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fizyr/keras-retinanet/issues/1038?email_source=notifications&email_token=AC7IWC4VKKKM3JY3Y22KV43P5USNVA5CNFSM4HZMRV72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZF3A5Q#issuecomment-508276854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7IWC7NDGREHFWC4HXXSJLP5USNVANCNFSM4HZMRV7Q
.

[lukasschmit]

@lvaleriu That's awesome! I can't get on that slack, but would you mind sharing briefly what you did? Or if it works well any chance you'll make a PR/fork I can look at?

[lvaleriu]

Maybe this link https://keras-slack-autojoin.herokuapp.com will work?

[mosheliv]

It works but I am not sure how to follow what you do there. I am not a slack guru or anything.
My success with efficientnets was with adam as optimizer and dropping lr by factor of 10 the minute validation stopped showing loss improvement. I actually got very fast convergence so I wonder why everyone say its painfully slow... I used the keras implementation https://github.com/qubvel/efficientnet

[lvaleriu]

I'm using the same implementation with some small changes (I suppose you've use it as a classifier backbone and not object detector, right?). The difficulty here is to choose the right backbone layers and the fact that the multigpu doesn't speed up the training (it is simply very long) so we need to wait days/week before seeing improvements in mAP.

I think we should need to add 2 things in order to speedup the training (like FastAI or ktrain "frameworks"):

• estimate an optimal learning rate for your model given your data using a Learning Rate Finder
• utilize learning rate schedules such as the triangular policy, the 1cycle policy, and SGDR to effectively minimize loss and improve generalization

[mosheliv]

Mmmm I used the "standard" lr of 0.0001. I did notice it did not like overly large batch sizes (8 worked very well) and no drop. I used a slightly more complicated head then the usual but this shouldn't really matter for features.
Yes, I used it for classification, but this is what most of the complaints are about. I am not very familiar (yet) with object detectors structure to give advice re that.
So, just give it a try, even with something as simple as

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.1,
    patience=1, #important!
    verbose=1,
    mode='auto',
    epsilon=0.0001)

[evolu8]

Scaling...

A huge part of EfficientNet's benefit is its compound scaling. I think this would be particularly useful in a detector scenario. Both for scaling the input and retaining good receptive fields, but also in scaling the feature extractor capacity to the desired maximum resource requirements (speed, size... )

Anyone looked at this yet?

[heng2j]

Hi everyone! This is seem to be more demands on verifying how well EfficentNet can perform on object detection tasks. I am curious on the progress we have made so far. Can anybody share what you have learned so far?

[lukasschmit]

I got Google's Retinanet to start training with their Efficientnet on GPU, but it always crashes two days in with an obscure CUDA error. Huge pain to debug so that's kind of a dead-end.

[vanvalen]

I've been able to get it to work on a fork/adaptation of the Retinanet library (https://github.com/vanvalenlab/deepcell-tf/blob/enhance_retinanet/deepcell/utils/backbone_utils.py is the file of interest). I didn't see striking performance on the datasets I was trying it on, but perhaps someone will have better luck on theirs.

[DableUTeeF]

Is there any update on this?
I'm using EfficientNetB0 with YoloV3 and it performs very good but RetinaNet doesn't seems to converge at all.
Loss always around 4.1 regardless of I freezed batchnorm or not, batch-size 1 or more, even with COCO dataset.
Though, I tried to implement NasNet before EfficientNet comes out but it doesn't converge either, even though both of them work well with Yolo.

[ZFTurbo]

We added EfficientNets with many other backbones to repository with this pull request:
https://github.com/fizyr/keras-retinanet/pull/1158

I currently train EfficientNetB0 with Open Images Dataset. It converges well:
EfficientNetB0 Freezed backbone (384-512px) + 500 classes + pretrained decoder from REsNet152:
Ep 1: 8238s 824ms/step - loss: 2.7720 - regression_loss: 1.8499 - classification_loss: 0.9221 mAP: 0.0218
Ep 2: 8130s 813ms/step - loss: 2.4208 - regression_loss: 1.6520 - classification_loss: 0.7688 mAP: 0.0528
Ep 3: 8151s 815ms/step - loss: 2.3317 - regression_loss: 1.6046 - classification_loss: 0.7271 mAP: 0.0794
Ep 4: 8215s 822ms/step - loss: 2.2677 - regression_loss: 1.5663 - classification_loss: 0.7014 mAP: 0.0935

[hgaiser]

Fixed with https://github.com/fizyr/keras-retinanet/pull/1158 , thank you @ZFTurbo .

[RituYadav92]

Hi,
Anyone here is able to get better results on efficient net as compared to resnet for object detection??

[ianstath]

from efficientnet.model import BASE_WEIGHTS_PATH
ImportError: cannot import name 'BASE_WEIGHTS_PATH'

I am receiving this error trying to use " !python3 keras_retinanet/bin/train.py --backbone EfficientNetB0 --random-transform --lr 1e-4 --imagenet-weights --batch-size 8 --steps 300 --epochs 50 --no-resize --compute-val-loss csv annotations.csv classes.csv --val-annotations annotations_val.csv

I am using colab.
Thank in advance for any suggestions