Mmdetection: How to use Adam instead of SGD?

[GYee]

All the optimizers are defined as:
optimizer = dict(type='SGD', lr=2e-3, momentum=0.9, weight_decay=5e-4)
But I want to change it to Adam, how should I do ?
Can anyone give me an example?Thx!

[ZwwWayne]

Hi @GYee ,
You can set the type='Adam' then set other parameters as usual.
The call is in this line, and you can see that any subclass of torch.optim and their parameters are applicable.

[ZwwWayne]

BTW, we did not try Adam before, so it might cause some bug when you try it.

[StringBottle]

@GYee

As far as I used Adam instead of SGD, expecting faster training speed, the loss diverged to larger than 1000 so I stopped the training. I haven't got a satisfactory training result, but at least training was stable using SGD.

[Hemantr05]

@StringBottle +1

[ZwwWayne]

The loss diverged simply because the learning rate=0.02 is too large for Adam.
Try 1e-3 or 3e-4 and you will get reasonable results. However, the results are still much lower than that of using SGD (bbox mAP 31.1 and segm mAP 28.6 with 3e-4 on Mask R-CNN using ResNet-50). We suggest more hyper-parameter tuning when using Adam if Adam is inevitable in your experiments.