Insightface: training acc is stayed about 0.63 with DeepGlint (181K ids/6.75M images)

Created on 24 Jan 2019 · 4Comments · Source: deepinsight/insightface

Thank you for share.

I have used this insightface code and pretrained params(LResNet100E-IR,ArcFace@ms1m-refine-v2) you share to fine-tune on DeepGlint,
but the training acc is stayed at about 0.63, stoping increase.

I use
--per-batch-size 50 on 3 GPU
--emb-size 512
gpu num: 3
num_layers 100
image_size [112, 112]
num_classes 180855
Called with argument: Namespace(batch_size=150, beta=1000.0, beta_freeze=0, beta_min=5.0, bn_mom=0.9, ce_loss=False, ckpt=2, color=0, ctx_num=3, cutoff=0, data_dir='../data/faces_glint', easy_margin=0, emb_size=512, end_epoch=100000, fc7_lr_mult=1.0, fc7_no_bias=False, fc7_wd_mult=1.0, gamma=0.12, image_channel=3, image_h=112, image_size='112,112', image_w=112, images_filter=0, loss_type=4, lr=5e-06, lr_steps='50000,100000,150000,200000', margin=4, margin_a=1.0, margin_b=0.0, margin_m=0.5, margin_s=64.0, max_steps=0, mom=0.9, network='r100', num_classes=180855, num_layers=100, per_batch_size=50, power=1.0, prefix='../ckpt/glint_faces_back2/glint_faces', pretrained='../ckpt/glint_faces_back2/glint_faces,46', rand_mirror=1, rescale_threshold=0, scale=0.9993, target='', use_deformable=0, verbose=2000, version_act='prelu', version_input=1, version_multiplier=1.0, version_output='E', version_se=0, version_unit=3, wd=1e-05)
loading ['../ckpt/glint_faces_back2/glint_faces', '46']
[19:14:22] src/engine/engine.cc:55: MXNet start using engine: ThreadedEnginePerDevice
init resnet 100
0 1 E 3 prelu

I tryed decreased learning rate(even to 1e-5), decreased weight decay(wd, to 5e-6), but the training acc is still stay at about 0.63?

I wonder whether the clsss num 180855 is too large for the embedding(512) to map to?

So can you help me check it, thank you very much.

Source