Insightface: AssertionError when retrain r100 using triplet loss option.

[AnhVPB]

Hi, I got an error that the same with #528. I followed README and sample_config.py but cannot solve this bug.

Here is my command:
CUDA_VISIBLE_DEVICES='0,1' python -u train.py --network r100 --loss triplet --lr 0.005 --pretrained ../models/model-r100-ii/model --pretrained-epoch 0

Ouput:
gpu num: 2
prefix ./models/r100-triplet-emore/model
image_size [112, 112]
num_classes 344
Called with argument: Namespace(batch_size=120, ckpt=3, ctx_num=2, dataset='emore', frequent=20, image_channel=3, kvstore='device', loss='triplet', lr=0.005, lr_steps='100000,160000,220000', models_root='./models', mom=0.9, network='r100', per_batch_size=60, pretrained='../models/model-r100-ii/model', pretrained_epoch=0, rescale_threshold=0, verbose=2000, wd=0.0005) {'net_act': 'prelu', 'emb_size': 512, 'data_images_filter': 0, 'num_layers': 100, 'loss_name': 'triplet', 'val_targets': ['lfw', 'cfp_fp', 'agedb_30'], 'ce_loss': True, 'net_input': 1, 'image_shape': [112, 112, 3], 'lr': 0.05, 'triplet_bag_size': 1720, 'fc7_lr_mult': 1.0, 'ckpt_embedding': True, 'triplet_max_ap': 0.0, 'net_unit': 3, 'net_output': 'E', 'data_rand_mirror': True, 'num_workers': 1, 'triplet_alpha': 0.3, 'dataset': 'emore', 'num_classes': 344, 'fc7_no_bias': False, 'loss': 'triplet', 'data_color': 0, 'dataset_path': '../datasets/split_data/train', 'data_cutoff': False, 'images_per_identity': 5, 'net_se': 0, 'net_multiplier': 1.0, 'fc7_wd_mult': 1.0, 'network': 'r100', 'per_batch_size': 60, 'net_name': 'fresnet', 'workspace': 256, 'max_steps': 0, 'bn_mom': 0.9}
loading ../models/model-r100-ii/model 0
[09:47:02] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
[09:47:02] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
0 1 E 3 prelu
INFO:root:loading recordio ../datasets/split_data/train/train.rec...
header0 label [49296. 49640.]
id2range 344
49295
rand_mirror True
5 12 1744
triplet_seq 1720
loading bin 0
loading bin 1000
loading bin 2000
loading bin 3000
loading bin 4000
loading bin 5000
loading bin 6000
loading bin 7000
loading bin 8000
loading bin 9000
loading bin 10000
loading bin 11000
loading bin 12000
loading bin 13000
loading bin 14000
loading bin 15000
loading bin 16000
loading bin 17000
loading bin 18000
loading bin 19000
loading bin 20000
loading bin 21000
loading bin 22000
loading bin 23000
(23458L, 3L, 112L, 112L)
ver lfw
lr_steps [100000, 160000, 220000]
call reset()
eval 1720 images.. 0
triplet time stat [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/anhvo/miniconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/anhvo/miniconda2/lib/python2.7/threading.py", line 754, in run
self.__target(self.__args, *self.__kwargs)
File "/home/anhvo/miniconda2/lib/python2.7/site-packages/mxnet/io.py", line 401, in prefetch_func
self.next_batch[i] = self.iters[i].next()
File "/home/anhvo/insightface/recognition/triplet_image_iter.py", line 480, in next
self.reset()
File "/home/anhvo/insightface/recognition/triplet_image_iter.py", line 390, in reset
self.select_triplets()
File "/home/anhvo/insightface/recognition/triplet_image_iter.py", line 253, in select_triplets
self.mx_model.forward(db, is_train=False)
File "/home/anhvo/miniconda2/lib/python2.7/site-packages/mxnet/module/module.py", line 591, in forward
assert self.binded and self.params_initialized
AssertionError

/home/anhvo/miniconda2/lib/python2.7/site-packages/mxnet/module/base_module.py:504: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (0.5 vs. 0.00833333333333). Is this intended?
optimizer_params=optimizer_params)

[wangxm345566462]

    model = mx.mod.Module(
        context=ctx,
        symbol=sym,
    )
    model.bind([("data", (args.batch_size, args.image_channel, image_size[0], image_size[1]))], [("softmax_label", (args.batch_size,))])

[AnhVPB]

Tks so much @wangxm345566462. So the problem is the model isn't binded. How about params_initialized? Do I need to initialize params after bind model?

[wangxm345566462]

非常感谢@ wangxm345566462。所以问题是模型没有绑定。params_initialized怎么样？我需要在绑定模型后初始化params吗？

before
self.mx_model.forward（db，is_train = False）

[AnhVPB]

ok tks @wangxm345566462