Insightface: iccv19 light weight face recognition challenge tips

Created on 16 Apr 2019  ·  193Comments  ·  Source: deepinsight/insightface

Challenge Information:

HomePage, please read carefully about the training and testing rules.

How To Start:

Please check https://github.com/deepinsight/insightface/tree/master/iccv19-challenge for detail.

Any question can be left here.

Most helpful comment

@nttstar Can the test server still be used for the purpose of evaluation after competition is over?
Thanks,

All 193 comments

HI @nttstar, my traditional question regarding the dataset, is it already 5-keypoints aligned? Sorry, have not downloaded it.

@borisgribkov Yes already aligned, so no need to further process on it.

@nttstar what's the difference between ms1m-retinaface-t1.zip and ms1m-v2(emore)?

@zhouwei5113 ms1m-retinaface has different clean list than ms1m-arcface and it has been re-aligned with our RetinaFace detector. You must use ms1m-retinaface to participate in this challenge, all other training-sets including ms1m-arcface are not allowed.

Hi, @nttstar, could you please share with us how the provided dataset generated from ms-1m aligned dataset. I tried to upscale images in ms-1m aligned dataset to 400x400, but the landmarks do not match. It would be great if you could share with us the preparation process.

@leolordwz it's not allowed for this challenge.

How can I take part in the iccv19 light weight face recognition challenge?

@chenshen03 Test datasets and test server will be ready soon, any question on training?

@nttstar Can you provide more info about RetinaFace Detector ?

@nttstar
Is the test set aligned by RetinaFace? If not, will you release the RetinaFace Detector?
Many thanks!

@LCorleone They're both aligned by RetinaFace, so no need to do anything on training or testing datasets.

@BingranHu We can get higher accuracy on LFW, CFP_FP and AgeDB_30 by the help of ms1m-retinaface.

@nttstar Hi, where is the download links of test data?

@xizi Not ready yet.

Does the organizing association have some limitation on the number of participants?
@nttstar

@GewelsJI No

@nttstar When do you expect test server will be ready?

@grib0ed0v Soon~

I have a question about the size of parameters.
Does the 20M parameters mean total (include FC layers) size or only backbone network's size?

@sybyun
From the input layer to embedding layer.
By the way, it is 20MB size which means 5M parameters.

The test server could not log in.

@chenshen03 what's the problem? I can login successfully.

@nttstar
The account can login normally on http://trillionpairs.deepglint.com, but there was an error when it logged on to the test server http://39.104.128.76.

@chenshen03 sign up a new account, database not shared with deepglint server.

@nttstar The database is not shared, but the system prompts "user name or email already exists" when registering. Do we need to use a different email address to register?

Also, when I register with a new email box, I get an "unknown error". When I try to register again, "user or email already exists" will appear.

@chenshen03 Hi, what is your old email address?

@nttstar email: [email protected] username: Shen Chen. Thank you.

@chenshen03 Can you try again to sign up a new account with [email protected] ? We have done a cleaning.

@nttstar Successfully registered, thank you!

@nttstar I couldn't find where to download test data's pair list. Can you tell me the download link of the pair list for making the binary feature matrix (ImageNum x FeatureDim in float32) to the test server?

@yeluyue Pair list is not required.

@nttstar Then how to generate the binary feature matrix? Now I have downloaded the test data and the feature_tools from the trillion Pairs, and I think the images needing to make feature matrix exists the insightface.bin. Can you tell me whether my understanding is correct?

@yeluyue Please check the tutorial here. I have provided the feature generation tools.

@nttstar Thank you ~

Entity validate failed: user name is empty when i sumbit the feature bin. @nttstar

@tianxingyzxq Try logout and login again?

Is data augmentation(random resize and crop) permitted?

@NothingL It is allowed. But you must always use pre-aligned test dataset so normally you will get a worse result by using training time random resize and crop.

hi @nttstar ,it is too slow to upload the bin file, 7 hours passed and it only upload 75%. is the right ?
how long do you take to upload the 1.8G bin file ? thank you very much.

@ckybit less than 5 minutes. where are you?

@ckybit less than 5 minutes. where are you?
i'm in china and i use company's network. i will change the network to try again,thks for your quick reply.

@nttstar Is there any approach to upload test bin file on my training server directly, other than by browser?

@zhouwei5113 Sorry no command line tool

@NothingL It is allowed. But you must always use pre-aligned test dataset so normally you will get a worse result by using training time random resize and crop.

thanks for clarification, is that mean we cannot resize and crop test image, even the cropped size is also 112*112 ?(means using a smaller region of face)

Is there any differences between the feature matrices that need to be submitted for deepglint-light and deepglint-large, or should the feature matrices be the same for both of them? @nttstar

How do I open the .z01 and .z02 files to get the image? I am using a ubuntu system.

@NothingL Right

@yeluyue The shape of submission feature should always be (N, feature_dim) N=num of images. feature_dim can be 512, 256, etc..

@YongdongTan Please check the testing section of tutorial.

how to get Network y2 and Network r100fc?

I executed to unzip: the command always reports zip I/O error: No such file or directory
Zip error: File not found or no read permission (iQIYI-VID-FACE.zip)

@olojuwin check sample_config.py

@olojuwin check sample_config.py

Thanks a lot.How to get cfp_fp.bin or agedb_30.bin. I remember there used to be connected, but now I can't find it.

in training dataset

It's very embarrassing,^_^

Hi~ I submitted two feature files, as shown in the figure below, the ROC curve demonstrate that r50 256 rf model should achieve TPR@FAR1e-4=61%, but the result shown in table is the same as last model. May I ask the is there any rule for naming filename and description or anything I missed?

image

@luzai Hi, we will look into this issue soon, and also we will update a new iqiyi ground-truth.

@luzai Hi, we will look into this issue soon, and also we will update a new iqiyi ground-truth.

Thank you very much for your quick response and kind remind for updating ground-truth!

I am quite sorry because I just find there is a bug in my code. Thus, it is reasonable for the performance keeping the same as the last model, since new method is not applied in fact. After fixing it and resubmitting, there is some improvement on performance now.

In a word, I guess the number in the table is correct and the ROC curve may have some problems.

Hello @nttstar. On the workshops webpage you mention the release of baseline models and corresponding training logs. Any target date for this?

Cheers

for the submit test image feature, the file size is about 3.8GB. How long does it take to upload to the test server from overseas?

It seems to be very slow?

@nttstar Hellow, how can I get the FLOPs of my models?

where to get network y2?

@MinZhaoLin It will output FLOPs when you start training. See common/flops_counter.py for detail.

@ZHAIXINGZHAIYUE See recognition/sample_config.py

@hengck23 How long did you take? Where're you?

@nttstar @MinZhaoLin

@MinZhaoLin It will output FLOPs when you start training. See common/flops_counter.py for detail.

just count conv and fc?

@ZHAIXINGZHAIYUE any else operator you want to use?

@nttstar

@hengck23 How long did you take? Where're you?

I am in singapore. I cannot upload for test image (feature file size is 3.8 Gb) because of connection timeout. for test video, the feature file size is about 500 mb and it takes about 1 hour for me.

@hengck23 My friend from Singapore said he had no problem with submitting. I guess your network provider has some connection issue with AliCloud.

i am starting a discussion slack team at : https://join.slack.com/t/private-5ay1415/shared_invite/enQtNjI4NDA2MjUzNzkzLThlZDIxODBlMjZiMmIzZGVhMDRiNjExZGY2YzE5ZTEyOTczZDUxZjc0OWUyNTQzNDZkMDYxNjNkYWFkODY5OTc

joint the channel "#iccv2019-low-face-rec"

Please feel free to join. Objectives of the slack team:

  1. share some baseline code and results.

  2. discussion possible strategy and papers, etc to achieve best results.

As a start, i am open sourcing by pytorch baseline code and trained model below. Join the slack for more information!

ir50

Why I cannot find the iQIYI-VID-FACE.z01, iQIYI-VID-FACE.z02, and iQIYI-VID-FACE.zip files from the iQIYI AI completion platform? It seems that they have changed the name of the data.
So how to get the video test data?

@LeonLIU08 I can find it at http://challenge.ai.iqiyi.com/data-cluster

@LeonLIU08 I can find it at http://challenge.ai.iqiyi.com/data-cluster

Thanks for your quick response. The website jumping is quite confusing after the registration.

@nttstar I've got upload error after 100 %, when tried to upload 1.8 GB feature file from Russia. Any suggestion?

@nttstar The submission contains features only, so how you verify the model complexity of the submission?

@nttstar The submission contains features only, so how you verify the model complexity of the submission?

i suggest all team to submit a onnx model to the submission server, then the organizer can make an automatic script to compute the flops and ouput on leaderboard. It can be based on the original count_flops.py of mxnet

@mariolew @hengck23 We will make sure the leader submissions should be reproducible.

@grib0ed0v Try logout and login again

https://ibug.doc.ic.ac.uk/resources/lightweight-face-recognition-challenge-workshop/

"The upper bound of model size is 20MB."

i suppose this is number of parameters and not storage in megabytes? (1 float = 4 bytes)

@hengck23 20MB is the fp32 storage size, and 5M for number of parameters.

@nttstar The submission of feature size is limited to 5GB, which I think is not enough for video features.

@mariolew video feature file should be smaller

@mariolew

you have to combine image_features to video_feature, e.g.

video_feature = average(image_features)

or

video_feature = median(image_features)

or

video_feature = max(image_features), etc ...

@nttstar Would you be sharing the y2 and r100fc baseline pre-trained models to the public? Thanks.

@hengck23 My friend from Singapore said he had no problem with submitting. I guess your network provider has some connection issue with AliCloud.

i now set up AWS EC2 cloud computer to do the upload to the submission server, I am getting only uploading speed of about 50 to 100 kbits/sec for ec2 in singapore and hongkong. This is very slow. (the network limit of my ec2 is 5 Mbit/sec. I use google drive to transfer the file from my PC to the cloud PC. It talks less than a minute to upload and download 3.8GB feature bin file in these case)

Can anyone report there upload speed and their location? I would like to setup cloud PC there.

@hengck23 How do you generate submit bin file?Do you use this repo gen_image_feature.py ?
I don't know how to generate this easier cause I use python3 and Gluon....
@nttstar Or could you please provide a common format for the bin file?

yes, i use code from gen_image_feature.py. you can just use the copied code:

import struct
def write_binary(binary_file, data):
    rows, cols = data.shape
    with open(binary_file, 'wb') as f:
        f.write(struct.pack('4i', rows, cols, cols*4, 5))
        f.write(data.data)


def read_binary(binary_file):
    with open(binary_file, 'rb') as f:
        bytes = f.read()

    rows, cols, d0, d1 = struct.unpack_from('4i',bytes)
    assert(d0==cols*4)
    assert(d1==5)

    offset = len(struct.pack('4i', rows, cols, cols*4, 5))
    data = np.frombuffer(bytes, dtype=np.float32, offset=offset)
    data = data.reshape(rows, cols)
    return data

data is an np array with array.shape = (num_of_face_images, feature_dim).

and fyi, mxnet insightface is python3 compatible after small changes (e.g. xrange to range, and change python3 pickle load function to read python2 created pickle files)

@hengck23 Thanks anyway.

@hengck23 Thank U.

Is L2 normalization of embedding vectors required in this challenge?
L2 normalization is performed in the ArcFace, but there is another method that L2 normalization of embedding vectors is not performed.

@believeitmyway you need not to submit L2 normalization features,I think the task should use euclidean distance but not cosine distance to get similarity, so the L2 normalization is not necessary.

I share a Gluon/Pytorch(need a little change) way to get the submission bin file for image/video(later update) task.
https://gist.github.com/59d312ff89ddcb891a812fc001e15d62
@hengck23 Could you please test it if could get totally same file with gen_image_feature.py, it should be very easy refactor to Pytorch style. If you are convenient.

@believeitmyway @PistonY Evaluation metric is cosine.

So it’s only support margin based loss...

No, all embedding based approaches should be fine. (Softmax, Triplet Loss, Arcface, etc..)

@nttstar @PistonY Thanks.
So distance based loss like contrastive loss and triplet loss (without L2 normalization)is not available in this challenge, right?
If we use L2 normalization, L2 distance and cosine is equal.
But if we don't use L2 normalization, they are different.

If evaluation metric is simple L2 distance, both of distance based loss and angular based loss is available.

@believeitmyway We should always compare l2 normed embeddings, for all algorithms you have metioned, contrastive loss, triplet loss, arcface. So no problem of cosine evaluation metric.

@nttstar Recent methods perform L2 normalization as you say.
But I do not think that it is essential for the future.
When targeting a metric space different from the conventional method, L2 normalization may not be essential.
But if evaluation metric has already been decided, I will follow it.
Thank you.

@believeitmyway Yes L2 distance metric is more general. We will use cosine this time, as we already have many submissions, Thank you for your advice!

@believeitmyway you just need to set embedding layer's bias=False and pass L2 normalization, it should be ok with cosine distance

Has anyone validated the flops of r100? I just got 12.1GFLOPS.

@nttstar would it be possible to upload baselines in Dropbox? Thanks.

Hi! My training speed is about 530 samples/sec. However, the speed recorded in the log is about 700 samples/sec. How can I speed up my training?

Thank you for interesting competition!

(1) All participants must use this dataset for training without any modification (e.g. re-alignment or changing image size are both prohibited).

I'd like to confirm the above rule.
It means that ALL data augmentation including random crop or contrast change in training are prohibited, right?
How about normalization?

@yu4u The alignment method should be kept, no change. But color jittering is allowed. What do you mean by normalization?

@nttstar Thank you for immediate reply!

But color jittering is allowed

Oh, I see.
Sorry, I meant standardization in terms of mean and variance by normalization.
This of course should be allowed as color jittering is allowed (and seems to be done in reference code)

Changing input image size is prohibited?

@PistonY Training and testing images should not be changed like re-alignment or something else. In training time, data augmentation methods are allowed(random flipping, color jittering). In testing time, only flipping test is allowed.

@nttstar

Assuming we do distillation or domain shift. Are the following allowed?

1) train a teacher network using train. Treat test data as unlabeled, and use teacher to distill 'train+test' to student network.

2) Treat test data as unlabeled target domain and train as labeled source domain. Learn features that minimize "feature difference" of source and target domain

@hengck23 (1) All participants must use this dataset for training without any modification (e.g. re-alignment or changing image size are both prohibited).
(2) Any external dataset is not allowed.
I don't think it's allowed.

@hengck23 Test data is not allowed to be used in any training process.

@nttstar In this repo, FLOPS are counted as a base of 1000, not 1024, so, there might be some models of 1.03G(base 1000) FLOPS, but it is not really 1.03G(base 1024).

@nttstar hi, how can i get a validation dataset? Splitting from the training datasets or use lfw datasets?

@mariolew Not a big problem, we will validate all top submissions after ddl.

@cbstyle LFW, CFP, AgeDB in training dataset.

@nttstar Why it has to multiply by 2 in the calculation of FLOPs?
image

@MrGF Same as opencv dnn

Hi, I have successfully submitted one result for "deepglint-light" to the server, but I still cannot get any result shown from the webpage. How long it will take to provide the performance?

image
We have created a discussion group for the competition and welcome everyone to join us.

Hi! I use my model to extract features and submit it on the test server. The acc is 0. However, my model's performance on LFW is normal. I have also tried to submit the features to the official trillionpair's website and the result is also correct. I can not figure out the possible reason for that. My file is in the binary format and the data is 1,862,120 * 256 in float32.
Screenshot from 2019-05-27 16-58-02
Screenshot from 2019-05-27 16-57-50

@jingyi1997 I'm not quite sure about your question. But many others can get correct score by 256 or 512 embeddings.

@nttstar Can I train models with the IQIYI_DIV_TRAIN for the video test?

@cham-3 no

The rules says Modification (e.g. re-alignment or resize) on test images is not allowed. May I ask is it permitted to use deconvolution or upsampling to increase the resolution of the features or even the input images? Upsampling seems to be an essential part of high resolution net. Or is there some restrictions on how to use this operations?

@luzai Do not resize on images. You can use upsampling or deconvolution but make sure to add the FLOPs of deconvolution operator which is not contained in current FLOPs counter.

For PyTorch users, you can use flops-counter.

However note that it outputs MACs not FLOPs - so you need to have model less than 500 MMac

DEL

@nttstar where is video_filelist.txt?

@xizi in testdata-image.zip

@nttstar ok.

We have created a wechat group for Chinese discussion:
Click for QR code: https://github.com/deepinsight/insightface/blob/master/resources/lfr19_wechat1.jpg

Any idea for where we can build a chat group for English?

@nttstar Maybe whatsapp?

@grib0ed0v Thanks, but it's a little bit complex for me to download whatsapp app..

My uploads are terminating after some time saying Upload error. Please refresh and try again. Any idea why this is happening?

@sarathknv Hi, sorry for the inconvenience. Have you tried again? Where are you(country and city)?

@sarathknv You can try to logout and re-login before submitting the results.

Is there a recommended way to measure FLOPs in tensorflow? I saw that your sample code is for mxnet. Is that equivalent of
opts = tf.profiler.ProfileOptionBuilder.float_operation()
flops = tf.profiler.profile(graph, run_meta=run_meta, cmd='op', options=opts)

@nttstar Hi, I have a question.
How do you make sure the participants strictly follow the rules such as no extra data and no modification on the data?
You've mentioned above that you'll make sure the leading results are reproducible, so how to do this?
Should the participants upload model and training code?

@gehaocool We will make sure about this and let you all know about it. No worries.

@nttstar Hi, is it allowed to use less training data than the full dataset? e.g., use only 50% of the training data. Thank you!

Hi @nttstar, I have several questions.

  1. How can we obtain raw 112x112 face images extracted by RetinaFace? I'm using Tensorflow to perform training and validation, so I need to access to raw images.

  2. As @alphanerdj asked, is there standard FLOPs calculation function for Tensorflow (which is similar to current MXNet-based function)

Thank you.

@vitoralbiero It is allowed. But you can not use external datasets/models to determine the subset selection.

@caocuong0306

  1. Use https://github.com/deepinsight/insightface/blob/master/recognition/data/rec2image.py to unpack images from rec
  2. You need to write tensorflow FLOPs counter by yourself.

@nttstar ,

  1. I did unpack training images from .rec files, but validation sets (LFW, agedb, cfp) were packed in .bin files.

Can you provided direct access to raw images for validation purpose?

Thank you.

@caocuong0306 You may refer to the 1:1 verification code to see how it be loaded. Then you will know how to unpack it.

@nttstar Is flops-counter approved by official for calculating FLOPs value which is just half of Mac value?

@paulcx Please refer to our FLOPs counter.

We have updated the ground-truth of glint dataset and all incoming submission will get the updated accuracy. We will also make update on top existing submissions soon.

@nttstar I found very strange result from the new light image ranking list.
yajundang_test1's model name is exactly same as my image feature name of 6/13 ('20190613102605_l5_dw_conv6_fc512_image47').
I don't know how it is possible.
I didn't follow public naming rule so it can be same by accident and the first part of name couldn't be manually given.
Can you explain it?

@sybyun we will check it asap, all feature files are kept, dont worry

@sybyun yujundang is our test account to submit existing features, it should be no problem.

I've been trying to upload feature file multiple times since yesterday but the test server is always stuck at 1% upload. I'm in the US. Can you help check that upload is working?

@alphanerdj Can you try WIN/MAC + Chrome to submit the feature? And try using another computer if it doesnot work.

It started working, thanks!

LB scores are delaying as too many submissions, please wait.

Can we use your baseline model as a pre-trained model?
I'd like to try finetuning of the model with different losses or parameters.

@believeitmyway Yes you can. But the pretrained models from our ArcFace paper are not allowed.

Thanks!

@nttstar please update the Chinese wechat QR code , thanks

We will accept the solutions with FLOPs<=1024^3 and model-size<=20MB for light tracks.
For large tracks: FLOPs<=30*1024^3.

What about model size?

@alphanerdj comment updated.

funding pool updated: now $17000

Hi! I use my model to extract features and submit it on the test server. The acc is 0. However, my model's performance on LFW is normal. I have also tried to submit the features to the official trillionpair's website and the result is also correct. I can not figure out the possible reason for that. My file is in the binary format and the data is 1,862,120 * 256 in float32.
Screenshot from 2019-05-27 16-58-02
Screenshot from 2019-05-27 16-57-50

@jingyi1997 I have encountered the same problem. Were you able to understand the problem and solve it?

Should imageset and video set features be generated using same model to be considered in final ranking?

@alphanerdj Using different models is allowed.

Hello! I tried to download the solution many times from work and from home wiht Windows and Linux, but the download does not go beyond 1%, can you fix it before the deadline?

@nttstar same problem as @boomberung has
Ubuntu 18.04 + Chrome
Could you, please, help me? Or can deadline be postponed?

@boomberung You mean submit?

@boomberung @ms-lilibeth You can send your feature files to insightface.[email protected], before deadline, with your username and password. One for each track at most.

@nttstar, Can we use any cloud platform for upload feature files and attach the link to the mail? Or can we send md5 of feature file and upload after deadline?

@luzai Yes you can use google drive or dropbox, before DDL only. I think there is no link problem for Chinese participants.

Game closed. We will send emails to top participants.

We have sent final check emails to top 20 participants. If you do not receive it, please contact us.

@nttstar Can the test server still be used for the purpose of evaluation after competition is over?
Thanks,

@nttstar I need to do a practical work in camera.
ms1m-retinaface is aligned by retina, the align algorithm is same as '/src/common/face_preprocess.py' ?
Thank for your help.

@nttstar Are you going to provide online video translation of the сhallenge on Monday? Thanks!

@borisgribkov hmm...no

I can post some pictures here if you need.

bad news, would be nice to watch and hear about winner solutions.... that is why I asked...
about a month ago I asked ICCV organizers the same question about online YouTube translation and they replied: "yes" ( and we decided not to fly to Seoul :) ), seems like it was about conference talks not about workshops and tutorials.

@borisgribkov Maybe I will open a stream at http://live.bilibili.com/13394860 on next Monday, but no translation.

Thanks a lot! Regarding language, will it be in Chinese? ;)

Ah, it should be English.

Great! Thanks!

Hi @nttstar ! We are idiots, because of time shift we missed it :(((( Maybe you or someone record something? Thanks a lot....

@versus-x Thanks a lot!

@nttstar Are you going to see any release for lightweight models ?

@nttstar
where can i see the published papers in this workshop?

Hi @nttstar, do you have the training logs for the baselines provided in the paper? If so, could you gently upload them in dropbox or google drive please?

@nttstar HI, I want to test my modle under the framework of this challenge but it seems that the test server doesn't work now. How can I workout the modle performance offline? Can I get the result by following the training and testing rules in the front page without the server?

Was this page helpful?
0 / 5 - 0 ratings