Yet-another-efficientdet-pytorch: custom dataset failure

[guitar9]

Hello I want to detect mushrooms so I downloaded from Google Open Images V6 the mushroom dataset and converted it with OIDv4 toolkit to Pascal VOC. I wrote then a small script to convert the Pascal VOC Format to COCO JSON Format.
Here is the COCO JSON Mushroom dataset.
Google Drive linkIt contains 758 train images and 32 validation images.
I followed the steps in train_shape.ipynb and trained the model with this parameter
python train.py -c 0 -p mushroom --head_only True --lr 1e-3 --batch_size 32 --load_weights weights/efficientdet-d7.pth --num_epochs 100 --save_interval 100

I used efficientdet-d7 pretrained weights but in my logs folder the model is named efficientdet-d0_99_2300.pth? Why is the name efficientdet-d0?

this is my project.yaml

project_name: mushroom  # also the folder name of the dataset that under data_path folder
train_set: train
val_set: val
num_gpus: 1

# mean and std in RGB order, actually this part should remain unchanged as long as your dataset is similar to coco.
mean: [0.485, 0.456, 0.406]
std: [0.229, 0.224, 0.225]

# this anchor is adapted to the dataset
anchors_scales: '[2 ** 0, 2 ** (1.0 / 3.0), 2 ** (2.0 / 3.0)]'
anchors_ratios: '[(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)]'

obj_list: ['mushroom']

However i got very bad results.
This is an example prediciton:

I

Does somebody know what I am doing wrong?

[MGudik]

Hello.
It is because you specify your compound coefficient to d0.

python train.py -c 0

Change to:

python train.py -c 7

[zylo117]

For simple dataset like this, d0 should be enough

[jinfagang]

@guitar9 did u got reasonable results after take @MGudik advises?

[guitar9]

I tried with d0 and batch_size 32, head only true lr 1e-3 and 100 epoch. But the result is not good.

With d7 and batch size 32, head only true lr 1e-3 and 100 epoch i got error:
CUDA out of memory. Tried to allocate 3.94 GiB (GPU 0; 10.92 GiB total capacity; 8.91 GiB already allocated; 1.42 GiB free; 8.94 GiB reserved in total by PyTorch)

I have a geforce gtx 1080 ti 11gb.

[jinfagang]

@guitar9 AFAIK, all efficientdet implementation all not fully fullfill custom dataset training (some have bugs inside, some doesn't take care special cases in custom dataset). This is far more better when comes to YoloV5 which gives us a better robustness training framework.

You can have a try, I have using Yolov5 trained a faster and accurate detector.

[MGudik]

I got decent results on my custom dataset, you are trying to allocate 10 gigs of data on you GPU. Try lower batch size, furthermore, I got better results by training all the network, not just head only.
Worked for me, my regression were far better this way.

[GiangHLe]

Hi, I met the same problem as @guitar9, after training the model, the predicted box usually moves to the corner of the image.

My solution right now is comment out model.eval() in the inference step, even I knew that was wrong, but it brought the better result. Therefore, I could consider that the problem comes from batch norm parameters while learning (running mean and running std)

I am not sure it is because of me or because of the training code or the new batch norm way that they introduce in their code (compute mean and std on all data then spit back to each GPUs if I understand correctly). Does anyone have the same problem?

[zylo117]

@TobyCurtis149 can you share your training command?

[GiangHLe]

Thank you for your fast response,
I modified it a lot, even the training file, dataset, or utils so maybe it is my fault. I will check again and ask later.

[thhung]

@TobyCurtis149 Do you have any update?

[GiangHLe]

@thhung You met the same problem?
I am trying to modify the number of channels of input, so I have to write a new dataset to load it and modify the number of classes in the classifier head. And the result will return some ridiculous bbox as I described above,
Because of the emergency of my project, I temporary switch to mmdetection, I will come back to fix the problem in this repo latter