Keras-yolo3: why is my loss so high ? and my val_loss is nan?

Created on 18 Dec 2018 · 20Comments · Source: qqwweee/keras-yolo3

hi guys,
i configurate my yolov3_1c.cfg with
Line 3: set batch=24, this means we will be using 24 images for every training step
Line 4: set subdivisions=8, the batch will be divided by 8 to decrease GPU VRAM requirements.
Line 603: set filters=(classes + 5)3 in our case filters=15
Line 610: set classes=1, the number of categories we want to detect
Line 689: set filters=(classes + 5)3 in our case filters=15
Line 696: set classes=1, the number of categories we want to detect
Line 776: set filters=(classes + 5)*3 in our case filters=15
Line 783: set classes=1, the number of categories we want to detect

and i get the following output, here you can see that my loss is very high and my val loss ist nan.
colab_vallossnann

i have read that i should to change my code referenced on this issue #79
but i downloaded the repository a few days ago
i have checked, that my code is updated with this reference #79
please help me :)
best regards

Source

phamngocthanhtrung

Most helpful comment

Same issue

Hazelsuko07 on 19 Dec 2018

👍5 😄1

All 20 comments

phamngocthanhtrung on 19 Dec 2018

Same issue

Hazelsuko07 on 19 Dec 2018

👍5 😄1

hi guys,
i configurate my yolov3_1c.cfg with
Line 3: set batch=24, this means we will be using 24 images for every training step
Line 4: set subdivisions=8, the batch will be divided by 8 to decrease GPU VRAM requirements.
Line 603: set filters=(classes + 5)3 in our case filters=15
Line 610: set classes=1, the number of categories we want to detect
Line 689: set filters=(classes + 5)3 in our case filters=15
Line 696: set classes=1, the number of categories we want to detect
Line 776: set filters=(classes + 5)*3 in our case filters=15
Line 783: set classes=1, the number of categories we want to detect

and i get the following output, here you can see that my loss is very high and my val loss ist nan.

i have read that i should to change my code referenced on this issue #79
but i downloaded the repository a few days ago
i have checked, that my code is updated with this reference #79
please help me :)
best regards

why your's batch_size is only 32 but per step is so slowly?

S601327412 on 11 Jan 2019

I trained the model without pretrain weights to solved this issue.

younghuvee on 26 Mar 2019

Why the loss is so big? Same issue. The loss is right?

sxyxf66 on 17 May 2019

I trained the model without pretrain weights to solved this issue.

How did you do this? Can you explain? I'm currently having this issue as well

remcova on 13 Jun 2019

@phamngocthanhtrung have you resolved the problem yet?

vuthede on 4 Jul 2019

Guys I recommend to reduce your batch size if you get a crazy high loss. Batch size 32 is pretty big and not suitable for some projects.

remcova on 4 Jul 2019

thanks your suggestion @remcova .
Actually, now I try to train with only one sample with batchsize =1 to see if the model can overfit this sample or not. But the train_loss is converged at ~ 5.00, I am trying to manually reduce the learning rate but it didn't help.
Am I right to say the train loss should be in range [0,1]
Do you have any idea?

vuthede on 4 Jul 2019

@vuthede Makes sense but I'm not sure if it's correct. It's weird that the train_loss is converged at 5.00. If you want to know how well your model performs, I would recommend to use other metrics for example the F-measure (F1-score). That's what i'm using currently. It would be nice if @qqwweee takes a look at this issue.

remcova on 4 Jul 2019

I posted my solution in here. Hope it would help
https://github.com/qqwweee/keras-yolo3/issues/171#issuecomment-522287613