Darknet: periodic nan

[z1b1y]

yolo-plane
Learning Rate: 0.001, Momentum: 0.9, Decay: 0.0005
Loaded: 0.080210 seconds
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.416674, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.495442, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.061872, Class: 0.762901, Obj: 0.756638, No Obj: 0.533884, .5R: 0.000000, .75R: 0.000000, count: 14
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.417603, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.495068, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.045231, Class: 0.805219, Obj: 0.701775, No Obj: 0.532683, .5R: 0.038462, .75R: 0.000000, count: 26
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.415004, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.494662, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.155087, Class: 0.677625, Obj: 0.654264, No Obj: 0.538743, .5R: 0.086957, .75R: 0.000000, count: 23
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.415762, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.496444, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.124388, Class: 0.723703, Obj: 0.534797, No Obj: 0.540116, .5R: 0.000000, .75R: 0.000000, count: 23
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.416948, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.495494, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.122755, Class: 0.643624, Obj: 0.424036, No Obj: 0.539240, .5R: 0.037037, .75R: 0.000000, count: 27
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.415624, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.496122, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.078515, Class: 0.729269, Obj: 0.613455, No Obj: 0.538039, .5R: 0.045455, .75R: 0.000000, count: 22
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.418147, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.496003, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.194639, Class: 0.699301, Obj: 0.640963, No Obj: 0.539157, .5R: 0.105263, .75R: 0.000000, count: 19
Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.418285, .5R: -nan, .75R: -nan, count: 0
Region 94 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.494792, .5R: -nan, .75R: -nan, count: 0
Region 106 Avg IOU: 0.047548, Class: 0.708768, Obj: 0.535864, No Obj: 0.535721, .5R: 0.000000, .75R: 0.000000, count: 17

[pjreddie]

You must be training on a lot of small objects! nan's appear when there are no objects in a batch of images since i definitely divide by zero. For example, Avg IOU is the sum of IOUs for all objects at that level / # of objects, if that is zero you get nan. I could probably change this so it just does a check for zero 1st, just wasn't a priority.

[z1b1y]

thx

[EricWang-ML]

I changed bach_size from 64 to 128 resolved this issue.

[aedirn]

You must be training on a lot of small objects! nan's appear when there are no objects in a batch of images since i definitely divide by zero. For example, Avg IOU is the sum of IOUs for all objects at that level / # of objects, if that is zero you get nan. I could probably change this so it just does a check for zero 1st, just wasn't a priority.

Hi Joseph, @pjreddie , I am using another version (PyTorch) of YOLOv3 that still has this fundamental flaw. The way it manifests itself, loss functions from torch.nn library get fed tensors with one dimension equal to zero.
When you detect that there are no bounding boxes, how would you calculate the loss?