Darknet: Why I get "-nan" when trainning ??

[khiemntu]

0|yolo     | Region 94 Avg IOU: 0.862941, Class: 0.999465, Obj: 0.999374, No Obj: 0.008375, .5R: 1.000000, .75R: 1.000000,  count: 32
0|yolo     | Region 106 Avg IOU: 0.813918, Class: 0.999045, Obj: 0.999629, No Obj: 0.000558, .5R: 1.000000, .75R: 0.866667,  count: 15
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0
0|yolo     | Region 94 Avg IOU: 0.874563, Class: 0.999423, Obj: 0.999260, No Obj: 0.007178, .5R: 1.000000, .75R: 1.000000,  count: 17
0|yolo     | Region 106 Avg IOU: 0.872806, Class: 0.999047, Obj: 0.999295, No Obj: 0.002211, .5R: 1.000000, .75R: 0.953488,  count: 43
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0
0|yolo     | Region 94 Avg IOU: 0.889933, Class: 0.995632, Obj: 0.999651, No Obj: 0.003007, .5R: 1.000000, .75R: 1.000000,  count: 11
0|yolo     | Region 106 Avg IOU: 0.727005, Class: 0.969984, Obj: 0.975210, No Obj: 0.001956, .5R: 0.829268, .75R: 0.634146,  count: 41
0|yolo     | 10313: 0.576849, 0.677636 avg, 0.002000 rate, 3.992994 seconds, 1320064 images
0|yolo     | Loaded: 0.000031 seconds
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0
0|yolo     | Region 94 Avg IOU: 0.822985, Class: 0.999521, Obj: 0.999227, No Obj: 0.009055, .5R: 1.000000, .75R: 0.941176,  count: 34
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0
0|yolo     | Region 106 Avg IOU: 0.806361, Class: 0.966724, Obj: 0.994280, No Obj: 0.001520, .5R: 0.961538, .75R: 0.807692,  count: 26
0|yolo     | Region 94 Avg IOU: 0.893847, Class: 0.999776, Obj: 0.999865, No Obj: 0.004611, .5R: 1.000000, .75R: 1.000000,  count: 24
0|yolo     | Region 106 Avg IOU: 0.809407, Class: 0.998615, Obj: 0.999211, No Obj: 0.002214, .5R: 1.000000, .75R: 0.750000,  count: 36
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0
0|yolo     | Region 94 Avg IOU: 0.810882, Class: 0.999915, Obj: 0.999428, No Obj: 0.002502, .5R: 1.000000, .75R: 1.000000,  count: 9
0|yolo     | Region 106 Avg IOU: 0.756235, Class: 0.937499, Obj: 0.978064, No Obj: 0.002279, .5R: 0.921569, .75R: 0.745098,  count: 51
0|yolo     | Region 82 Avg IOU: -nan, Class: -nan, Obj: -nan, No Obj: 0.000001, .5R: -nan, .75R: -nan,  count: 0

I got many -nan in log when trainning yolov3. Is this normal ?

[AlexeyAB]

Training is going well.

https://github.com/AlexeyAB/darknet/issues/636#issuecomment-381400954
Only if nan occurs for avg loss for several dozen consecutive iterations, then training went wrong. Otherwise, the training goes well.

[ForeverStrongCheng]

@AlexeyAB
Thank you very much.

[Richard-Bebin]

@AlexeyAB , I am getting avg loss as nan after 5000 iterations while training tiny yolo with custom dataset. Could you please help me to resolve the issue.

[frincyc]

I have been getting good values until iteration 41, when all of a sudden avg loss started becoming nan and then everything became nan. What can be the reason?