Keras-yolo3: nan in loss

[maiminh1996]

@qqwweee I don't know why the loss become nan when i trained on total parameter and i got this result when i trained in 3 last parameters!
Can you help me fix this problem, pls!

[xm1112]

@maiminh1996 Have you solved the same problem? Expected reply

[mbalty]

I also have the same problem

[wtx8887]

I also have the same problem, val_loss=Nan. Expected reply

[AGrosserHH]

Any solution to this?

[datduonguva]

I have a solution here that works for me. Basically if the box's width or height is too small, after resizing to size or 416x416, either or both box's width and height become 0. If this is the case, the loss function will fail at this step:

def yolo_loss(args, anchors, num_classes, ignore_thresh=.2, print_loss=False):
...
raw_true_wh = K.log(y_true[l][..., 2:4] / anchors[anchor_mask[l]] * input_shape[::-1])
raw_true_wh = K.switch(object_mask, raw_true_wh, K.zeros_like(raw_true_wh)) # avoid log(0)=-inf

due to the fact that the switch was intended to avoid where there is no box, not where the box was accidentally removed.
To fix this, simply adding a tiny term to the K.log function:

def yolo_loss(args, anchors, num_classes, ignore_thresh=.2, print_loss=False):
...
raw_true_wh = K.log(y_true[l][..., 2:4] / anchors[anchor_mask[l]] * input_shape[::-1] + 1e-10)
raw_true_wh = K.switch(object_mask, raw_true_wh, K.zeros_like(raw_true_wh)) # avoid log(0)=-inf

which won't result in any performance changes.