Pytorch-yolov3: Test and eval are not right! model(x) in eval mode output all nan.

[beckhamchen]

The training stage seems good. But when detect and test, the output of model(x) are all nan even after the first conv layer? Anybody know why?

[Aslizy]

I have the same problem.

[zwbdmm]

I have the same issue, the detect.py doesn't work on the newly trained net. output 0s, through the train stage looks good.

[jclevesque]

The save_weights and load_weights functions are not saving/loading all the weights. Some are left uninitialized, which most likely causes a division by zero somewhere. A workaround would be to save and load the state_dict instead of manually going through layers. E.g. torch.save(model.state_dict(), "model.weights") and model.load_state_dict(torch.load("model.weights")).

[kabrapankaj32]

thanks for the suggestion @jclevesque, I checked this works!

[selimarikan]

Thanks @jclevesque, it works in PyTorch 1.0. I wonder how it worked with load_weights at the first place. Maybe only on PyTorch 0.4.

[eriklindernoren]

Great observation @jclevesque . I have changed to saving and loading state dicts instead in master.This seems to solve the issue with checkpoint models producing junk when evaluated. There is probably an issue in Darknet.save_weights.

[SKRohit]

@eriklindernoren I am getting nan values in outputs during evaluation even with pre-trained yolo weights without any finetuning. Is it because I am trying to detect objects which are not in MS COCO dataset?

[SKRohit]

@eriklindernoren I could solve this issue by running my model in train mode. I have this issue only in eval mode. But could you please suggest why is it happening?