Keras-yolo3: retrain model with VOC dataset, but can't find box in yolo.py

[cooli7wa]

Thank you for this great model.

I'm new in yolo, I retrained this model with VOC dataset and pretrained darknet53_weights.
After 30 epoches, I got a trained_weights.h5 with about 39.0 loss.
In my training, I modified config in train.py to auto save model in h5, as follow:

    checkpoint = ModelCheckpoint(log_dir + "ep{epoch:03d}-loss{loss:.3f}-val_loss{val_loss:.3f}.h5",
        monitor='val_loss', save_weights_only=False, save_best_only=True)

So I just modified model_path and classes_path in yolo.py.
But using python yolo.py, my model can't find any box in my test car picture(yolo.h5 can find cars)

...
model_data/trained_weights.h5 model, anchors, and classes loaded.
Input image filename:/home/cooli7wa/Desktop/car/1.jpg
(416, 416, 3)
Found 0 boxes for img
2.1818076580530033

After this, I modified score from 0.3 to 0.0, to see all boxes.

Found 400 boxes for img
tvmonitor 0.24 (1078, 716) (1440, 1030)
tvmonitor 0.24 (854, 52) (1249, 365)
tvmonitor 0.24 (565, 0) (1096, 441)
tvmonitor 0.24 (1010, 411) (1440, 1080)
tvmonitor 0.24 (300, 164) (696, 475)
tvmonitor 0.24 (233, 0) (763, 441)
tvmonitor 0.24 (743, 605) (1139, 919)
tvmonitor 0.24 (632, 716) (1029, 1029)
tvmonitor 0.24 (856, 717) (1248, 1030)
tvmonitor 0.24 (852, 937) (1251, 1080)
tvmonitor 0.24 (187, 938) (587, 1080)
tvmonitor 0.24 (564, 524) (1097, 1080)
tvmonitor 0.24 (1075, 939) (1440, 1080)
tvmonitor 0.24 (299, 272) (696, 587)
tvmonitor 0.24 (411, 52) (806, 365)
tvmonitor 0.24 (522, 162) (918, 476)
tvmonitor 0.24 (744, 495) (1138, 808)
tvmonitor 0.24 (630, 938) (1030, 1080)
tvmonitor 0.25 (190, 53) (584, 365)
tvmonitor 0.26 (79, 162) (473, 477)
train 0.24 (854, 52) (1249, 365)
train 0.24 (564, 415) (1097, 1080)
train 0.24 (79, 275) (472, 586)
train 0.24 (852, 828) (1251, 1080)
train 0.24 (786, 0) (1318, 666)
train 0.24 (343, 415) (875, 1080)
train 0.24 (787, 529) (1317, 1080)
train 0.24 (119, 750) (655, 1080)
train 0.24 (522, 162) (918, 476)
train 0.24 (854, 605) (1248, 920)
train 0.24 (563, 749) (1099, 1080)
train 0.24 (856, 717) (1248, 1030)
train 0.24 (79, 162) (473, 477)
train 0.24 (455, 195) (984, 886)
train 0.24 (342, 0) (875, 554)
train 0.24 (80, 494) (471, 808)
train 0.24 (854, 163) (1250, 476)
train 0.24 (0, 83) (431, 776)
train 0.24 (676, 303) (1206, 998)
train 0.25 (14, 417) (539, 1080)
sofa 0.25 (521, 606) (918, 919)
sofa 0.25 (565, 0) (1096, 441)
sofa 0.25 (0, 163) (360, 477)
sofa 0.25 (232, 195) (764, 884)
sofa 0.25 (565, 304) (1095, 998)
sofa 0.25 (233, 0) (763, 441)
sofa 0.25 (0, 0) (319, 671)
sofa 0.25 (1011, 524) (1440, 1080)
sofa 0.25 (676, 194) (1206, 886)
sofa 0.25 (898, 0) (1427, 554)
sofa 0.25 (232, 639) (765, 1080)
sofa 0.25 (523, 826) (915, 1080)
sofa 0.25 (342, 0) (875, 554)
sofa 0.25 (455, 195) (984, 886)
sofa 0.25 (452, 640) (987, 1080)
sofa 0.25 (9, 644) (544, 1080)
sofa 0.25 (0, 0) (429, 445)
sofa 0.25 (784, 642) (1321, 1080)
sofa 0.26 (120, 0) (655, 549)
sofa 0.26 (78, 826) (474, 1080)
sheep 0.25 (521, 383) (918, 699)
sheep 0.25 (898, 0) (1427, 554)
sheep 0.25 (300, 383) (695, 698)
sheep 0.25 (522, 162) (918, 476)
sheep 0.25 (852, 828) (1251, 1080)
sheep 0.25 (521, 273) (918, 587)
sheep 0.25 (0, 749) (429, 1080)
sheep 0.25 (523, 495) (916, 809)
sheep 0.26 (455, 195) (984, 886)
sheep 0.26 (301, 717) (696, 1029)
sheep 0.26 (963, 162) (1362, 476)
sheep 0.26 (0, 0) (430, 667)
sheep 0.26 (744, 495) (1138, 808)
sheep 0.26 (854, 605) (1248, 920)
sheep 0.26 (299, 272) (696, 587)
sheep 0.26 (140, 350) (246, 563)
sheep 0.26 (740, 938) (1142, 1080)
sheep 0.26 (232, 195) (764, 884)
sheep 0.26 (0, 163) (360, 477)
sheep 0.26 (521, 606) (918, 919)
pottedplant 0.24 (786, 82) (1317, 777)
pottedplant 0.24 (1076, 606) (1440, 919)
pottedplant 0.24 (233, 0) (763, 441)
pottedplant 0.24 (675, 0) (1208, 554)
pottedplant 0.24 (676, 303) (1206, 998)
pottedplant 0.24 (13, 530) (540, 1080)
pottedplant 0.24 (452, 415) (987, 1080)
pottedplant 0.24 (12, 0) (540, 440)
pottedplant 0.24 (1010, 411) (1440, 1080)
pottedplant 0.24 (898, 0) (1428, 666)
pottedplant 0.24 (229, 84) (767, 774)
pottedplant 0.24 (0, 637) (427, 1080)
pottedplant 0.24 (564, 524) (1097, 1080)
pottedplant 0.24 (743, 605) (1139, 919)
pottedplant 0.24 (0, 0) (429, 330)
pottedplant 0.24 (565, 194) (1095, 886)
pottedplant 0.24 (0, 83) (431, 776)
pottedplant 0.25 (124, 194) (652, 885)
pottedplant 0.25 (344, 192) (873, 888)
pottedplant 0.25 (235, 526) (762, 1080)
person 0.26 (343, 415) (875, 1080)
person 0.26 (343, 0) (874, 440)
person 0.26 (232, 305) (764, 996)
person 0.26 (966, 494) (1361, 809)
person 0.26 (522, 162) (918, 476)
person 0.26 (854, 0) (1249, 253)
person 0.26 (898, 0) (1427, 554)
person 0.26 (454, 0) (986, 555)
person 0.26 (565, 0) (1096, 441)
person 0.26 (1145, 188) (1348, 614)
person 0.26 (896, 303) (1429, 998)
person 0.26 (78, 826) (474, 1080)
person 0.26 (457, 303) (983, 999)
person 0.27 (121, 418) (653, 1080)
person 0.27 (675, 0) (1208, 554)
person 0.27 (120, 0) (655, 549)
person 0.27 (676, 194) (1206, 886)
person 0.27 (12, 0) (540, 440)
person 0.27 (232, 0) (764, 663)
person 0.27 (787, 0) (1317, 440)
motorbike 0.24 (898, 0) (1427, 554)
motorbike 0.24 (966, 606) (1360, 920)
motorbike 0.24 (854, 163) (1250, 476)
motorbike 0.24 (343, 415) (875, 1080)
motorbike 0.24 (521, 52) (918, 366)
motorbike 0.24 (676, 303) (1206, 998)
motorbike 0.24 (412, 495) (806, 809)
motorbike 0.24 (1008, 639) (1440, 1080)
motorbike 0.24 (854, 52) (1249, 365)
motorbike 0.25 (232, 305) (764, 996)
motorbike 0.25 (0, 526) (428, 1080)
motorbike 0.25 (1077, 53) (1440, 365)
motorbike 0.25 (121, 530) (653, 1080)
motorbike 0.25 (455, 195) (984, 886)
motorbike 0.25 (522, 162) (918, 476)
motorbike 0.25 (521, 606) (918, 919)
motorbike 0.25 (454, 0) (986, 555)
motorbike 0.25 (14, 417) (539, 1080)
motorbike 0.25 (12, 0) (540, 553)
motorbike 0.26 (0, 83) (431, 776)
horse 0.25 (411, 52) (806, 365)
horse 0.25 (1010, 411) (1440, 1080)
horse 0.25 (854, 163) (1250, 476)
horse 0.25 (854, 52) (1249, 365)
horse 0.25 (522, 162) (918, 476)
horse 0.25 (0, 939) (364, 1080)
horse 0.25 (854, 0) (1249, 253)
horse 0.25 (851, 273) (1251, 587)
horse 0.25 (232, 305) (764, 996)
horse 0.25 (521, 273) (918, 587)
horse 0.25 (675, 0) (1208, 554)
horse 0.25 (452, 415) (987, 1080)
horse 0.25 (523, 495) (916, 809)
horse 0.25 (521, 383) (918, 699)
horse 0.25 (675, 413) (1207, 1080)
horse 0.25 (786, 0) (1318, 666)
horse 0.25 (344, 192) (873, 888)
horse 0.26 (453, 0) (986, 666)
horse 0.26 (14, 417) (539, 1080)
horse 0.26 (565, 194) (1095, 886)
dog 0.26 (674, 749) (1209, 1080)
dog 0.26 (631, 383) (1030, 699)
dog 0.26 (963, 937) (1362, 1080)
dog 0.26 (742, 162) (1139, 477)
dog 0.26 (676, 194) (1206, 886)
dog 0.26 (409, 937) (809, 1080)
dog 0.27 (744, 495) (1138, 808)
dog 0.27 (786, 82) (1317, 777)
dog 0.27 (78, 52) (474, 366)
dog 0.27 (521, 606) (918, 919)
dog 0.27 (299, 53) (696, 364)
dog 0.27 (78, 826) (474, 1080)
dog 0.27 (0, 163) (360, 477)
dog 0.27 (10, 750) (543, 1080)
dog 0.27 (191, 164) (583, 476)
dog 0.27 (77, 939) (476, 1080)
dog 0.27 (341, 85) (876, 775)
dog 0.27 (740, 938) (1142, 1080)
dog 0.27 (522, 162) (918, 476)
dog 0.27 (299, 272) (696, 587)
diningtable 0.25 (563, 749) (1099, 1080)
diningtable 0.25 (742, 162) (1139, 477)
diningtable 0.25 (342, 0) (875, 554)
diningtable 0.25 (674, 82) (1208, 776)
diningtable 0.25 (10, 750) (543, 1080)
diningtable 0.25 (411, 826) (807, 1080)
diningtable 0.25 (409, 937) (809, 1080)
diningtable 0.25 (785, 0) (1319, 553)
diningtable 0.25 (300, 383) (695, 698)
diningtable 0.25 (411, 52) (806, 365)
diningtable 0.25 (854, 52) (1249, 365)
diningtable 0.25 (963, 162) (1362, 476)
diningtable 0.25 (521, 273) (918, 587)
diningtable 0.25 (301, 495) (695, 808)
diningtable 0.25 (630, 938) (1030, 1080)
diningtable 0.25 (232, 305) (764, 996)
diningtable 0.25 (521, 606) (918, 919)
diningtable 0.26 (522, 162) (918, 476)
diningtable 0.26 (77, 939) (476, 1080)
diningtable 0.26 (79, 162) (473, 477)
cow 0.25 (410, 716) (807, 1030)
cow 0.25 (565, 0) (1096, 667)
cow 0.25 (966, 494) (1361, 809)
cow 0.25 (852, 828) (1251, 1080)
cow 0.25 (411, 52) (806, 365)
cow 0.25 (676, 303) (1206, 998)
cow 0.25 (856, 717) (1248, 1030)
cow 0.25 (78, 52) (474, 366)
cow 0.25 (631, 383) (1030, 699)
cow 0.25 (1078, 716) (1440, 1030)
cow 0.25 (1076, 606) (1440, 919)
cow 0.26 (0, 0) (430, 667)
cow 0.26 (633, 162) (1027, 477)
cow 0.26 (1077, 53) (1440, 365)
cow 0.26 (854, 52) (1249, 365)
cow 0.26 (854, 163) (1250, 476)
cow 0.26 (898, 0) (1428, 666)
cow 0.26 (744, 495) (1138, 808)
cow 0.26 (854, 605) (1248, 920)
cow 0.26 (631, 607) (1029, 918)
chair 0.24 (632, 716) (1029, 1029)
chair 0.24 (411, 383) (806, 698)
chair 0.24 (301, 495) (695, 808)
chair 0.24 (854, 163) (1250, 476)
chair 0.24 (1075, 939) (1440, 1080)
chair 0.24 (1076, 606) (1440, 919)
chair 0.24 (411, 826) (807, 1080)
chair 0.25 (523, 495) (916, 809)
chair 0.25 (410, 716) (807, 1030)
chair 0.25 (744, 495) (1138, 808)
chair 0.25 (630, 938) (1030, 1080)
chair 0.25 (965, 53) (1361, 365)
chair 0.25 (745, 825) (1138, 1080)
chair 0.25 (411, 52) (806, 365)
chair 0.25 (852, 383) (1251, 698)
chair 0.25 (0, 939) (364, 1080)
chair 0.25 (854, 605) (1248, 920)
chair 0.25 (521, 606) (918, 919)
chair 0.25 (1011, 524) (1440, 1080)
chair 0.26 (966, 494) (1361, 809)
cat 0.25 (563, 749) (1099, 1080)
cat 0.25 (1077, 53) (1440, 365)
cat 0.25 (521, 383) (918, 699)
cat 0.25 (521, 0) (918, 141)
cat 0.25 (854, 605) (1248, 920)
cat 0.25 (520, 937) (919, 1080)
cat 0.25 (0, 0) (429, 445)
cat 0.25 (963, 937) (1362, 1080)
cat 0.25 (854, 52) (1249, 365)
cat 0.25 (410, 716) (807, 1030)
cat 0.25 (744, 495) (1138, 808)
cat 0.25 (522, 162) (918, 476)
cat 0.25 (521, 606) (918, 919)
cat 0.25 (852, 828) (1251, 1080)
cat 0.26 (190, 53) (584, 365)
cat 0.26 (632, 716) (1029, 1029)
cat 0.26 (740, 938) (1142, 1080)
cat 0.26 (77, 939) (476, 1080)
cat 0.26 (411, 52) (806, 365)
cat 0.26 (633, 827) (1027, 1080)
car 0.25 (565, 194) (1095, 886)
car 0.25 (232, 305) (764, 996)
car 0.25 (452, 415) (987, 1080)
car 0.26 (523, 495) (916, 809)
car 0.26 (785, 414) (1318, 1080)
car 0.26 (411, 52) (806, 365)
car 0.26 (966, 606) (1360, 920)
car 0.26 (299, 827) (697, 1080)
car 0.26 (965, 53) (1361, 365)
car 0.26 (301, 495) (695, 808)
car 0.26 (897, 191) (1428, 889)
car 0.26 (521, 606) (918, 919)
car 0.26 (80, 494) (471, 808)
car 0.26 (79, 162) (473, 477)
car 0.27 (124, 194) (652, 885)
car 0.27 (189, 275) (584, 585)
car 0.27 (76, 384) (476, 699)
car 0.28 (14, 417) (539, 1080)
car 0.28 (0, 83) (431, 776)
car 0.28 (0, 274) (364, 586)
bus 0.24 (521, 606) (918, 919)
bus 0.24 (189, 275) (584, 585)
bus 0.24 (565, 194) (1095, 886)
bus 0.24 (123, 0) (651, 442)
bus 0.24 (743, 605) (1139, 919)
bus 0.24 (189, 384) (585, 698)
bus 0.24 (854, 52) (1249, 365)
bus 0.24 (343, 415) (875, 1080)
bus 0.24 (676, 0) (1206, 666)
bus 0.24 (563, 749) (1099, 1080)
bus 0.24 (79, 162) (473, 477)
bus 0.25 (232, 305) (764, 996)
bus 0.25 (785, 0) (1319, 553)
bus 0.25 (785, 305) (1320, 997)
bus 0.25 (124, 194) (652, 885)
bus 0.25 (675, 413) (1207, 1080)
bus 0.25 (630, 938) (1030, 1080)
bus 0.25 (898, 412) (1427, 1080)
bus 0.25 (897, 80) (1428, 778)
bus 0.26 (13, 87) (539, 771)
bottle 0.25 (812, 298) (1016, 726)
bottle 0.25 (0, 0) (185, 173)
bottle 0.25 (591, 465) (795, 891)
bottle 0.25 (92, 413) (297, 832)
bottle 0.25 (916, 796) (1021, 1004)
bottle 0.25 (1090, 628) (1292, 1060)
bottle 0.25 (1138, 517) (1243, 730)
bottle 0.25 (370, 77) (572, 504)
bottle 0.25 (141, 793) (245, 1006)
bottle 0.25 (37, 298) (239, 725)
bottle 0.25 (1090, 357) (1294, 777)
bottle 0.25 (38, 133) (241, 558)
bottle 0.25 (758, 77) (960, 504)
bottle 0.25 (202, 356) (407, 779)
bottle 0.25 (258, 465) (463, 891)
bottle 0.26 (150, 469) (350, 887)
bottle 0.26 (1193, 463) (1298, 673)
bottle 0.26 (140, 350) (246, 563)
bottle 0.26 (204, 132) (406, 559)
bottle 0.27 (93, 247) (296, 666)
boat 0.23 (124, 194) (652, 885)
boat 0.23 (342, 0) (875, 554)
boat 0.23 (0, 0) (430, 667)
boat 0.23 (1117, 0) (1440, 668)
boat 0.23 (187, 497) (586, 806)
boat 0.23 (898, 412) (1427, 1080)
boat 0.23 (10, 0) (543, 329)
boat 0.23 (123, 0) (651, 442)
boat 0.23 (854, 163) (1250, 476)
boat 0.23 (1010, 0) (1440, 555)
boat 0.23 (674, 82) (1208, 776)
boat 0.23 (14, 417) (539, 1080)
boat 0.24 (785, 0) (1319, 553)
boat 0.24 (344, 192) (873, 888)
boat 0.24 (787, 192) (1317, 888)
boat 0.24 (1009, 191) (1440, 889)
boat 0.24 (565, 194) (1095, 886)
boat 0.24 (897, 80) (1428, 778)
boat 0.24 (1008, 0) (1440, 218)
boat 0.24 (13, 87) (539, 771)
bird 0.26 (0, 827) (363, 1080)
bird 0.26 (412, 495) (806, 809)
bird 0.26 (521, 52) (918, 366)
bird 0.26 (13, 0) (540, 663)
bird 0.26 (852, 828) (1251, 1080)
bird 0.26 (0, 274) (364, 586)
bird 0.26 (740, 938) (1142, 1080)
bird 0.26 (77, 939) (476, 1080)
bird 0.26 (0, 0) (362, 142)
bird 0.26 (0, 495) (362, 808)
bird 0.26 (521, 273) (918, 587)
bird 0.26 (411, 826) (807, 1080)
bird 0.26 (745, 52) (1138, 366)
bird 0.27 (631, 607) (1029, 918)
bird 0.27 (119, 87) (655, 771)
bird 0.27 (522, 162) (918, 476)
bird 0.27 (411, 604) (806, 920)
bird 0.27 (0, 382) (365, 699)
bird 0.27 (189, 275) (584, 585)
bird 0.28 (79, 162) (473, 477)
bicycle 0.25 (411, 272) (806, 587)
bicycle 0.25 (758, 77) (960, 504)
bicycle 0.25 (302, 605) (695, 918)
bicycle 0.25 (521, 606) (918, 919)
bicycle 0.25 (189, 827) (585, 1080)
bicycle 0.25 (740, 938) (1142, 1080)
bicycle 0.25 (78, 603) (473, 921)
bicycle 0.25 (411, 52) (806, 365)
bicycle 0.25 (631, 383) (1030, 699)
bicycle 0.25 (742, 162) (1139, 477)
bicycle 0.26 (0, 827) (363, 1080)
bicycle 0.26 (522, 162) (918, 476)
bicycle 0.26 (745, 52) (1138, 366)
bicycle 0.26 (523, 495) (916, 809)
bicycle 0.26 (0, 495) (362, 808)
bicycle 0.26 (744, 495) (1138, 808)
bicycle 0.26 (411, 383) (806, 698)
bicycle 0.26 (189, 384) (585, 698)
bicycle 0.26 (189, 275) (584, 585)
bicycle 0.26 (301, 495) (695, 808)
aeroplane 0.23 (631, 607) (1029, 918)
aeroplane 0.23 (1008, 639) (1440, 1080)
aeroplane 0.23 (784, 642) (1321, 1080)
aeroplane 0.23 (854, 0) (1250, 35)
aeroplane 0.23 (813, 630) (1016, 1058)
aeroplane 0.23 (13, 87) (539, 771)
aeroplane 0.23 (411, 52) (806, 365)
aeroplane 0.23 (342, 0) (875, 554)
aeroplane 0.24 (425, 576) (628, 1002)
aeroplane 0.24 (79, 0) (474, 34)
aeroplane 0.24 (1146, 741) (1347, 1080)
aeroplane 0.24 (0, 939) (364, 1080)
aeroplane 0.24 (0, 274) (364, 586)
aeroplane 0.24 (1090, 796) (1292, 1080)
aeroplane 0.24 (37, 191) (242, 612)
aeroplane 0.24 (813, 134) (1016, 558)
aeroplane 0.24 (79, 162) (473, 477)
aeroplane 0.24 (76, 384) (476, 699)
aeroplane 0.24 (189, 275) (584, 585)
aeroplane 0.25 (758, 77) (960, 504)

all class have almost the same score..
Other, I found my trained_weights.h5's size is not same as yolo.h5

-rw-rw-r--  1 cooli7wa cooli7wa 248491304 6月   4 15:33 trained_weights.h5
-rw-rw-r--  1 cooli7wa cooli7wa 248686632 6月   1 09:18 yolo.h5

Is that size ok?
Or my loss too high(39.0)?
I don't know why.

This blocked me two days.
Thank you for your help!

[AlphaRalph]

1. when(mAP, Recall, etc.) did you stop training the yolo in darknet?
2. did the same image work in darknet/ yolo?
3. did you use the default anchors?

It sounds to me, that the yolo wasn't trained enough in darknet and therefore 30 epochs in keras was to less. When EarlyStopping is active, it should stop when the validation loss isn't further decreasing. Did you observe that?

Size looks fine, i got ~247 MB

[cooli7wa]

@AlphaRalph
Hi, thank you.

1. I just use the default config(30 epoches), and I see the loss(39) is not large, so I think this maybe not good enough, but should work.
2. Yes, use yolo.h5, can find car in my test picture.
3. Yes, no change.

I don't observe EarlyStopping, just 30 epoches.
Can you tell me, how many epoches you trained? And what is the loss?
Thank you.

[AlphaRalph]

I fear you misunderstand me. You trained your yolo first of all in Dartknet(no yolo.h5)?
And then converted it using the convert.py from this repo? There you get the yolo.h5.
What where the final values when training in darknet?

My initial loss is about 26 and decreases during ~15 epochs to 24.8 loss and 25.1 val_loss
But since I don't have any reference I can't tell, whether these values are good or not. The only thing I can tell, is that EarlyStopping kills the process at 15 epochs.
Try training more epochs, like +50, and observe EarlyStopping.

[cooli7wa]

@AlphaRalph
Hi, thank you.
I download yolov3.cfg and yolov3.weights from pjreddie.com, no new train on them.
Then I use convert.py to generate yolo.h5 for this model.
This yolo.h5 works well on my test picture.

But I want to train this model by myself, so I follow readme.md:

    1. wget https://pjreddie.com/media/files/darknet53.conv.74  
    2. rename it as darknet53.weights  
    3. python convert.py -w darknet53.cfg darknet53.weights model_data/darknet53_weights.h5  
    4. use model_data/darknet53_weights.h5 in train.py

use pretrained darknet53.weights for classification, and use VOC dataset to train model to detect object.
After 30epoches, I got my trained_weights.h5 in log/000, but this h5 works bad.

I think you are right, I trained too less, I will try to train more epoches.

And I have a question:
Do you use pretrained yolo.h5 in your training ? And your model has the same classes (80) with yolo.h5?
So your initial loss is so small (I got about 2000 loss in first epoch)?

[AlphaRalph]

Ok, I see.
Unfortunately, I fear, is this train.py script rather for fine tuning the pretrained yolo and not for training the model from scratch(darknet53.weights are from scratch in this context).

You get that high loss, because the darknet53.weights aren't fit to your dataset yet.
My workflow:

• Train yolov3 in Darknet until the avg loss is pretty low (~7k iterations)
• convert to yolo.h5
• finetune with train.py
• enjoy ;-)

[cooli7wa]

@AlphaRalph

I know, thank you for your help.

Current training result is not good:

524/524 [==============================] - 830s 2s/step - loss: 473.6577 - val_loss: 4544.1667
Epoch 2/30
524/524 [==============================] - 896s 2s/step - loss: 59.1415 - val_loss: 4384.1684
Epoch 3/30
524/524 [==============================] - 920s 2s/step - loss: 45.0539 - val_loss: 4359.2531

val_loss is too high, when loss is low.
this is high varience?
Is this a problem?

[yqyao]

My pytorch implementation of yolov3 is also not good in voc-dataset. I've been stuck in it for days... It is too hard to train from scratch.

[AlphaRalph]

@cooli7wa
Don't measure the training process on the first few epochs.
Let it train with +70 epochs over night and see what happens.
What's your batch size/ amount of training images?

@yqyao
I don't have experience in pytorch but in the newest yolo paper they say, that they were able to improve training significantly.

My advice: training in darknet, finetuning in Keras/ pytorch, inference in Keras/ pytorch

[cooli7wa]

@AlphaRalph
Batch size default is 32, but in my computer cause OOM error, so I modify it to 10,
training set is about 6000.
When have result, I stick the results here.

Thank you for your advice and help.

[cooli7wa]

before current train, have already trained about 50epoches, current train is 30/70 epoches
as you say, @AlphaRalph , val_loss is going down when loss is always 31.xx. I don't know why.

524/524 [==============================] - 1029s 2s/step - loss: 31.3760 - val_loss: 937.2539
Epoch 21/70
524/524 [==============================] - 1030s 2s/step - loss: 31.3853 - val_loss: 914.7923
Epoch 22/70
524/524 [==============================] - 1029s 2s/step - loss: 31.3793 - val_loss: 865.4986
Epoch 23/70
524/524 [==============================] - 1028s 2s/step - loss: 31.2008 - val_loss: 845.0844
Epoch 24/70
524/524 [==============================] - 1029s 2s/step - loss: 31.3034 - val_loss: 804.9360
Epoch 25/70
524/524 [==============================] - 1029s 2s/step - loss: 31.3679 - val_loss: 790.2106
Epoch 26/70
524/524 [==============================] - 1029s 2s/step - loss: 31.4060 - val_loss: 747.5976
Epoch 27/70
524/524 [==============================] - 1029s 2s/step - loss: 31.4660 - val_loss: 732.0469
Epoch 28/70
524/524 [==============================] - 1030s 2s/step - loss: 31.4133 - val_loss: 698.6978
Epoch 29/70
524/524 [==============================] - 1080s 2s/step - loss: 31.1222 - val_loss: 676.6485
Epoch 30/70
524/524 [==============================] - 1091s 2s/step - loss: 31.4689 - val_loss: 646.6978

[cooli7wa]

I use the last h5 to detect car, but result is still 0 boxes, and all classes' scores are still almost the same.
I think maybe something is wrong.

tvmonitor 0.14 (0, 1698) (1997, 3120)
tvmonitor 0.14 (0, 146) (1098, 1051)
tvmonitor 0.14 (3057, 2718) (4160, 3120)
tvmonitor 0.14 (1143, 0) (2377, 84)
tvmonitor 0.14 (3075, 2073) (4160, 2960)
tvmonitor 0.14 (822, 0) (2061, 402)
tvmonitor 0.14 (0, 794) (778, 1681)
tvmonitor 0.14 (8, 149) (1587, 2334)
tvmonitor 0.14 (2247, 784) (3831, 2977)
tvmonitor 0.14 (0, 0) (1984, 3120)
tvmonitor 0.14 (2129, 0) (4160, 1365)
tvmonitor 0.14 (2117, 1732) (4160, 3120)
tvmonitor 0.14 (874, 1705) (4160, 3120)
tvmonitor 0.15 (16, 1099) (1578, 3120)
tvmonitor 0.15 (222, 0) (3935, 2363)
tvmonitor 0.15 (880, 398) (4160, 3120)
tvmonitor 0.15 (2126, 430) (4160, 3120)
tvmonitor 0.15 (0, 0) (3607, 3120)
tvmonitor 0.15 (0, 0) (3291, 1377)
tvmonitor 0.16 (876, 0) (4160, 1391)
train 0.14 (2108, 469) (3337, 1363)
train 0.14 (2585, 0) (4132, 1691)
train 0.14 (0, 0) (1984, 3120)
train 0.14 (3058, 156) (4160, 1036)
train 0.14 (2104, 0) (3338, 84)
train 0.14 (1465, 0) (2700, 401)
train 0.14 (504, 0) (1737, 85)
train 0.14 (876, 0) (4160, 1391)
train 0.14 (2741, 2399) (3985, 3120)
train 0.14 (3057, 2718) (4160, 3120)
train 0.14 (2419, 158) (3670, 1037)
train 0.15 (2128, 773) (4160, 3120)
train 0.15 (1473, 0) (4160, 2403)
train 0.15 (503, 0) (1739, 406)
train 0.15 (242, 1073) (3910, 3120)
train 0.15 (0, 1710) (2952, 3120)
train 0.15 (222, 0) (3935, 2363)
train 0.15 (1214, 85) (4160, 3120)
train 0.15 (1193, 1716) (4160, 3120)
train 0.16 (0, 0) (3607, 3120)
sofa 0.14 (16, 1099) (1578, 3120)
sofa 0.14 (0, 794) (778, 1681)
sofa 0.14 (0, 482) (786, 1359)
sofa 0.14 (3077, 1107) (4160, 2006)
sofa 0.14 (3371, 3033) (4160, 3120)
sofa 0.14 (3057, 2718) (4160, 3120)
sofa 0.14 (2902, 459) (4160, 2664)
sofa 0.14 (1449, 2705) (2709, 3120)
sofa 0.15 (0, 0) (2976, 1383)
sofa 0.15 (1186, 0) (4160, 1403)
sofa 0.15 (0, 1698) (1997, 3120)
sofa 0.15 (0, 0) (2645, 2697)
sofa 0.15 (0, 153) (776, 1039)
sofa 0.15 (1807, 1413) (4160, 3120)
sofa 0.15 (0, 424) (2002, 3120)
sofa 0.15 (2147, 0) (4160, 3120)
sofa 0.16 (0, 0) (3607, 3120)
sofa 0.16 (227, 0) (3927, 2050)
sofa 0.16 (242, 1073) (3910, 3120)
sofa 0.17 (1201, 421) (4160, 3120)
sheep 0.15 (824, 155) (2059, 1043)
sheep 0.15 (189, 800) (1416, 1683)
sheep 0.15 (1468, 153) (2694, 1045)
sheep 0.15 (508, 0) (1735, 729)
sheep 0.15 (3058, 156) (4160, 1036)
sheep 0.15 (2093, 3034) (3347, 3120)
sheep 0.15 (2413, 2715) (3673, 3120)
sheep 0.15 (822, 0) (2061, 402)
sheep 0.15 (903, 89) (4160, 3120)
sheep 0.15 (2431, 0) (3658, 726)
sheep 0.15 (2432, 466) (3651, 1358)
sheep 0.15 (1785, 472) (3019, 1361)
sheep 0.15 (2419, 158) (3670, 1037)
sheep 0.15 (0, 0) (1097, 730)
sheep 0.15 (0, 2084) (1103, 2961)
sheep 0.15 (1449, 2705) (2709, 3120)
sheep 0.15 (0, 146) (1098, 1051)
sheep 0.15 (1465, 0) (2700, 401)
sheep 0.15 (251, 1400) (3899, 3120)
sheep 0.15 (177, 0) (1427, 408)
pottedplant 0.15 (3076, 475) (4160, 1363)
pottedplant 0.15 (3075, 2073) (4160, 2960)
pottedplant 0.15 (1449, 2705) (2709, 3120)
pottedplant 0.15 (1143, 0) (2380, 399)
pottedplant 0.15 (503, 0) (1739, 406)
pottedplant 0.15 (2428, 2075) (3656, 2965)
pottedplant 0.15 (1143, 0) (2377, 84)
pottedplant 0.15 (1155, 0) (2370, 728)
pottedplant 0.15 (2106, 0) (3340, 726)
pottedplant 0.15 (0, 0) (1984, 3120)
pottedplant 0.15 (0, 1077) (3578, 3120)
pottedplant 0.15 (180, 0) (1414, 83)
pottedplant 0.15 (0, 0) (1095, 403)
pottedplant 0.15 (0, 1756) (1099, 2647)
pottedplant 0.16 (183, 2082) (1427, 2962)
pottedplant 0.16 (2126, 430) (4160, 3120)
pottedplant 0.16 (0, 0) (3610, 1391)
pottedplant 0.17 (0, 0) (3264, 2679)
pottedplant 0.17 (1186, 0) (4160, 1403)
pottedplant 0.17 (880, 398) (4160, 3120)
person 0.15 (2106, 0) (3340, 726)
person 0.15 (2099, 1105) (3345, 2008)
person 0.15 (829, 0) (2056, 728)
person 0.15 (1465, 0) (2700, 401)
person 0.15 (1145, 794) (2380, 1682)
person 0.15 (0, 0) (1990, 2373)
person 0.15 (3077, 1107) (4160, 2006)
person 0.15 (0, 1698) (1997, 3120)
person 0.15 (1929, 468) (3509, 2655)
person 0.15 (2580, 466) (4139, 2656)
person 0.15 (876, 0) (4160, 1391)
person 0.15 (1471, 0) (2695, 729)
person 0.15 (1193, 1716) (4160, 3120)
person 0.16 (2128, 773) (4160, 3120)
person 0.16 (0, 0) (2325, 3120)
person 0.16 (0, 1388) (3560, 3120)
person 0.17 (0, 0) (3607, 3120)
person 0.17 (222, 0) (3935, 2363)
person 0.17 (1853, 0) (4160, 3120)
person 0.18 (880, 398) (4160, 3120)
motorbike 0.14 (0, 1110) (1094, 2008)
motorbike 0.14 (0, 0) (1100, 87)
motorbike 0.14 (2742, 0) (3983, 404)
motorbike 0.14 (0, 1756) (1099, 2647)
motorbike 0.14 (503, 0) (1739, 406)
motorbike 0.14 (1784, 0) (3021, 403)
motorbike 0.14 (2428, 2075) (3656, 2965)
motorbike 0.14 (1455, 2383) (2709, 3120)
motorbike 0.14 (1610, 1742) (3188, 3120)
motorbike 0.14 (0, 735) (2937, 3120)
motorbike 0.14 (228, 0) (3931, 1388)
motorbike 0.14 (0, 0) (1095, 403)
motorbike 0.14 (0, 0) (3283, 3120)
motorbike 0.14 (1449, 2705) (2709, 3120)
motorbike 0.14 (0, 2705) (1094, 3120)
motorbike 0.14 (0, 2381) (1104, 3120)
motorbike 0.15 (0, 2084) (1103, 2961)
motorbike 0.15 (0, 0) (3603, 2377)
motorbike 0.15 (1214, 85) (4160, 3120)
motorbike 0.16 (579, 1050) (4160, 3120)
horse 0.14 (1147, 1440) (2379, 2320)
horse 0.14 (874, 1705) (4160, 3120)
horse 0.14 (1468, 153) (2694, 1045)
horse 0.14 (1785, 472) (3019, 1361)
horse 0.14 (2093, 3034) (3347, 3120)
horse 0.14 (2753, 1752) (3975, 2643)
horse 0.14 (828, 790) (2055, 1686)
horse 0.14 (1470, 1110) (2695, 2008)
horse 0.14 (2106, 0) (3340, 726)
horse 0.14 (2126, 430) (4160, 3120)
horse 0.14 (2099, 1105) (3345, 2008)
horse 0.14 (3075, 2073) (4160, 2960)
horse 0.14 (1455, 2383) (2709, 3120)
horse 0.14 (0, 2084) (1103, 2961)
horse 0.14 (0, 0) (1095, 403)
horse 0.14 (1449, 2705) (2709, 3120)
horse 0.15 (0, 1110) (1094, 2008)
horse 0.15 (0, 1756) (1099, 2647)
horse 0.15 (586, 0) (4160, 3120)
horse 0.15 (0, 1432) (1095, 2327)
dog 0.15 (1449, 2705) (2709, 3120)
dog 0.15 (2742, 0) (3983, 404)
dog 0.15 (3061, 0) (4160, 730)
dog 0.15 (2743, 1430) (3988, 2329)
dog 0.15 (3077, 1107) (4160, 2006)
dog 0.15 (0, 146) (1098, 1051)
dog 0.15 (1143, 0) (2380, 399)
dog 0.15 (3058, 156) (4160, 1036)
dog 0.15 (2413, 2715) (3673, 3120)
dog 0.15 (0, 1756) (1099, 2647)
dog 0.15 (2106, 152) (3340, 1040)
dog 0.15 (3075, 2073) (4160, 2960)
dog 0.15 (0, 0) (1097, 730)
dog 0.15 (2106, 0) (3340, 726)
dog 0.15 (227, 0) (3927, 2050)
dog 0.15 (3057, 2718) (4160, 3120)
dog 0.16 (0, 0) (3607, 3120)
dog 0.16 (1850, 94) (4160, 3120)
dog 0.16 (177, 0) (1427, 408)
dog 0.16 (889, 746) (4160, 3120)
diningtable 0.14 (3058, 156) (4160, 1036)
diningtable 0.14 (2093, 3034) (3347, 3120)
diningtable 0.14 (1836, 0) (4160, 3014)
diningtable 0.14 (3053, 3036) (4160, 3120)
diningtable 0.14 (0, 2381) (1104, 3120)
diningtable 0.15 (0, 2705) (1094, 3120)
diningtable 0.15 (1807, 1413) (4160, 3120)
diningtable 0.15 (0, 1698) (1997, 3120)
diningtable 0.15 (2413, 2715) (3673, 3120)
diningtable 0.15 (183, 2082) (1427, 2962)
diningtable 0.15 (3057, 2718) (4160, 3120)
diningtable 0.15 (3075, 2073) (4160, 2960)
diningtable 0.15 (0, 0) (2645, 2697)
diningtable 0.15 (814, 3026) (2070, 3120)
diningtable 0.15 (0, 1078) (3285, 3120)
diningtable 0.15 (222, 0) (3935, 2363)
diningtable 0.15 (0, 122) (2969, 3120)
diningtable 0.16 (586, 0) (4160, 3120)
diningtable 0.16 (2126, 430) (4160, 3120)
diningtable 0.16 (889, 746) (4160, 3120)
cow 0.13 (1155, 0) (2370, 728)
cow 0.13 (177, 0) (1427, 408)
cow 0.13 (1139, 1110) (2384, 2007)
cow 0.13 (495, 3030) (1746, 3120)
cow 0.14 (0, 1441) (1257, 3120)
cow 0.14 (0, 424) (2002, 3120)
cow 0.14 (1509, 1717) (4160, 3120)
cow 0.14 (0, 1756) (1099, 2647)
cow 0.14 (1150, 1751) (2374, 2647)
cow 0.14 (2106, 0) (3340, 726)
cow 0.14 (2093, 3034) (3347, 3120)
cow 0.14 (0, 1698) (1997, 3120)
cow 0.14 (0, 153) (776, 1039)
cow 0.14 (192, 0) (1414, 730)
cow 0.14 (1214, 85) (4160, 3120)
cow 0.14 (0, 2084) (1103, 2961)
cow 0.14 (531, 0) (4160, 2034)
cow 0.14 (0, 1704) (3589, 3120)
cow 0.15 (0, 119) (3594, 3120)
cow 0.15 (0, 0) (2967, 2040)
chair 0.15 (2413, 2715) (3673, 3120)
chair 0.15 (3066, 1754) (4160, 2637)
chair 0.15 (1449, 2705) (2709, 3120)
chair 0.15 (0, 794) (778, 1681)
chair 0.15 (1193, 1716) (4160, 3120)
chair 0.15 (189, 800) (1416, 1683)
chair 0.15 (3057, 2718) (4160, 3120)
chair 0.15 (1473, 473) (2688, 1362)
chair 0.15 (2099, 1105) (3345, 2008)
chair 0.15 (2743, 1430) (3988, 2329)
chair 0.15 (3075, 2073) (4160, 2960)
chair 0.15 (0, 0) (2645, 2697)
chair 0.16 (3058, 156) (4160, 1036)
chair 0.16 (2147, 0) (4160, 3120)
chair 0.16 (0, 129) (3288, 3120)
chair 0.16 (1823, 754) (4160, 3120)
chair 0.16 (3077, 1107) (4160, 2006)
chair 0.16 (227, 0) (3927, 2050)
chair 0.16 (242, 1073) (3910, 3120)
chair 0.17 (903, 89) (4160, 3120)
cat 0.14 (2117, 1732) (4160, 3120)
cat 0.14 (0, 468) (1261, 2653)
cat 0.14 (0, 0) (1095, 403)
cat 0.14 (1147, 1440) (2379, 2320)
cat 0.14 (334, 1108) (1902, 3120)
cat 0.14 (0, 752) (1997, 3120)
cat 0.14 (876, 0) (4160, 1391)
cat 0.14 (254, 1708) (3900, 3120)
cat 0.14 (2741, 2399) (3985, 3120)
cat 0.14 (1784, 0) (3021, 403)
cat 0.14 (1150, 1751) (2374, 2647)
cat 0.14 (1290, 1430) (2866, 3120)
cat 0.14 (966, 788) (2551, 2974)
cat 0.14 (1143, 0) (2380, 399)
cat 0.15 (503, 0) (1739, 406)
cat 0.15 (1929, 468) (3509, 2655)
cat 0.15 (1455, 2383) (2709, 3120)
cat 0.15 (0, 2084) (1103, 2961)
cat 0.15 (0, 1441) (1257, 3120)
cat 0.15 (889, 746) (4160, 3120)
car 0.14 (512, 1108) (1735, 2010)
car 0.14 (505, 793) (1739, 1683)
car 0.14 (1489, 0) (4160, 2055)
car 0.14 (1449, 2705) (2709, 3120)
car 0.14 (3077, 1107) (4160, 2006)
car 0.14 (0, 0) (2645, 2697)
car 0.14 (0, 2705) (1094, 3120)
car 0.14 (3057, 2718) (4160, 3120)
car 0.14 (0, 797) (1101, 1682)
car 0.15 (0, 478) (1097, 1365)
car 0.15 (1139, 1110) (2384, 2007)
car 0.15 (1145, 794) (2380, 1682)
car 0.15 (0, 0) (3602, 2043)
car 0.15 (0, 2381) (1104, 3120)
car 0.15 (2126, 430) (4160, 3120)
car 0.15 (0, 460) (2331, 3120)
car 0.15 (251, 1400) (3899, 3120)
car 0.15 (0, 2084) (1103, 2961)
car 0.16 (0, 0) (3607, 3120)
car 0.17 (880, 398) (4160, 3120)
bus 0.14 (0, 797) (1101, 1682)
bus 0.14 (505, 793) (1739, 1683)
bus 0.14 (2432, 466) (3651, 1358)
bus 0.14 (2138, 0) (4160, 2364)
bus 0.14 (3375, 474) (4160, 1356)
bus 0.14 (1473, 473) (2688, 1362)
bus 0.14 (2117, 1732) (4160, 3120)
bus 0.14 (2431, 0) (3658, 726)
bus 0.14 (3077, 1107) (4160, 2006)
bus 0.14 (827, 480) (2055, 1361)
bus 0.14 (0, 2084) (1103, 2961)
bus 0.14 (0, 453) (2620, 3120)
bus 0.14 (251, 1400) (3899, 3120)
bus 0.14 (0, 0) (2976, 1383)
bus 0.14 (0, 0) (2645, 2697)
bus 0.15 (1186, 0) (4160, 1403)
bus 0.15 (227, 0) (3927, 2050)
bus 0.15 (0, 0) (3607, 3120)
bus 0.15 (2126, 430) (4160, 3120)
bus 0.16 (880, 398) (4160, 3120)
bottle 0.14 (3077, 1107) (4160, 2006)
bottle 0.14 (1150, 1751) (2374, 2647)
bottle 0.14 (1150, 478) (2373, 1361)
bottle 0.14 (0, 1728) (2336, 3120)
bottle 0.14 (0, 0) (1984, 3120)
bottle 0.14 (3058, 156) (4160, 1036)
bottle 0.14 (1149, 2387) (2373, 3120)
bottle 0.14 (0, 2076) (1263, 3120)
bottle 0.14 (1816, 1717) (4160, 3120)
bottle 0.14 (0, 2084) (1103, 2961)
bottle 0.14 (1610, 1742) (3188, 3120)
bottle 0.14 (2093, 3034) (3347, 3120)
bottle 0.14 (0, 2705) (1094, 3120)
bottle 0.14 (3075, 2073) (4160, 2960)
bottle 0.14 (196, 2384) (1412, 3120)
bottle 0.15 (335, 1759) (1901, 3120)
bottle 0.15 (0, 0) (3264, 2679)
bottle 0.15 (0, 1704) (3589, 3120)
bottle 0.15 (1853, 0) (4160, 3120)
bottle 0.16 (880, 398) (4160, 3120)
boat 0.14 (2432, 466) (3651, 1358)
boat 0.14 (0, 1441) (1257, 3120)
boat 0.14 (827, 480) (2055, 1361)
boat 0.14 (196, 2384) (1412, 3120)
boat 0.14 (1473, 473) (2688, 1362)
boat 0.14 (0, 0) (1984, 3120)
boat 0.15 (0, 0) (2323, 2358)
boat 0.15 (0, 1704) (3589, 3120)
boat 0.15 (228, 0) (3931, 1388)
boat 0.15 (1455, 2383) (2709, 3120)
boat 0.15 (3053, 3036) (4160, 3120)
boat 0.15 (0, 797) (1101, 1682)
boat 0.15 (183, 2082) (1427, 2962)
boat 0.15 (0, 431) (3580, 3120)
boat 0.15 (2750, 158) (3969, 1037)
boat 0.15 (0, 0) (3603, 2377)
boat 0.15 (2117, 1732) (4160, 3120)
boat 0.16 (917, 0) (4160, 3120)
boat 0.16 (887, 1052) (4160, 3120)
boat 0.16 (2126, 430) (4160, 3120)
bird 0.14 (822, 0) (2061, 402)
bird 0.14 (828, 790) (2055, 1686)
bird 0.15 (0, 0) (1975, 2047)
bird 0.15 (189, 800) (1416, 1683)
bird 0.15 (827, 480) (2055, 1361)
bird 0.15 (196, 2384) (1412, 3120)
bird 0.15 (1853, 0) (4160, 3120)
bird 0.15 (0, 146) (1098, 1051)
bird 0.15 (3075, 2073) (4160, 2960)
bird 0.15 (1828, 1078) (4160, 3120)
bird 0.15 (0, 1110) (1094, 2008)
bird 0.15 (874, 1705) (4160, 3120)
bird 0.15 (177, 0) (1427, 408)
bird 0.15 (227, 0) (3927, 2050)
bird 0.15 (0, 2084) (1103, 2961)
bird 0.15 (0, 0) (2953, 2686)
bird 0.16 (0, 752) (1997, 3120)
bird 0.16 (880, 398) (4160, 3120)
bird 0.16 (16, 1099) (1578, 3120)
bird 0.16 (0, 1690) (3270, 3120)
bicycle 0.13 (1929, 468) (3509, 2655)
bicycle 0.13 (3371, 3033) (4160, 3120)
bicycle 0.13 (0, 2084) (1103, 2961)
bicycle 0.13 (3057, 2718) (4160, 3120)
bicycle 0.13 (2419, 3033) (3662, 3120)
bicycle 0.13 (0, 0) (2006, 1392)
bicycle 0.13 (0, 1110) (1094, 2008)
bicycle 0.14 (1836, 0) (4160, 3014)
bicycle 0.14 (0, 424) (2002, 3120)
bicycle 0.14 (1449, 2705) (2709, 3120)
bicycle 0.14 (0, 0) (3291, 1377)
bicycle 0.14 (2135, 0) (4160, 2027)
bicycle 0.14 (0, 1698) (1997, 3120)
bicycle 0.14 (1186, 0) (4160, 1403)
bicycle 0.14 (1527, 1408) (4160, 3120)
bicycle 0.15 (0, 0) (2607, 3020)
bicycle 0.15 (2126, 430) (4160, 3120)
bicycle 0.15 (0, 0) (3603, 2377)
bicycle 0.15 (0, 1078) (3285, 3120)
bicycle 0.16 (880, 398) (4160, 3120)
aeroplane 0.15 (2427, 0) (3657, 85)
aeroplane 0.15 (1464, 0) (2698, 85)
aeroplane 0.15 (3058, 156) (4160, 1036)
aeroplane 0.15 (180, 0) (1414, 83)
aeroplane 0.15 (1143, 0) (2380, 399)
aeroplane 0.15 (819, 0) (2062, 84)
aeroplane 0.15 (1784, 0) (3021, 403)
aeroplane 0.15 (3061, 0) (4160, 88)
aeroplane 0.15 (0, 2084) (1103, 2961)
aeroplane 0.15 (2128, 773) (4160, 3120)
aeroplane 0.15 (0, 0) (2327, 1393)
aeroplane 0.15 (0, 1756) (1099, 2647)
aeroplane 0.15 (0, 0) (2325, 3120)
aeroplane 0.15 (3059, 0) (4160, 406)
aeroplane 0.15 (0, 750) (3273, 3120)
aeroplane 0.15 (0, 0) (3610, 1391)
aeroplane 0.16 (222, 0) (3935, 2363)
aeroplane 0.16 (880, 398) (4160, 3120)
aeroplane 0.16 (1186, 0) (4160, 1403)
aeroplane 0.16 (0, 0) (3607, 3120)

[cococolor]

I retrained this model with VOC dataset
After 20 epoches, I got a trained_weights.h5 with about 7.0 loss.
So I just modified model_path and classes_path in yolo.py.
But using python yolo.py, my model can't find any box in my test pictures.
This blocked me one month.
I can't find where the problem is.
Thank you for your help!

[qqwweee]

I don't know what was wrong with your training. I have tried many times and it usually works. The loss is 20~17 when input shape is 416*416, 40~100 epochs, VOC2007 trainval.
Even if you freeze most of the layers, the trained model is not bad. I think that's because the model capacity is very very large.

[yqyao]

How about the map in VOC2007 test? @qqwweee

[cooli7wa]

@qqwweee
Thank you for reply.

loss: 31.3760 - val_loss: 937.2539

loss decrease quickly, but val_loss quite slowly, and they have large margin.
Is this true during your training? @qqwweee

[qqwweee]

@cooli7wa It is not normal. The loss and val_loss is supposed to have a small margin.

[zqburde]

@cooli7wa
i have the same problem, do you solve it?

[cooli7wa]

@zqburde
no, not solve, when i solve, i will paste reason here.

[zqburde]

@cooli7wa
i have trained the model in VOC07+12, and now i get the train loss 15 and val loss 13. First i train the model according to the author's readme, after 50 epoches i get a terrible loss like you, train loss 31 and val loss 1000, then i have continued to train 50 epoches, the train loss is still 31 but the val loss decrease to 100+, these processes are all in frozen layers. Last i unfreeze layers and continue training 50 epoches, but i change the batch size to 16 and comment the code of frozen layers. although the final loss is not great, it is not very effective when testing with my own data.

[Pichairen]

I got the same problem today... .. .
and I haven‘t any idea now ... .. .

[jinxie0731]

I suffered the same problem when I tried to train a model on VOC dataset, the final trained_weights_final.h5 which got from the train.py can't find any box in yolo.py.
Did anyone succeed in training VOC dataset?

[jinxie0731]

Finally, I succeed in training on VOC dataset which only recognize bicycle.
I think the root cause is my train.txt didn't contain any valid Box data, what it has is only image_file_path, To be more detailed, a standard row format of train.txt required by this repo is:
"image_file_path box1 box2 ... boxN"
but my train.txt is like this:
"image_file_path"
this repo has already provide annotation file(e.g. voc_annotation.py) to generate train.txt for us, this script will generate 2007_test.txt,2007_train.txt and 2007_val.txt, what extra we need to do is cat them together to get a train.txt.
what a stupid mistake. Currently, the core is not so high as my epochs is only 4, so I will train more to see if I can get a better score.

[swy0915]

i train voc2007 bitch16, and not load retrain model, use titan 1080 , after 2days, loss =19,it looks not good.

[linydf]

anyone who train the datadat the loss is lower？？？I use 1080,and train for two days,the loss about 39.the test result is not good

[KiveeDong]

I got mAP about 72% on voc2007_test, and loss is about 17 after 100 epoches with batchsize is 32.(on one TITAN xp)

[fffupeng]

24/24 [==============================] - 32s 1s/step - loss: 1855.2923 - val_loss: 3390728.9688
Epoch 2/100
24/24 [==============================] - 19s 775ms/step - loss: 228.4187 - val_loss: 17582.2413
Epoch 3/100
24/24 [==============================] - 19s 775ms/step - loss: 130.7130 - val_loss: 223.0762
Epoch 4/100
24/24 [==============================] - 19s 777ms/step - loss: 94.9265 - val_loss: 264.2446
Epoch 5/100
24/24 [==============================] - 19s 776ms/step - loss: 73.9083 - val_loss: 283.4447
Epoch 6/100
24/24 [==============================] - 19s 781ms/step - loss: 62.4733 - val_loss: 93.2500
Epoch 7/100
24/24 [==============================] - 19s 777ms/step - loss: 57.1280 - val_loss: 17010529.1250
Epoch 8/100
24/24 [==============================] - 19s 775ms/step - loss: 63.5084 - val_loss: 332598188.1875
Epoch 9/100
24/24 [==============================] - 19s 775ms/step - loss: 60.8199 - val_loss: 3841.8596
Epoch 10/100
24/24 [==============================] - 19s 777ms/step - loss: 46.3737 - val_loss: 416.8000
Epoch 11/100
24/24 [==============================] - 19s 775ms/step - loss: 40.0810 - val_loss: 320.2411
Epoch 12/100
24/24 [==============================] - 19s 779ms/step - loss: 38.6060 - val_loss: 34.1380
Epoch 13/100
24/24 [==============================] - 19s 782ms/step - loss: 34.4895 - val_loss: 34.1830
Epoch 14/100
24/24 [==============================] - 19s 781ms/step - loss: 31.6124 - val_loss: 30.9661
Epoch 15/100
24/24 [==============================] - 19s 777ms/step - loss: 31.2436 - val_loss: 28.4113
Epoch 16/100
24/24 [==============================] - 19s 777ms/step - loss: 29.9609 - val_loss: 29.0701
Epoch 17/100
24/24 [==============================] - 19s 775ms/step - loss: 33.9855 - val_loss: 35.7120
Epoch 18/100
24/24 [==============================] - 19s 777ms/step - loss: 29.3629 - val_loss: 26.9789
Epoch 19/100
24/24 [==============================] - 19s 777ms/step - loss: 27.5470 - val_loss: 26.0849
Epoch 20/100
24/24 [==============================] - 19s 776ms/step - loss: 26.7831 - val_loss: 27.6721
Epoch 21/100
24/24 [==============================] - 19s 776ms/step - loss: 26.0673 - val_loss: 26.0159
Epoch 22/100
24/24 [==============================] - 19s 776ms/step - loss: 25.5499 - val_loss: 25.1829
Epoch 23/100
24/24 [==============================] - 19s 778ms/step - loss: 25.3891 - val_loss: 26.1550
Epoch 24/100
24/24 [==============================] - 19s 776ms/step - loss: 25.2917 - val_loss: 24.8415
Epoch 25/100
24/24 [==============================] - 19s 777ms/step - loss: 24.5065 - val_loss: 23.9405
Epoch 26/100
24/24 [==============================] - 19s 776ms/step - loss: 24.2171 - val_loss: 29.0975
Epoch 27/100
24/24 [==============================] - 19s 776ms/step - loss: 23.7548 - val_loss: 30.1721
Epoch 28/100
24/24 [==============================] - 19s 776ms/step - loss: 23.6231 - val_loss: 23.9454
Epoch 29/100
24/24 [==============================] - 19s 778ms/step - loss: 23.2079 - val_loss: 22.7171
Epoch 30/100
24/24 [==============================] - 19s 784ms/step - loss: 22.9274 - val_loss: 22.9661
Epoch 31/100
24/24 [==============================] - 19s 776ms/step - loss: 22.9505 - val_loss: 21.9244
Epoch 32/100
24/24 [==============================] - 19s 776ms/step - loss: 22.7322 - val_loss: 22.2343
Epoch 33/100
24/24 [==============================] - 19s 776ms/step - loss: 22.0856 - val_loss: 21.9918
Epoch 34/100
24/24 [==============================] - 19s 776ms/step - loss: 22.3082 - val_loss: 22.1995
Epoch 35/100
24/24 [==============================] - 19s 778ms/step - loss: 22.1402 - val_loss: 20.6331
Epoch 36/100
24/24 [==============================] - 19s 775ms/step - loss: 21.7534 - val_loss: 21.3660
Epoch 37/100
24/24 [==============================] - 19s 775ms/step - loss: 21.5871 - val_loss: 21.8522
Epoch 38/100
24/24 [==============================] - 19s 781ms/step - loss: 21.5326 - val_loss: 24.8214
Epoch 39/100
24/24 [==============================] - 19s 777ms/step - loss: 21.3279 - val_loss: 20.7667
Epoch 40/100
24/24 [==============================] - 19s 776ms/step - loss: 21.5276 - val_loss: 19.8768
Epoch 41/100
24/24 [==============================] - 19s 774ms/step - loss: 21.3221 - val_loss: 21.4741
Epoch 42/100
24/24 [==============================] - 19s 776ms/step - loss: 20.9914 - val_loss: 21.3854
Epoch 43/100
24/24 [==============================] - 19s 776ms/step - loss: 20.4727 - val_loss: 19.5242
Epoch 44/100
24/24 [==============================] - 19s 775ms/step - loss: 20.4470 - val_loss: 20.8995
Epoch 45/100
24/24 [==============================] - 19s 774ms/step - loss: 20.6613 - val_loss: 19.2053
Epoch 46/100
24/24 [==============================] - 19s 777ms/step - loss: 20.5620 - val_loss: 29.4835
Epoch 47/100
24/24 [==============================] - 19s 776ms/step - loss: 20.3510 - val_loss: 21.1771
Epoch 48/100
24/24 [==============================] - 19s 777ms/step - loss: 20.7037 - val_loss: 21.8391
Epoch 49/100
24/24 [==============================] - 19s 774ms/step - loss: 20.2857 - val_loss: 21.5700
Epoch 50/100
24/24 [==============================] - 19s 775ms/step - loss: 19.9783 - val_loss: 19.6167
Epoch 51/100
24/24 [==============================] - 19s 775ms/step - loss: 19.7407 - val_loss: 20.4395
Epoch 52/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6230 - val_loss: 18.2430
Epoch 53/100
24/24 [==============================] - 19s 776ms/step - loss: 19.9892 - val_loss: 20.8178
Epoch 54/100
24/24 [==============================] - 19s 776ms/step - loss: 19.6136 - val_loss: 19.7537
Epoch 55/100
24/24 [==============================] - 19s 777ms/step - loss: 20.1485 - val_loss: 25.1544
Epoch 56/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6000 - val_loss: 48.2337
Epoch 57/100
24/24 [==============================] - 19s 779ms/step - loss: 19.7300 - val_loss: 271.8140
Epoch 58/100
24/24 [==============================] - 19s 773ms/step - loss: 20.1995 - val_loss: 20.0360
Epoch 59/100
24/24 [==============================] - 19s 778ms/step - loss: 19.4499 - val_loss: 18.9209
Epoch 60/100
24/24 [==============================] - 19s 776ms/step - loss: 19.4476 - val_loss: 18.7522
Epoch 61/100
24/24 [==============================] - 19s 777ms/step - loss: 19.3816 - val_loss: 18.3116
Epoch 62/100
24/24 [==============================] - 19s 775ms/step - loss: 18.9603 - val_loss: 17.7904
Epoch 63/100
24/24 [==============================] - 19s 776ms/step - loss: 19.2028 - val_loss: 18.6347
Epoch 64/100
24/24 [==============================] - 19s 776ms/step - loss: 19.5405 - val_loss: 18.8356
Epoch 65/100
24/24 [==============================] - 19s 774ms/step - loss: 18.7260 - val_loss: 19.5147
Epoch 66/100
24/24 [==============================] - 19s 776ms/step - loss: 18.6204 - val_loss: 18.4056
Epoch 67/100
24/24 [==============================] - 19s 777ms/step - loss: 18.7061 - val_loss: 17.8649
Epoch 68/100
24/24 [==============================] - 19s 776ms/step - loss: 18.7692 - val_loss: 18.7073
Epoch 69/100
24/24 [==============================] - 19s 774ms/step - loss: 18.4413 - val_loss: 17.8672
Epoch 70/100
24/24 [==============================] - 19s 777ms/step - loss: 18.4271 - val_loss: 17.1728
Epoch 71/100
24/24 [==============================] - 19s 777ms/step - loss: 18.1385 - val_loss: 16.6848
Epoch 72/100
24/24 [==============================] - 19s 775ms/step - loss: 18.3120 - val_loss: 17.9856
Epoch 73/100
24/24 [==============================] - 19s 778ms/step - loss: 17.7598 - val_loss: 18.1857
Epoch 74/100
24/24 [==============================] - 19s 781ms/step - loss: 18.0805 - val_loss: 17.2571
Epoch 75/100
24/24 [==============================] - 19s 779ms/step - loss: 17.6560 - val_loss: 18.0094
Epoch 76/100
24/24 [==============================] - 19s 777ms/step - loss: 17.8811 - val_loss: 17.1414
Epoch 77/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6542 - val_loss: 18.0857
Epoch 78/100
24/24 [==============================] - 19s 780ms/step - loss: 17.8937 - val_loss: 17.0371
Epoch 79/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6179 - val_loss: 23.6756
Epoch 80/100
24/24 [==============================] - 19s 776ms/step - loss: 17.4508 - val_loss: 17.0178
Epoch 81/100
24/24 [==============================] - 19s 780ms/step - loss: 17.7438 - val_loss: 16.7333
Epoch 82/100
24/24 [==============================] - 19s 776ms/step - loss: 17.7676 - val_loss: 67.6749
Epoch 83/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1856 - val_loss: 17.5176
Epoch 84/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1831 - val_loss: 16.8703
Epoch 85/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9395 - val_loss: 18.3559
Epoch 86/100
24/24 [==============================] - 19s 777ms/step - loss: 17.4497 - val_loss: 17.6610
Epoch 87/100
24/24 [==============================] - 19s 777ms/step - loss: 16.7546 - val_loss: 18.7588
Epoch 88/100
24/24 [==============================] - 19s 775ms/step - loss: 17.3181 - val_loss: 17.9550
Epoch 89/100
24/24 [==============================] - 19s 777ms/step - loss: 16.6001 - val_loss: 15.6468
Epoch 90/100
24/24 [==============================] - 19s 779ms/step - loss: 16.9396 - val_loss: 17.5748
Epoch 91/100
24/24 [==============================] - 19s 776ms/step - loss: 16.6347 - val_loss: 17.1885
Epoch 92/100
24/24 [==============================] - 19s 775ms/step - loss: 17.0368 - val_loss: 16.5268
Epoch 93/100
24/24 [==============================] - 19s 777ms/step - loss: 16.3248 - val_loss: 16.6590
Epoch 94/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4596 - val_loss: 15.1912
Epoch 95/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9770 - val_loss: 15.4842
Epoch 96/100
24/24 [==============================] - 19s 776ms/step - loss: 16.4053 - val_loss: 17.7234
Epoch 97/100
24/24 [==============================] - 19s 777ms/step - loss: 16.1250 - val_loss: 16.5083
Epoch 98/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4032 - val_loss: 15.9407
Epoch 99/100
24/24 [==============================] - 19s 776ms/step - loss: 16.2643 - val_loss: 16.7436
Epoch 100/100
24/24 [==============================] - 19s 777ms/step - loss: 15.9776 - val_loss: 16.2618

I train yolo on raccon dataset. After 100 epoch, i still can not detect any object.

[gamaEugene]

I also got this problem,
My loss and val_loss are around 17 to 16 after 300 Epoch with all layer unfreeze,
but it still can't find any box, even in training dataset.
Does anyone know what's wrong?

[lightning20]

Also got this problem. Struggling for 1.5 weeks.

Only have one class(500 images), and anchors keep the same. The class and anchor files are proper set. The weight is downloaded from yolo3 site and converted by the command. The content format in train.txt is all correct with full file path and 4 number for position and 1 number for class.

The original loss is 7000 and val loss is always nan. Tried to train 200 echo and got loss as 700. When do the detection, it said 0 box detected even use back the same image in training.
Epoch 221/1000
19/19 [==============================] - 44s 2s/step - loss: 195.7787 - val_loss: nan
Epoch 222/1000
19/19 [==============================] - 45s 2s/step - loss: 194.0862 - val_loss: nan

Any suggestion, thanks.
alex

[vinicius121]

Hi there,

I have the same problem here.
Epoch 177/200
5/5 [==============================] - 14s 3s/step - loss: 11.2442 - val_loss: 10.6615

It can't detect any bounding box.

Any suggestion, thanks.

[ZhiweiDuan]

@qqwweee Have you encountered this problem: - val_loss: nan?
Epoch 2/500
576/576 [==============================] - 215s 374ms/step - loss: 1383.9500 - val_loss: nan
Epoch 3/500
576/576 [==============================] - 216s 374ms/step - loss: 686.2021 - val_loss: nan
Epoch 4/500
576/576 [==============================] - 217s 377ms/step - loss: 410.9110 - val_loss: nan

[phamngocthanhtrung]

@qqwweee I have the same problem as @ZhiweiDuan and @lightning20. my issue #304
pls help us :)

[michalkowalski94]

Same problem here. I'm training model from the beginning to detect abnormalities in breats mammograms. Training seems to be fine but the is something wrong with box visualization. I'm training on tesla k40. Now I have set the training for 3 more days to get even better loss. I'll paste my tensorboard and test results then.

[michalkowalski94]

For those of You who have problem with bounding boxes and score 0 #319

[enoceanwei]

@cooli7wa

Sorry to disturb you,

I am suffering the same problem, have you solve it? could you tell me how to fix the problem? many thanks.

By the way, I use the transfer pretrain weight from the command line, and finished the stage1 training about 50 epoch on my own dataset, the model can not detect anything on a image.

I am looking forward to your replay, many thanks.

Kind regards

Wei

[michalkowalski94]

Hi,
I have solved my problem by not using "random data augmentation" right now
i have modified yolo loss and I'm waiting for results".
Also i have drastically increased image size and reduced batch size to 1.
If You want to see code, feel free to visit my github page
michalkowalski94.github.io/MSADnn

Pon., 28 sty 2019, 07:57: enoceanwei [e-mail] napisał(a):

@cooli7wa https://github.com/cooli7wa

Sorry to disturb you,

I am suffering the same problem, have you solve it? could you tell me how
to fix the problem? many thanks.

By the way, I use the transfer pretrain weight from the command line, and
finished the stage1 training about 50 epoch on my own dataset, the model
can not detect anything on a image.

I am looking forward to your replay, many thanks.

Kind regards

Wei

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/qqwweee/keras-yolo3/issues/86#issuecomment-458017417,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AmQBuMXCj9ouyfRnOB4NcUEc7-LsLV03ks5vHp9TgaJpZM4UYscO
.

[extra159]

24/24 [==============================] - 32s 1s/step - loss: 1855.2923 - val_loss: 3390728.9688
Epoch 2/100
24/24 [==============================] - 19s 775ms/step - loss: 228.4187 - val_loss: 17582.2413
Epoch 3/100
24/24 [==============================] - 19s 775ms/step - loss: 130.7130 - val_loss: 223.0762
Epoch 4/100
24/24 [==============================] - 19s 777ms/step - loss: 94.9265 - val_loss: 264.2446
Epoch 5/100
24/24 [==============================] - 19s 776ms/step - loss: 73.9083 - val_loss: 283.4447
Epoch 6/100
24/24 [==============================] - 19s 781ms/step - loss: 62.4733 - val_loss: 93.2500
Epoch 7/100
24/24 [==============================] - 19s 777ms/step - loss: 57.1280 - val_loss: 17010529.1250
Epoch 8/100
24/24 [==============================] - 19s 775ms/step - loss: 63.5084 - val_loss: 332598188.1875
Epoch 9/100
24/24 [==============================] - 19s 775ms/step - loss: 60.8199 - val_loss: 3841.8596
Epoch 10/100
24/24 [==============================] - 19s 777ms/step - loss: 46.3737 - val_loss: 416.8000
Epoch 11/100
24/24 [==============================] - 19s 775ms/step - loss: 40.0810 - val_loss: 320.2411
Epoch 12/100
24/24 [==============================] - 19s 779ms/step - loss: 38.6060 - val_loss: 34.1380
Epoch 13/100
24/24 [==============================] - 19s 782ms/step - loss: 34.4895 - val_loss: 34.1830
Epoch 14/100
24/24 [==============================] - 19s 781ms/step - loss: 31.6124 - val_loss: 30.9661
Epoch 15/100
24/24 [==============================] - 19s 777ms/step - loss: 31.2436 - val_loss: 28.4113
Epoch 16/100
24/24 [==============================] - 19s 777ms/step - loss: 29.9609 - val_loss: 29.0701
Epoch 17/100
24/24 [==============================] - 19s 775ms/step - loss: 33.9855 - val_loss: 35.7120
Epoch 18/100
24/24 [==============================] - 19s 777ms/step - loss: 29.3629 - val_loss: 26.9789
Epoch 19/100
24/24 [==============================] - 19s 777ms/step - loss: 27.5470 - val_loss: 26.0849
Epoch 20/100
24/24 [==============================] - 19s 776ms/step - loss: 26.7831 - val_loss: 27.6721
Epoch 21/100
24/24 [==============================] - 19s 776ms/step - loss: 26.0673 - val_loss: 26.0159
Epoch 22/100
24/24 [==============================] - 19s 776ms/step - loss: 25.5499 - val_loss: 25.1829
Epoch 23/100
24/24 [==============================] - 19s 778ms/step - loss: 25.3891 - val_loss: 26.1550
Epoch 24/100
24/24 [==============================] - 19s 776ms/step - loss: 25.2917 - val_loss: 24.8415
Epoch 25/100
24/24 [==============================] - 19s 777ms/step - loss: 24.5065 - val_loss: 23.9405
Epoch 26/100
24/24 [==============================] - 19s 776ms/step - loss: 24.2171 - val_loss: 29.0975
Epoch 27/100
24/24 [==============================] - 19s 776ms/step - loss: 23.7548 - val_loss: 30.1721
Epoch 28/100
24/24 [==============================] - 19s 776ms/step - loss: 23.6231 - val_loss: 23.9454
Epoch 29/100
24/24 [==============================] - 19s 778ms/step - loss: 23.2079 - val_loss: 22.7171
Epoch 30/100
24/24 [==============================] - 19s 784ms/step - loss: 22.9274 - val_loss: 22.9661
Epoch 31/100
24/24 [==============================] - 19s 776ms/step - loss: 22.9505 - val_loss: 21.9244
Epoch 32/100
24/24 [==============================] - 19s 776ms/step - loss: 22.7322 - val_loss: 22.2343
Epoch 33/100
24/24 [==============================] - 19s 776ms/step - loss: 22.0856 - val_loss: 21.9918
Epoch 34/100
24/24 [==============================] - 19s 776ms/step - loss: 22.3082 - val_loss: 22.1995
Epoch 35/100
24/24 [==============================] - 19s 778ms/step - loss: 22.1402 - val_loss: 20.6331
Epoch 36/100
24/24 [==============================] - 19s 775ms/step - loss: 21.7534 - val_loss: 21.3660
Epoch 37/100
24/24 [==============================] - 19s 775ms/step - loss: 21.5871 - val_loss: 21.8522
Epoch 38/100
24/24 [==============================] - 19s 781ms/step - loss: 21.5326 - val_loss: 24.8214
Epoch 39/100
24/24 [==============================] - 19s 777ms/step - loss: 21.3279 - val_loss: 20.7667
Epoch 40/100
24/24 [==============================] - 19s 776ms/step - loss: 21.5276 - val_loss: 19.8768
Epoch 41/100
24/24 [==============================] - 19s 774ms/step - loss: 21.3221 - val_loss: 21.4741
Epoch 42/100
24/24 [==============================] - 19s 776ms/step - loss: 20.9914 - val_loss: 21.3854
Epoch 43/100
24/24 [==============================] - 19s 776ms/step - loss: 20.4727 - val_loss: 19.5242
Epoch 44/100
24/24 [==============================] - 19s 775ms/step - loss: 20.4470 - val_loss: 20.8995
Epoch 45/100
24/24 [==============================] - 19s 774ms/step - loss: 20.6613 - val_loss: 19.2053
Epoch 46/100
24/24 [==============================] - 19s 777ms/step - loss: 20.5620 - val_loss: 29.4835
Epoch 47/100
24/24 [==============================] - 19s 776ms/step - loss: 20.3510 - val_loss: 21.1771
Epoch 48/100
24/24 [==============================] - 19s 777ms/step - loss: 20.7037 - val_loss: 21.8391
Epoch 49/100
24/24 [==============================] - 19s 774ms/step - loss: 20.2857 - val_loss: 21.5700
Epoch 50/100
24/24 [==============================] - 19s 775ms/step - loss: 19.9783 - val_loss: 19.6167
Epoch 51/100
24/24 [==============================] - 19s 775ms/step - loss: 19.7407 - val_loss: 20.4395
Epoch 52/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6230 - val_loss: 18.2430
Epoch 53/100
24/24 [==============================] - 19s 776ms/step - loss: 19.9892 - val_loss: 20.8178
Epoch 54/100
24/24 [==============================] - 19s 776ms/step - loss: 19.6136 - val_loss: 19.7537
Epoch 55/100
24/24 [==============================] - 19s 777ms/step - loss: 20.1485 - val_loss: 25.1544
Epoch 56/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6000 - val_loss: 48.2337
Epoch 57/100
24/24 [==============================] - 19s 779ms/step - loss: 19.7300 - val_loss: 271.8140
Epoch 58/100
24/24 [==============================] - 19s 773ms/step - loss: 20.1995 - val_loss: 20.0360
Epoch 59/100
24/24 [==============================] - 19s 778ms/step - loss: 19.4499 - val_loss: 18.9209
Epoch 60/100
24/24 [==============================] - 19s 776ms/step - loss: 19.4476 - val_loss: 18.7522
Epoch 61/100
24/24 [==============================] - 19s 777ms/step - loss: 19.3816 - val_loss: 18.3116
Epoch 62/100
24/24 [==============================] - 19s 775ms/step - loss: 18.9603 - val_loss: 17.7904
Epoch 63/100
24/24 [==============================] - 19s 776ms/step - loss: 19.2028 - val_loss: 18.6347
Epoch 64/100
24/24 [==============================] - 19s 776ms/step - loss: 19.5405 - val_loss: 18.8356
Epoch 65/100
24/24 [==============================] - 19s 774ms/step - loss: 18.7260 - val_loss: 19.5147
Epoch 66/100
24/24 [==============================] - 19s 776ms/step - loss: 18.6204 - val_loss: 18.4056
Epoch 67/100
24/24 [==============================] - 19s 777ms/step - loss: 18.7061 - val_loss: 17.8649
Epoch 68/100
24/24 [==============================] - 19s 776ms/step - loss: 18.7692 - val_loss: 18.7073
Epoch 69/100
24/24 [==============================] - 19s 774ms/step - loss: 18.4413 - val_loss: 17.8672
Epoch 70/100
24/24 [==============================] - 19s 777ms/step - loss: 18.4271 - val_loss: 17.1728
Epoch 71/100
24/24 [==============================] - 19s 777ms/step - loss: 18.1385 - val_loss: 16.6848
Epoch 72/100
24/24 [==============================] - 19s 775ms/step - loss: 18.3120 - val_loss: 17.9856
Epoch 73/100
24/24 [==============================] - 19s 778ms/step - loss: 17.7598 - val_loss: 18.1857
Epoch 74/100
24/24 [==============================] - 19s 781ms/step - loss: 18.0805 - val_loss: 17.2571
Epoch 75/100
24/24 [==============================] - 19s 779ms/step - loss: 17.6560 - val_loss: 18.0094
Epoch 76/100
24/24 [==============================] - 19s 777ms/step - loss: 17.8811 - val_loss: 17.1414
Epoch 77/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6542 - val_loss: 18.0857
Epoch 78/100
24/24 [==============================] - 19s 780ms/step - loss: 17.8937 - val_loss: 17.0371
Epoch 79/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6179 - val_loss: 23.6756
Epoch 80/100
24/24 [==============================] - 19s 776ms/step - loss: 17.4508 - val_loss: 17.0178
Epoch 81/100
24/24 [==============================] - 19s 780ms/step - loss: 17.7438 - val_loss: 16.7333
Epoch 82/100
24/24 [==============================] - 19s 776ms/step - loss: 17.7676 - val_loss: 67.6749
Epoch 83/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1856 - val_loss: 17.5176
Epoch 84/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1831 - val_loss: 16.8703
Epoch 85/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9395 - val_loss: 18.3559
Epoch 86/100
24/24 [==============================] - 19s 777ms/step - loss: 17.4497 - val_loss: 17.6610
Epoch 87/100
24/24 [==============================] - 19s 777ms/step - loss: 16.7546 - val_loss: 18.7588
Epoch 88/100
24/24 [==============================] - 19s 775ms/step - loss: 17.3181 - val_loss: 17.9550
Epoch 89/100
24/24 [==============================] - 19s 777ms/step - loss: 16.6001 - val_loss: 15.6468
Epoch 90/100
24/24 [==============================] - 19s 779ms/step - loss: 16.9396 - val_loss: 17.5748
Epoch 91/100
24/24 [==============================] - 19s 776ms/step - loss: 16.6347 - val_loss: 17.1885
Epoch 92/100
24/24 [==============================] - 19s 775ms/step - loss: 17.0368 - val_loss: 16.5268
Epoch 93/100
24/24 [==============================] - 19s 777ms/step - loss: 16.3248 - val_loss: 16.6590
Epoch 94/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4596 - val_loss: 15.1912
Epoch 95/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9770 - val_loss: 15.4842
Epoch 96/100
24/24 [==============================] - 19s 776ms/step - loss: 16.4053 - val_loss: 17.7234
Epoch 97/100
24/24 [==============================] - 19s 777ms/step - loss: 16.1250 - val_loss: 16.5083
Epoch 98/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4032 - val_loss: 15.9407
Epoch 99/100
24/24 [==============================] - 19s 776ms/step - loss: 16.2643 - val_loss: 16.7436
Epoch 100/100
24/24 [==============================] - 19s 777ms/step - loss: 15.9776 - val_loss: 16.2618

I train yolo on raccon dataset. After 100 epoch, i still can not detect any object.

I have the same problem can you share your solution?

[michalkowalski94]

Could You tell what is Your resolution and aspect ratio?
Also have You created Your own anchors?

[extra159]

Could You tell what is Your resolution and aspect ratio?
Also have You created Your own anchors?

I am using 416*416 image size and I also created my own anchors using k means method

[michalkowalski94]

What is the range of original images height and width.
Currently I'm using simmilar implementation of YOLO ( I've modified this one a bit but I can't share it until my publication comes out) on 4800 x 4800 images with One class detection, and "different" activation function on classifier and "continuous" threshold estimation are doing trick for me.

[extra159]

What is the range of original images height and width.
Currently I'm using simmilar implementation of YOLO ( I've modified this one a bit but I can't share it until my publication comes out) on 4800 x 4800 images with One class detection, and "different" activation function on classifier and "continuous" threshold estimation are doing trick for me.

I had 1280 X 720 images but I have converted them into 416X416. Then I draw the bounding box for each image and also calculated K-means to get anchors. I am using 6 classes.

[Niranjankumar-c]

@michalkowalski94 @qqwweee

1. I have the same problem, i had 1280 x 960 images with bounding boxes and I haven't resized them. I directly passed them into the model. Model is not able to detect any object while testing. do I need to resize the images?

1. How to invoke tensorboard while training to monitor loss?

[fourth-archive]

@Niranjankumar-c @yqyao @jayanij @michalkowalski94 this YOLOv3 tutorial may help you:
https://github.com/ultralytics/yolov3/wiki/Train-Custom-Data

The accompanying repository works on MacOS, Windows and Linux, includes multigpu and multithreading, performs inference on images, videos, webcams, and an iOS app. It also tests to slightly higher mAPs than darknet, including on the latest YOLOv3-SPP.weights (60.7 COCO mAP), and offers the ability to train custom datasets from scratch to darknet performance, all using PyTorch :)
https://github.com/ultralytics/yolov3

[tabsong]

encounter the same problem. It seems that there is no effective solution until now

[liumgg]

24/24 [==============================] - 32s 1s/step - loss: 1855.2923 - val_loss: 3390728.9688
Epoch 2/100
24/24 [==============================] - 19s 775ms/step - loss: 228.4187 - val_loss: 17582.2413
Epoch 3/100
24/24 [==============================] - 19s 775ms/step - loss: 130.7130 - val_loss: 223.0762
Epoch 4/100
24/24 [==============================] - 19s 777ms/step - loss: 94.9265 - val_loss: 264.2446
Epoch 5/100
24/24 [==============================] - 19s 776ms/step - loss: 73.9083 - val_loss: 283.4447
Epoch 6/100
24/24 [==============================] - 19s 781ms/step - loss: 62.4733 - val_loss: 93.2500
Epoch 7/100
24/24 [==============================] - 19s 777ms/step - loss: 57.1280 - val_loss: 17010529.1250
Epoch 8/100
24/24 [==============================] - 19s 775ms/step - loss: 63.5084 - val_loss: 332598188.1875
Epoch 9/100
24/24 [==============================] - 19s 775ms/step - loss: 60.8199 - val_loss: 3841.8596
Epoch 10/100
24/24 [==============================] - 19s 777ms/step - loss: 46.3737 - val_loss: 416.8000
Epoch 11/100
24/24 [==============================] - 19s 775ms/step - loss: 40.0810 - val_loss: 320.2411
Epoch 12/100
24/24 [==============================] - 19s 779ms/step - loss: 38.6060 - val_loss: 34.1380
Epoch 13/100
24/24 [==============================] - 19s 782ms/step - loss: 34.4895 - val_loss: 34.1830
Epoch 14/100
24/24 [==============================] - 19s 781ms/step - loss: 31.6124 - val_loss: 30.9661
Epoch 15/100
24/24 [==============================] - 19s 777ms/step - loss: 31.2436 - val_loss: 28.4113
Epoch 16/100
24/24 [==============================] - 19s 777ms/step - loss: 29.9609 - val_loss: 29.0701
Epoch 17/100
24/24 [==============================] - 19s 775ms/step - loss: 33.9855 - val_loss: 35.7120
Epoch 18/100
24/24 [==============================] - 19s 777ms/step - loss: 29.3629 - val_loss: 26.9789
Epoch 19/100
24/24 [==============================] - 19s 777ms/step - loss: 27.5470 - val_loss: 26.0849
Epoch 20/100
24/24 [==============================] - 19s 776ms/step - loss: 26.7831 - val_loss: 27.6721
Epoch 21/100
24/24 [==============================] - 19s 776ms/step - loss: 26.0673 - val_loss: 26.0159
Epoch 22/100
24/24 [==============================] - 19s 776ms/step - loss: 25.5499 - val_loss: 25.1829
Epoch 23/100
24/24 [==============================] - 19s 778ms/step - loss: 25.3891 - val_loss: 26.1550
Epoch 24/100
24/24 [==============================] - 19s 776ms/step - loss: 25.2917 - val_loss: 24.8415
Epoch 25/100
24/24 [==============================] - 19s 777ms/step - loss: 24.5065 - val_loss: 23.9405
Epoch 26/100
24/24 [==============================] - 19s 776ms/step - loss: 24.2171 - val_loss: 29.0975
Epoch 27/100
24/24 [==============================] - 19s 776ms/step - loss: 23.7548 - val_loss: 30.1721
Epoch 28/100
24/24 [==============================] - 19s 776ms/step - loss: 23.6231 - val_loss: 23.9454
Epoch 29/100
24/24 [==============================] - 19s 778ms/step - loss: 23.2079 - val_loss: 22.7171
Epoch 30/100
24/24 [==============================] - 19s 784ms/step - loss: 22.9274 - val_loss: 22.9661
Epoch 31/100
24/24 [==============================] - 19s 776ms/step - loss: 22.9505 - val_loss: 21.9244
Epoch 32/100
24/24 [==============================] - 19s 776ms/step - loss: 22.7322 - val_loss: 22.2343
Epoch 33/100
24/24 [==============================] - 19s 776ms/step - loss: 22.0856 - val_loss: 21.9918
Epoch 34/100
24/24 [==============================] - 19s 776ms/step - loss: 22.3082 - val_loss: 22.1995
Epoch 35/100
24/24 [==============================] - 19s 778ms/step - loss: 22.1402 - val_loss: 20.6331
Epoch 36/100
24/24 [==============================] - 19s 775ms/step - loss: 21.7534 - val_loss: 21.3660
Epoch 37/100
24/24 [==============================] - 19s 775ms/step - loss: 21.5871 - val_loss: 21.8522
Epoch 38/100
24/24 [==============================] - 19s 781ms/step - loss: 21.5326 - val_loss: 24.8214
Epoch 39/100
24/24 [==============================] - 19s 777ms/step - loss: 21.3279 - val_loss: 20.7667
Epoch 40/100
24/24 [==============================] - 19s 776ms/step - loss: 21.5276 - val_loss: 19.8768
Epoch 41/100
24/24 [==============================] - 19s 774ms/step - loss: 21.3221 - val_loss: 21.4741
Epoch 42/100
24/24 [==============================] - 19s 776ms/step - loss: 20.9914 - val_loss: 21.3854
Epoch 43/100
24/24 [==============================] - 19s 776ms/step - loss: 20.4727 - val_loss: 19.5242
Epoch 44/100
24/24 [==============================] - 19s 775ms/step - loss: 20.4470 - val_loss: 20.8995
Epoch 45/100
24/24 [==============================] - 19s 774ms/step - loss: 20.6613 - val_loss: 19.2053
Epoch 46/100
24/24 [==============================] - 19s 777ms/step - loss: 20.5620 - val_loss: 29.4835
Epoch 47/100
24/24 [==============================] - 19s 776ms/step - loss: 20.3510 - val_loss: 21.1771
Epoch 48/100
24/24 [==============================] - 19s 777ms/step - loss: 20.7037 - val_loss: 21.8391
Epoch 49/100
24/24 [==============================] - 19s 774ms/step - loss: 20.2857 - val_loss: 21.5700
Epoch 50/100
24/24 [==============================] - 19s 775ms/step - loss: 19.9783 - val_loss: 19.6167
Epoch 51/100
24/24 [==============================] - 19s 775ms/step - loss: 19.7407 - val_loss: 20.4395
Epoch 52/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6230 - val_loss: 18.2430
Epoch 53/100
24/24 [==============================] - 19s 776ms/step - loss: 19.9892 - val_loss: 20.8178
Epoch 54/100
24/24 [==============================] - 19s 776ms/step - loss: 19.6136 - val_loss: 19.7537
Epoch 55/100
24/24 [==============================] - 19s 777ms/step - loss: 20.1485 - val_loss: 25.1544
Epoch 56/100
24/24 [==============================] - 19s 777ms/step - loss: 19.6000 - val_loss: 48.2337
Epoch 57/100
24/24 [==============================] - 19s 779ms/step - loss: 19.7300 - val_loss: 271.8140
Epoch 58/100
24/24 [==============================] - 19s 773ms/step - loss: 20.1995 - val_loss: 20.0360
Epoch 59/100
24/24 [==============================] - 19s 778ms/step - loss: 19.4499 - val_loss: 18.9209
Epoch 60/100
24/24 [==============================] - 19s 776ms/step - loss: 19.4476 - val_loss: 18.7522
Epoch 61/100
24/24 [==============================] - 19s 777ms/step - loss: 19.3816 - val_loss: 18.3116
Epoch 62/100
24/24 [==============================] - 19s 775ms/step - loss: 18.9603 - val_loss: 17.7904
Epoch 63/100
24/24 [==============================] - 19s 776ms/step - loss: 19.2028 - val_loss: 18.6347
Epoch 64/100
24/24 [==============================] - 19s 776ms/step - loss: 19.5405 - val_loss: 18.8356
Epoch 65/100
24/24 [==============================] - 19s 774ms/step - loss: 18.7260 - val_loss: 19.5147
Epoch 66/100
24/24 [==============================] - 19s 776ms/step - loss: 18.6204 - val_loss: 18.4056
Epoch 67/100
24/24 [==============================] - 19s 777ms/step - loss: 18.7061 - val_loss: 17.8649
Epoch 68/100
24/24 [==============================] - 19s 776ms/step - loss: 18.7692 - val_loss: 18.7073
Epoch 69/100
24/24 [==============================] - 19s 774ms/step - loss: 18.4413 - val_loss: 17.8672
Epoch 70/100
24/24 [==============================] - 19s 777ms/step - loss: 18.4271 - val_loss: 17.1728
Epoch 71/100
24/24 [==============================] - 19s 777ms/step - loss: 18.1385 - val_loss: 16.6848
Epoch 72/100
24/24 [==============================] - 19s 775ms/step - loss: 18.3120 - val_loss: 17.9856
Epoch 73/100
24/24 [==============================] - 19s 778ms/step - loss: 17.7598 - val_loss: 18.1857
Epoch 74/100
24/24 [==============================] - 19s 781ms/step - loss: 18.0805 - val_loss: 17.2571
Epoch 75/100
24/24 [==============================] - 19s 779ms/step - loss: 17.6560 - val_loss: 18.0094
Epoch 76/100
24/24 [==============================] - 19s 777ms/step - loss: 17.8811 - val_loss: 17.1414
Epoch 77/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6542 - val_loss: 18.0857
Epoch 78/100
24/24 [==============================] - 19s 780ms/step - loss: 17.8937 - val_loss: 17.0371
Epoch 79/100
24/24 [==============================] - 19s 777ms/step - loss: 17.6179 - val_loss: 23.6756
Epoch 80/100
24/24 [==============================] - 19s 776ms/step - loss: 17.4508 - val_loss: 17.0178
Epoch 81/100
24/24 [==============================] - 19s 780ms/step - loss: 17.7438 - val_loss: 16.7333
Epoch 82/100
24/24 [==============================] - 19s 776ms/step - loss: 17.7676 - val_loss: 67.6749
Epoch 83/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1856 - val_loss: 17.5176
Epoch 84/100
24/24 [==============================] - 19s 777ms/step - loss: 17.1831 - val_loss: 16.8703
Epoch 85/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9395 - val_loss: 18.3559
Epoch 86/100
24/24 [==============================] - 19s 777ms/step - loss: 17.4497 - val_loss: 17.6610
Epoch 87/100
24/24 [==============================] - 19s 777ms/step - loss: 16.7546 - val_loss: 18.7588
Epoch 88/100
24/24 [==============================] - 19s 775ms/step - loss: 17.3181 - val_loss: 17.9550
Epoch 89/100
24/24 [==============================] - 19s 777ms/step - loss: 16.6001 - val_loss: 15.6468
Epoch 90/100
24/24 [==============================] - 19s 779ms/step - loss: 16.9396 - val_loss: 17.5748
Epoch 91/100
24/24 [==============================] - 19s 776ms/step - loss: 16.6347 - val_loss: 17.1885
Epoch 92/100
24/24 [==============================] - 19s 775ms/step - loss: 17.0368 - val_loss: 16.5268
Epoch 93/100
24/24 [==============================] - 19s 777ms/step - loss: 16.3248 - val_loss: 16.6590
Epoch 94/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4596 - val_loss: 15.1912
Epoch 95/100
24/24 [==============================] - 19s 777ms/step - loss: 16.9770 - val_loss: 15.4842
Epoch 96/100
24/24 [==============================] - 19s 776ms/step - loss: 16.4053 - val_loss: 17.7234
Epoch 97/100
24/24 [==============================] - 19s 777ms/step - loss: 16.1250 - val_loss: 16.5083
Epoch 98/100
24/24 [==============================] - 19s 777ms/step - loss: 16.4032 - val_loss: 15.9407
Epoch 99/100
24/24 [==============================] - 19s 776ms/step - loss: 16.2643 - val_loss: 16.7436
Epoch 100/100
24/24 [==============================] - 19s 777ms/step - loss: 15.9776 - val_loss: 16.2618
I train yolo on raccon dataset. After 100 epoch, i still can not detect any object.

I have the same problem can you share your solution?

have you solved this problem now?

[wanghzhh]

Finally, I succeed in training on VOC dataset which only recognize bicycle.
I think the root cause is my train.txt didn't contain any valid Box data, what it has is only image_file_path, To be more detailed, a standard row format of train.txt required by this repo is:
"image_file_path box1 box2 ... boxN"
but my train.txt is like this:
"image_file_path"
this repo has already provide annotation file(e.g. voc_annotation.py) to generate train.txt for us, this script will generate 2007_test.txt,2007_train.txt and 2007_val.txt, what extra we need to do is cat them together to get a train.txt.
what a stupid mistake. Currently, the core is not so high as my epochs is only 4, so I will train more to see if I can get a better score.

Sorry, what's the meaning of 'cat'? How to cat the three txt together to get a train.txt?

[2206487433]

I know what's going on, because we didn't run the voc_annotation. Py file to generate the right TXT file, right
Correct TXT format is: / VOCdevkit VOC2007 JPEGImages / 1000. JPG,0,512,111,2 2274217489, 1, 136104232187, 2 380