Darkflow: NaN training loss after first iteration

Created on 2 Feb 2018 · 6Comments · Source: thtrieu/darkflow

I am attempting to further train the yolo network on the coco dataset as a starting point for "training my own data." I converted the coco json into .xml annotations, but when I try to train, I get NaN loss starting at the second step. Most issues regarding NaN loss seem to be centered around incorrect annotations, however, I have checked over mine multiple times for correctness. I have copied coco.names and overwritten labels.txt with it.

I use the following command to train:
./flow --model cfg/yolo.cfg --load bin/yolo.weights --train --dataset /path/to/JPEGImages/ --annotation /path/to/Annotations/ --gpu 0.95

I get the following when training:

Training statistics: 
        Learning rate : 1e-05
        Batch size    : 16
        Epoch number  : 1000
        Backup every  : 2000
step 1 - loss 9.513481140136719 - moving ave loss 9.51348114013672
step 2 - loss nan - moving ave loss nan
step 3 - loss nan - moving ave loss nan

I've read in other issues that there is a parse history that one can delete if there was previously a mistake in the annotations before, but I cannot find it. Looking for any ideas on reason for this to happen.

Source

matlabninja

Most helpful comment

A lot of NaN problems can actually be fixed by clipping the gradients here:
https://github.com/thtrieu/darkflow/blob/master/darkflow/net/help.py#L18

thtrieu on 7 Feb 2018

👍2 ❤1 🎉1

All 6 comments

I'm also having the same issue as above, except, I've reached step 46K before it started happening. No error is showing when I start my training, and previous issues provide next to no instructions on how to fix this problem.

I don't necessarily want to open a new issue, however, I'm training my network on a custom dataset with 3 labels using a copy of the tiny-yolo-voc weights.

My command to train is as follows:
./flow --model cfg/tiny-yolo-voc-3c.cfg --load -1 --train --annotation train/annotations --dataset train/images

Here's a preview of when it started happening (it never happened prior to these steps):

step 41734 - loss 1.8608825206756592 - moving ave loss 1.2506067808148458
step 41735 - loss 0.8213188052177429 - moving ave loss 1.2076779832551356
step 41736 - loss 0.8962286710739136 - moving ave loss 1.1765330520370134
step 41737 - loss 1.6367404460906982 - moving ave loss 1.222553791442382
step 41738 - loss 0.7614827156066895 - moving ave loss 1.1764466838588128
step 41739 - loss nan - moving ave loss nan
step 41740 - loss nan - moving ave loss nan
step 41741 - loss nan - moving ave loss nan
libpng warning: iCCP: known incorrect sRGB profile
step 41742 - loss nan - moving ave loss nan
step 41743 - loss nan - moving ave loss nan
step 41744 - loss nan - moving ave loss nan
step 41745 - loss nan - moving ave loss nan

I get the following when initiating training:

```Parsing cfg/tiny-yolo-voc-3c.cfg
Loading None ...
Finished in 6.341934204101562e-05s

Enter training ...

cfg/tiny-yolo-voc-3c.cfg parsing train/annotations
Parsing for ['car', 'person', 'traffic_light']
[====================>]100% car_00000575.xmlml
Statistics:
traffic_light: 36
person: 787
car: 679
Dataset size: 1025
Dataset of 1025 instance(s)
Training statistics:
Learning rate : 1e-05
Batch size : 16
Epoch number : 1000
Backup every : 2000
step 45501 - loss nan - moving ave loss nan
step 45502 - loss nan - moving ave loss nan
step 45503 - loss nan - moving ave loss nan
step 45504 - loss nan - moving ave loss nan
```
I don't feel it's necessary to provide my config, as it's just a copy of tiny-yolo-voc.weights with the recommended changes to make it work with my custom dataset.

Let me know if I can provide any more details - hopefully we can get this issue sorted.

illQuo on 6 Feb 2018

Might also be related to #383
Looking into the code for using yolo.cfg looks like the nan is coming straight out of the call to Tensorflow on line 56 of flow.py
fetched = self.sess.run(fetches, feed_dict) loss = fetched[1]
It also appears that yolo.cfg with coco labels calls the YOLOv2 constructor. The loss function for the network is given in "darkflow/net/yolov2/train.py" I don't see anything at first glance that looks like it could go NaN, but it's where we would have to start.

matlabninja on 6 Feb 2018

A lot of NaN problems can actually be fixed by clipping the gradients here:
https://github.com/thtrieu/darkflow/blob/master/darkflow/net/help.py#L18

thtrieu on 7 Feb 2018

👍2 ❤1 🎉1

Hi @thtrieu ,

I'm fairly new to darkflow and deep learning in general. By terms of clipping the gradients, how would you suppose I, a novice, go about this?

Any resources I can read or examples I can follow - I'm not exactly where to start with regards to clipping gradients with darkflow.

Thanks.

illQuo on 8 Feb 2018

My problem at least has gone away. I was comparing yolo.cfg to yolo-voc.cfg, since I was able to successfully train on the voc dataset with yolo-voc. There is a convolutional layer in yolo.cfg that is not present in yolo-voc.cfg starting at line 211 in yolo.cfg. I removed this layer and edited the route layer (formerly at line 222, now at 214) to
[route] layers=-1,-3
I also changed random to 0 at the end of the file.
I am now training just fine. I can see zero gradients becoming an issue further along in training, but I have doubts that gradient clipping would have solved the "second iteration" issue that I was having. I personally do not know why this issue would have been caused by that convolutional layer. Closing for now, hopefully anyone else who has the immediate nan problem like me can edit similarly, though I would welcome commentary on why that particular layer caused me such troubles.

matlabninja on 13 Feb 2018

Hey @illQuo, if you are still unsure about how to clip gradients, here's how I did it:

gradients = optimizer.compute_gradients(self.framework.loss)
clipped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients]
self.train_op = optimizer.apply_gradients(gradients)

Even after clipped the gradients, the problem still persists. Can anyone suggest things I can try?