Keras-yolo3: why i run train.py is ok,but run train_bottleneck is wrong:loss:nan,val_loss:nan

[AryaCao]

run the train_bottleneck :
start like this:

later change to this:

can someone tell me why?

[AryaCao]

It seems to be that the parameters:steps of " bottleneck_model.predict_generator" is wrong?

[Fingerl]

hello，can you tell me the different between the train.py and the train_bottleneck.py?Which one should I use?

[AryaCao]

hello , I think that the two file has same function.
The 'train_bottlenck.py' can speed up the training process in cpu or gpu, because it will compute the bottleneck features of the frozen model first(saved in the .npy files) and then load the features to train the last layers.
I think that you can just use the train.py, if you have a gpu (good compute capability ).

[Fingerl]

thank you

[matt-deboer]

I was able to fix this by adding the following lines around 383 of yolo3/model.py:
_(right below the existing K.switch... statement)_

# filter out inf/nan values encountered here:
raw_true_wh = K.switch(tf.is_inf(raw_true_wh), raw_true_wh, K.zeros_like(raw_true_wh)) 
raw_true_wh = K.switch(tf.is_nan(raw_true_wh), raw_true_wh, K.zeros_like(raw_true_wh))

This is a bit of a hack; there may be some data-related causes for those inf/nan values to occur, but I wasn't able to find them in my data set; then again, it can't hurt to be less sensitive to bogus data, right?

[CherishineNi]

@AryaCao 请问如何使用gpu运行train.py文件呢？需要额外的指令代码吗？

[monicatao]

I have the same problem. How did you solve it?