Keras-yolo3: Error after training

[Exorcismus]

Train on 66 samples, val on 7 samples, with batch size 32.
Epoch 1/50
2/2 [==============================] - 977s 489s/step - loss: 7732.9583 - val_loss: 6391.2349
Epoch 2/50 2/2 [==============================] - 553s 277s/step - loss: 6053.7227 - val_loss: 4881.7363
Epoch 3/50 2/2 [==============================] - 570s 285s/step - loss: 4689.0405 - val_loss: 3795.5706 Epoch 4/50
2/2 [==============================] - 533s 266s/step - loss: 3626.1056 - val_loss: 3024.2988
Epoch 5/50
2/2 [==============================] - 538s 269s/step - loss: 2811.5165 - val_loss: 2315.7861
Epoch 6/50
2/2 [==============================] - 540s 270s/step - loss: 2182.8245 - val_loss: 1855.2218
Epoch 7/50
2/2 [==============================] - 654s 327s/step - loss: 1700.2272 - val_loss: 1430.1632
Epoch 8/50
2/2 [==============================] - 574s 287s/step - loss: 1344.6483 - val_loss: 1096.7966
Epoch 9/50
2/2 [==============================] - 540s 270s/step - loss: 1068.5819 - val_loss: 929.7747
Epoch 10/50
2/2 [==============================] - 544s 272s/step - loss: 877.4108 - val_loss: 746.1503
Epoch 11/50
2/2 [==============================] - 536s 268s/step - loss: 726.8871 - val_loss: 617.6332
Epoch 12/50
2/2 [==============================] - 515s 258s/step - loss: 605.9974 - val_loss: 558.9324
Epoch 13/50
2/2 [==============================] - 537s 269s/step - loss: 523.0384 - val_loss: 461.3741
Epoch 14/50
2/2 [==============================] - 533s 266s/step - loss: 445.4352 - val_loss: 417.4197
Epoch 15/50
2/2 [==============================] - 503s 251s/step - loss: 397.8133 - val_loss: 357.0779
Epoch 16/50
2/2 [==============================] - 521s 260s/step - loss: 357.9242 - val_loss: 335.8514
Epoch 17/50
2/2 [==============================] - 507s 253s/step - loss: 313.6030 - val_loss: 294.7192
Epoch 18/50
2/2 [==============================] - 509s 255s/step - loss: 297.7372 - val_loss: 265.9477
Epoch 19/50
2/2 [==============================] - 506s 253s/step - loss: 266.4230 - val_loss: 244.6686
Epoch 20/50
2/2 [==============================] - 530s 265s/step - loss: 246.8596 - val_loss: 217.8799
Epoch 21/50
2/2 [==============================] - 577s 289s/step - loss: 233.8142 - val_loss: 217.8014
Epoch 22/50
2/2 [==============================] - 547s 274s/step - loss: 220.6562 - val_loss: 205.1769
Epoch 23/50
2/2 [==============================] - 506s 253s/step - loss: 204.7860 - val_loss: 198.4011
Epoch 24/50
2/2 [==============================] - 506s 253s/step - loss: 191.4808 - val_loss: 186.7932
Epoch 25/50
2/2 [==============================] - 530s 265s/step - loss: 185.1028 - val_loss: 181.4433
Epoch 26/50
2/2 [==============================] - 527s 264s/step - loss: 181.1230 - val_loss: 170.5103
Epoch 27/50
2/2 [==============================] - 517s 259s/step - loss: 170.9533 - val_loss: 166.2399
Epoch 28/50
2/2 [==============================] - 520s 260s/step - loss: 169.5441 - val_loss: 163.5740
Epoch 29/50
2/2 [==============================] - 526s 263s/step - loss: 163.3225 - val_loss: 161.3373
Epoch 30/50
2/2 [==============================] - 513s 256s/step - loss: 161.5418 - val_loss: 153.9463
Epoch 31/50
2/2 [==============================] - 604s 302s/step - loss: 149.0465 - val_loss: 151.6442
Epoch 32/50
2/2 [==============================] - 539s 270s/step - loss: 147.4427 - val_loss: 142.8413
Epoch 33/50
2/2 [==============================] - 527s 263s/step - loss: 141.4977 - val_loss: 149.4282
Epoch 34/50
2/2 [==============================] - 540s 270s/step - loss: 142.5769 - val_loss: 141.4876
Epoch 35/50
2/2 [==============================] - 532s 266s/step - loss: 136.8358 - val_loss: 138.6275
Epoch 36/50
2/2 [==============================] - 514s 257s/step - loss: 133.7030 - val_loss: 127.9899
Epoch 37/50
2/2 [==============================] - 521s 260s/step - loss: 131.6795 - val_loss: 130.0814
Epoch 38/50
2/2 [==============================] - 528s 264s/step - loss: 127.5341 - val_loss: 132.4926
Epoch 39/50
2/2 [==============================] - 519s 259s/step - loss: 124.0334 - val_loss: 115.9724
Epoch 40/50
2/2 [==============================] - 554s 277s/step - loss: 121.7912 - val_loss: 120.3269
Epoch 41/50
2/2 [==============================] - 518s 259s/step - loss: 118.9853 - val_loss: 116.2015
Epoch 42/50
2/2 [==============================] - 522s 261s/step - loss: 115.5685 - val_loss: 109.4524
Epoch 43/50
2/2 [==============================] - 524s 262s/step - loss: 115.7151 - val_loss: 116.0832
Epoch 44/50
2/2 [==============================] - 541s 271s/step - loss: 113.5680 - val_loss: 115.8116
Epoch 45/50
2/2 [==============================] - 514s 257s/step - loss: 109.5275 - val_loss: 110.3425
Epoch 46/50
2/2 [==============================] - 511s 255s/step - loss: 107.7783 - val_loss: 103.5918
Epoch 47/50
2/2 [==============================] - 516s 258s/step - loss: 109.1801 - val_loss: 102.2711
Epoch 48/50
2/2 [==============================] - 510s 255s/step - loss: 101.3820 - val_loss: 109.0947
Epoch 49/50
2/2 [==============================] - 508s 254s/step - loss: 104.4941 - val_loss: 99.4324
Epoch 50/50
2/2 [==============================] - 506s 253s/step - loss: 98.6054 - val_loss: 95.9730
Unfreeze all of the layers.
Train on 66 samples, val on 7 samples, with batch size 32.
Epoch 51/100
2018-11-25 08:22:11.334867: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[32,52,52,256] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
Traceback (most recent call last):
File "user\tensorflow\python\client\session.py", line 1327, in _do_call
return fn(*args)
File "user\tensorflow\python\client\session.py", line 1312, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "user\tensorflow\python\client\session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "user\tensorflow\python\framework\errors_impl.py", line 516, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,52,52,256] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: leaky_re_lu_22/LeakyRelu/mul = MulT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_clothes.py", line 190, in
_main()
File "train_clothes.py", line 84, in _main
callbacks=[logging, checkpoint, reduce_lr, early_stopping])
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(args, *kwargs)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
class_weight=class_weight)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
outputs = self.train_function(ins)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2721, in __call__
return self._legacy_call(inputs)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 2693, in _legacy_call
**self.session_kwargs)
File "user\tensorflow\python\client\session.py", line 905, in run
run_metadata_ptr)
File "user\tensorflow\python\client\session.py", line 1140, in _run
feed_dict_tensor, options, run_metadata)
File "user\tensorflow\python\client\session.py", line 1321, in _do_run
run_metadata)
File "user\tensorflow\python\client\session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,52,52,256] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: leaky_re_lu_22/LeakyRelu/mul = MulT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'leaky_re_lu_22/LeakyRelu/mul', defined at:
File "train_clothes.py", line 190, in
_main()
File "train_clothes.py", line 33, in _main
freeze_body=2, weights_path='model_data/yolo_weights.h5') # make sure you know what you freeze
File "train_clothes.py", line 116, in create_model
model_body = yolo_body(image_input, num_anchors//3, num_classes)
File "D:\WORK\PROJECTS\Repository\keras-yolo3\yolo3\model.py", line 72, in yolo_body
darknet = Model(inputs, darknet_body(inputs))
File "D:\WORK\PROJECTS\Repository\keras-yolo3\yolo3\model.py", line 51, in darknet_body
x = resblock_body(x, 256, 8)
File "D:\WORK\PROJECTS\Repository\keras-yolo3\yolo3\model.py", line 42, in resblock_body
DarknetConv2D_BN_Leaky(num_filters, (3,3)))(x)
File "D:\WORK\PROJECTS\Repository\keras-yolo3\yolo3\utils.py", line 16, in
return reduce(lambda f, g: lambda a, *kw: g(f(a, *kw)), funcs)
File "D:\WORK\PROJECTS\Repository\keras-yolo3\yolo3\utils.py", line 16, in
return reduce(lambda f, g: lambda a, *kw: g(f(a, *kw)), funcs)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\engine\base_layer.py", line 457, in __call__
output = self.call(inputs, **kwargs)_activations.py", line 48, in call
return K.relu(inputs, alpha=self.alpha)
File "C:\ProgramDataAnaconda3\lib\site-packages\keras\backend\tensorflow_backend.py", line 3173, in relu
return tf.nn.leaky_relu(x, alpha=alpha)
File "user\tensorflow\python\ops\nn_ops.py", line 1608, in leaky_relu
return math_ops.maximum(alpha * features, features)
File "user\tensorflow\python\ops\math_ops.py", line 971, in binary_op_wrapper
return func(x, y, name=name)
File "user\tensorflow\python\ops\math_ops.py", line 1198, in _mul_dispatch
return gen_math_ops.mul(x, y, name=name)
File "user\tensorflow\python\ops\gen_math_ops.py", line 4991, in mul
"Mul", x=x, y=y, name=name)
File "user\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "user\tensorflow\python\framework\ops.py", line 3290, in create_op
op_def=op_def)
File "user\tensorflow\python\framework\ops.py", line 1654, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[32,52,52,256] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
[[Node: leaky_re_lu_22/LeakyRelu/mul = MulT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

File "C:\ProgramDataAnaconda3\lib\site-packages\keras\layers\advanced

[phamngocthanhtrung]

Hi, did you solve your problem?

[zhuchenxi]

@Exorcismus I think you do not have enough memory for training. Try to reduce the batch size?

[peraktong]

Hi. I have the same problem. I think you can change the batch size from 32 to 16(There are two batch sizes, one for stage 1 and another for stage 2).
I'm using GTX 1070 Ti with 8GB VRAM and now it works.