Pytorch-yolov3: RuntimeError: CUDA out of memory after some epochs

[anirbansen3027]

I am running this on my own data set. It fails even at a batch size of 2 images.
It runs for some epochs:
Namespace(batch_size=2, checkpoint_interval=1, compute_map=False, data_config='config/custom.data', epochs=1, evaluation_interval=1, gradient_accumulations=2, img_size=416, model_def='yolov3-custom.cfg', multiscale_training=True, n_cpu=8, pretrained_weights='weights/darknet53.conv.74')

---- [Epoch 0/1, Batch 0/1518] ----
+------------+--------------+--------------+--------------+
| Metrics | YOLO Layer 0 | YOLO Layer 1 | YOLO Layer 2 |
+------------+--------------+--------------+--------------+
| grid_size | 13 | 26 | 52 |
| loss | 72.304230 | 72.418373 | 75.622971 |
| x | 0.089580 | 0.078698 | 0.089062 |
| y | 0.086233 | 0.084770 | 0.087637 |
| w | 2.158209 | 0.264621 | 0.168304 |
| h | 1.217486 | 1.044052 | 0.169049 |
| conf | 67.966827 | 70.160515 | 74.447456 |
| cls | 0.785900 | 0.785719 | 0.661462 |
| cls_acc | 100.00% | 100.00% | 100.00% |
| recall50 | 0.000000 | 0.072993 | 0.521429 |
| recall75 | 0.000000 | 0.000000 | 0.078571 |
| precision | 0.000000 | 0.005249 | 0.006851 |
| conf_obj | 0.547704 | 0.466817 | 0.518774 |
| conf_noobj | 0.477222 | 0.494759 | 0.519431 |
+------------+--------------+--------------+--------------+
Total loss 220.34556579589844
---- ETA 2:10:34.914327

---- [Epoch 0/1, Batch 1/1518] ----
+------------+--------------+--------------+--------------+
| Metrics | YOLO Layer 0 | YOLO Layer 1 | YOLO Layer 2 |
+------------+--------------+--------------+--------------+
| grid_size | 13 | 26 | 52 |
| loss | 71.990082 | 72.543518 | 75.521194 |
| x | 0.080743 | 0.095078 | 0.088256 |
| y | 0.117081 | 0.101007 | 0.098819 |
| w | 1.255917 | 0.191129 | 0.163498 |
| h | 1.631940 | 1.271208 | 0.203247 |
| conf | 68.130760 | 70.126694 | 74.384293 |
| cls | 0.773645 | 0.758403 | 0.583087 |
| cls_acc | 100.00% | 100.00% | 100.00% |
| recall50 | 0.037500 | 0.072917 | 0.587629 |
| recall75 | 0.000000 | 0.000000 | 0.092784 |
| precision | 0.006849 | 0.003755 | 0.005338 |
| conf_obj | 0.568219 | 0.472614 | 0.537714 |
| conf_noobj | 0.479578 | 0.494576 | 0.519314 |
+------------+--------------+--------------+--------------+
Total loss 220.05479431152344
---- ETA 1:14:06.976984

---- [Epoch 0/1, Batch 2/1518] ----
+------------+--------------+--------------+--------------+
| Metrics | YOLO Layer 0 | YOLO Layer 1 | YOLO Layer 2 |
+------------+--------------+--------------+--------------+
| grid_size | 10 | 20 | 40 |
| loss | 67.711563 | 63.988781 | 70.632484 |
| x | 0.104312 | 0.085154 | 0.095351 |
| y | 0.088750 | 0.095203 | 0.101118 |
| w | 2.285418 | 0.367559 | 0.142635 |
| h | 2.019693 | 1.379491 | 0.135650 |
| conf | 62.620461 | 61.348545 | 69.552940 |
| cls | 0.592924 | 0.712830 | 0.604792 |
| cls_acc | 100.00% | 100.00% | 100.00% |
| recall50 | 0.155844 | 0.034483 | 0.482759 |
| recall75 | 0.012987 | 0.000000 | 0.057471 |
| precision | 0.068571 | 0.005155 | 0.009556 |
| conf_obj | 0.482500 | 0.434160 | 0.501441 |
| conf_noobj | 0.397941 | 0.448465 | 0.495753 |
+------------+--------------+--------------+--------------+
Total loss 202.33282470703125
---- ETA 0:54:49.954170

But then there is this error:
Traceback (most recent call last):
File "train.py", line 106, in
loss, outputs = model(imgs, targets)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(input, *kwargs)
File "/kaggle/working/PyTorch-YOLOv3/models.py", line 252, in forward
x = module(x)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(input, *kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(input, *kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
exponential_average_factor, self.eps)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 1923, in batch_norm
training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.90 GiB total capacity; 2.13 GiB already allocated; 19.88 MiB free; 2.14 GiB reserved in total by PyTorch)

Kindly help me with this

[khasebou]

have you been able to solve the error ? i get it exactly after 1 batch

[Z-XQ]

you can try comment out these code:
# Tensorboard logging
# tensorboard_log = []
# for j, yolo in enumerate(model.yolo_layers):
# for name, metric in yolo.metrics.items():
# if name != "grid_size":
# tensorboard_log += [(f"{name}_{j+1}", metric)]
# tensorboard_log += [("loss", loss.item())]
# logger.list_of_scalars_summary(tensorboard_log, batches_done)

[zczjx]

you can try comment out these code:

Tensorboard logging

tensorboard_log = []

for j, yolo in enumerate(model.yolo_layers):

for name, metric in yolo.metrics.items():

if name != "grid_size":

tensorboard_log += [(f"{name}_{j+1}", metric)]

tensorboard_log += [("loss", loss.item())]

logger.list_of_scalars_summary(tensorboard_log, batches_done)

it is really can fix my CUDA out of memory issue when training

[sbbug]

you can try comment out these code:

Tensorboard logging

tensorboard_log = []

for j, yolo in enumerate(model.yolo_layers):

for name, metric in yolo.metrics.items():

if name != "grid_size":

tensorboard_log += [(f"{name}_{j+1}", metric)]

tensorboard_log += [("loss", loss.item())]

logger.list_of_scalars_summary(tensorboard_log, batches_done)

nice

[Flova]

I closed this issue due to inactivity. Feel free to reopen for further discussion.