Pytorch-yolov3: RuntimeError: CUDA error: device-side assert triggered

[JisongXie]

I use the code to train on my custom dataset. All input images size are 608 x 608, and there are 20 classes, each image has only a bounding box. I follow the structions to custom model and dataset. And I meet a training error，
RuntimeError: CUDA error: device-side assert triggered.
The error occurs uncertainly, for example, in Epoch 2 Batch 1200, or in Epoch 3 Batch 2000, while the previous epoch can finish training fluently.
Following pictures are the detailed error.

It seems that it occurs in build_targets function in model.py, the build_target function is in the utils/utils.py. This error is so difficult to debug!

[readicculus]

I'm getting same exact error ill update if i figure out issue

[JisongXie]

It seems clearly, that the error is index out of bounds, and that it occurs on predictions. Maybe sometimes the output predictions are empty and the code doesn't take this into account. I need time to understand the whole pipeline clearly to fix the bug.

[ghimiredhikura]

I'm also getting exact same error on my custom dataset. this is index out of bound error, will update if i solve it.

[readicculus]

I fixed it... i wasn't normalizing my inputs. I have my own data loader and need to input yolo format
cetner_x,center_y,w,h in [0,1] for labels. I was using pixel values.

[MaTao215]

I met the same problem too.Have you fixed it?

[QQ2737499951]

I met the same problem too.Have you fixed it?

"D:Program FilesPython368python.exe" B:/PyTorch/PyTorch-YOLOv3/train.py
Namespace(batch_size=8, checkpoint_interval=1, compute_map=False, data_config='data/coco.data', epochs=100, evaluation_interval=1, gradient_accumulations=2, img_size=416, model_def='cfg/yolov3.cfg', multiscale_training=True, n_cpu=8, pretrained_weights=None)
data/coco/train.txt data/coco/val.txt
data/coco/images/train.jpg
data/coco/images/000000000034.jpg
data/coco/images/000000000030.jpg
data/coco/images/000000000042.jpg
data/coco/images/000000000049.jpg
data/coco/images/000000000025.jpg
data/coco/images/000000000036.jpg
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: block: [0,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "B:/PyTorch/PyTorch-YOLOv3/train.py", line 107, in
loss, outputs = model(imgs, targets)
File "D:Program FilesPython368libsite-packagestorchnnmodulesmodule.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "B:PyTorchPyTorch-YOLOv3models.py", line 259, in forward
x, layer_loss = module0
File "D:Program FilesPython368libsite-packagestorchnnmodulesmodule.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "B:PyTorchPyTorch-YOLOv3models.py", line 188, in forward
ignore_thres=self.ignore_thres,
File "B:PyTorchPyTorch-YOLOv3utilsutils.py", line 317, in build_targets
class_mask[b, best_n, gj, gi] = (pred_cls[b, best_n, gj, gi].argmax(-1) == target_labels).float()
RuntimeError: CUDA error: device-side assert triggered

[dongfangduoshou123]

It seems clearly, that the error is index out of bounds, and that it occurs on predictions. Maybe sometimes the output predictions are empty and the code doesn't take this into account. I need time to understand the whole pipeline clearly to fix the bug.

Hi,
Have you fixed this bug?

[SmartGx]

It seems clearly, that the error is index out of bounds, and that it occurs on predictions. Maybe sometimes the output predictions are empty and the code doesn't take this into account. I need time to understand the whole pipeline clearly to fix the bug.

I met the same problem.Have you fixed it?

[Arthur77Wang]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

[BloomBerry]

same problem too.
---- [Epoch 0/100, Batch 0/14786] ----
+------------+--------------+--------------+--------------+
| Metrics | YOLO Layer 0 | YOLO Layer 1 | YOLO Layer 2 |
+------------+--------------+--------------+--------------+
| grid_size | 13 | 26 | 52 |
| loss | 71.552628 | 69.287781 | 70.698685 |
| x | 0.081982 | 0.095761 | 0.097411 |
| y | 0.114656 | 0.081447 | 0.083638 |
| w | 2.447639 | 0.842779 | 1.058129 |
| h | 2.727762 | 1.390022 | 1.789845 |
| conf | 65.440392 | 66.154922 | 66.964203 |
| cls | 0.740200 | 0.722845 | 0.705461 |
| cls_acc | 1.89% | 0.00% | 1.47% |
| recall50 | 0.000000 | 0.000000 | 0.000000 |
| recall75 | 0.000000 | 0.000000 | 0.000000 |
| precision | 0.000000 | 0.000000 | 0.000000 |
| conf_obj | 0.463119 | 0.486062 | 0.475699 |
| conf_noobj | 0.466011 | 0.474434 | 0.481249 |
+------------+--------------+--------------+--------------+
Total loss 211.53909301757812
---- ETA 3 days, 18:36:37.682694
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=59 : device-side assert triggered
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [69,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "/home/jang/Downloads/pycharm-community-2018.3.1/helpers/pydev/pydevd.py", line 1741, in
main()
File "/home/jang/Downloads/pycharm-community-2018.3.1/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/jang/Downloads/pycharm-community-2018.3.1/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/jang/Downloads/pycharm-community-2018.3.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"n", file, 'exec'), glob, loc)
File "/home/jang/PycharmProjects/PyTorch-YOLOv3/train.py", line 105, in
loss, outputs = model(imgs, targets)
File "/home/jang/virtualenv/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/home/jang/PycharmProjects/PyTorch-YOLOv3/models.py", line 259, in forward
x, layer_loss = module0
File "/home/jang/virtualenv/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(input, *kwargs)
File "/home/jang/PycharmProjects/PyTorch-YOLOv3/models.py", line 188, in forward
ignore_thres=self.ignore_thres,
File "/home/jang/PycharmProjects/PyTorch-YOLOv3/utils/utils.py", line 324, in build_targets
iou_scores[b, best_n, gj, gi] = bbox_iou(pred_boxes[b, best_n, gj, gi], target_boxes, x1y1x2y2=False)
File "/home/jang/PycharmProjects/PyTorch-YOLOv3/utils/utils.py", line 199, in bbox_iou
b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2
RuntimeError: CUDA error: device-side assert triggered

[kaikaizhu]

@BloomBerry
I met the same problem.Have you fixed it?

[zhaopeng0103]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

I met the same problem, too. But after I use the method mentioned above, the error does not appear. Thank you!

[susanhc]

解决方法参考：https://blog.csdn.net/susanhc/article/details/93876762

[kaikaizhu]

When I ran the program with a single gpu, everything was fine, but when I joined multiple gpus, I made this mistake, and my label started at 0.

[khgapa]

@kaikaizhu When I use multiple gpus, I have same problem.

[qwe3208620]

I encountered this problem when reducing the coco detection category, which has not been solved yet.

[qwe3208620]

@kaikaizhu
请问你是怎么确认的label从0开始了 我减少训练类别 没有改动代码 为什么label也会溢出呢

[LifeIsSoSolong]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

I solve this problem using your methed, thanks

[SWHL]

I meet the same problem.

/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:53: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "/home/test/SWHL/MetaPyTorch-YOLOv3-master/train.py", line 125, in <module>
    loss, outputs = model(imgs, targets)
  File "/home/test/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/test/SWHL/MetaPyTorch-YOLOv3-master/models.py", line 311, in forward
    x, layer_loss = module[0](x, targets, img_dim)
  File "/home/test/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/test/SWHL/MetaPyTorch-YOLOv3-master/models.py", line 233, in forward
    ignore_thres=self.ignore_thres,
  File "/home/test/SWHL/MetaPyTorch-YOLOv3-master/utils/utils.py", line 301, in build_targets
    obj_mask[b, best_n, gj, gi] = 1
RuntimeError: CUDA error: device-side assert triggered

[kirilcvetkov92]

I found a small workaround here :

Replace :

 obj_mask[b, best_n, gj, gi] = 1
 noobj_mask[b, best_n, gj, gi] = 0

with :

gi = torch.clamp(gi, 0, noobj_mask.size()[2]-1)
gj = torch.clamp(gj, 0, noobj_mask.size()[3]-1)
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

[NeuralFlux]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

Solved my problem, thank you so much!

[GangHu1993]

When i used multi-gpu, I have same problem. When I used single gpu, no problem.

[chenhuanan]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

The root cause of the problem is that there is no correct label format, not this one. If you solve the problem as above, you will not be able to train the desired result

[zhudd-hub]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~
优秀

[xiaoxiaoshu111]

When i used multi-gpu, I have same problem. When I used single gpu, no problem.

I also have the same problem... Did you solve it...

[bchugg]

I had the same problem on the ultralytics yolov3 implementation. The issue for me was that, even while my bounding box coordinates were normalized (between 0 and 1), the coordinates sometimes still specified a box that fell outside the image. For example, if we have

x_center = 0.88 
y_center = 0.27 
width = 0.2
height = 0.3 

then this was throwing an error because the right side of the bounding box has x coordinate greater than one, i.e., x_center + width/2 > 1.

Hope this helps someone!

[ujsyehao]

you can refer to https://github.com/ujsyehao/yolov3-multigpu

[TonyAlpha12]

When I ran the program with a single gpu, everything was fine, but when I joined multiple gpus, I made this mistake, and my label started at 0.

I meet the sample problem too,have you fix it?

[TonyAlpha12]

When i used multi-gpu, I have same problem. When I used single gpu, no problem.

你解决这个问题了吗，我的也是单卡跑没毛病，多卡一跑就报错了：noobj_mask[b[i], anchor_ious > ignore_thres, gj[i], gi[i]] = 0 RuntimeError: CUDA error: device-side assert triggered

[hlchen1043]

Exactly same problem here, training on custom data, single GPU, also report the problem after start running it at some time

[AlvinWen428]

I have solved this issue. Please check your labels and make sure that the normalized labels must satisfy 0< x_center +/- width <1 and 0 < y_center +/- height <1. Note that it must be $<$, not $<=$.

[zxin8218]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

I solve this problem using your methed too, thank you

[JessonKang]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

I solved this problem too，thank you very much!

[mama110]

check labels and the problem will be solved.

[bisonliao]

I met this problem too.
create_custom_model.sh with great class number than actual number can solve it.
COCO dataset has 80 classes. I create_custom_model 90 and generate yolo3-bison.cfg.
then I train with yolo3-bison.cfg and coco dataset , it can avoid this problem

[bisonliao]

I met this problem too.
create_custom_model.sh with great class number than actual number can solve it.
COCO dataset has 80 classes. I create_custom_model 90 and generate yolo3-bison.cfg.
then I train with yolo3-bison.cfg and coco dataset , it can avoid this problem

sorry. for me, it is because my label files contain improper class indices.

[sharoseali]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~
优秀

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

I solved this problem too，thank you very much!

Can you please explain? At which exact line number, these code lines should be added?

[hongjianyuan]

在Ultralytics yolov3的实现上，我遇到了

在Ultralytics yolov3的实现上，我遇到了同样的问题。对我来说，问题是，即使我的边界框坐标已标准化（介于0和1之间），有时坐标仍会指定一个超出图像范围的框。例如，如果我们有

x_center = 0.88 
y_center = 0.27 
width = 0.2
height = 0.3 

那么这将引发错误，因为边界框的右侧的x坐标大于1，即x_center + width / 2> 1。

希望这对某人有帮助！

How to standardize the data, I have now made the data (center_x, center_y, w, h)

[RRL-dev]

Came across too. Maybe it's because of lacking some boundary checking when the code was parsing your label files. Several ground truth boxes might partly outside image after +-*/.... I added rude boundary assignments in utils/utils.py, build_targets func:

b, target_labels = target[:, :2].long().t()
gx, gy = gxy.t()
gw, gh = gwh.t()
gi, gj = gxy.long().t()
########## TODO(arthur77wang):
gi[gi < 0] = 0
gj[gj < 0] = 0
gi[gi > nG - 1] = nG - 1
gj[gj > nG - 1] = nG - 1
###################
# Set masks
obj_mask[b, best_n, gj, gi] = 1
noobj_mask[b, best_n, gj, gi] = 0

This aims to avoid gi[i] and gj[i] exceeding bound of size[2, 3] of noobj_mask. You would make some more precise corrections~

Thank you. A real issue for out of image bounding box. Great and easy solution.

[Flova]

This happens when the bounding box is outside the image.
Reasons are, that the box coordinates are not normalized (between 0 and 1) or the box is normalized but at a position like 1.5.
An upcoming image augmentation pull request will likely result in the discarding of out of image labels as a side effect.
I think the issue is solved and therefore close it. Feel free to reopen for further questions,