Faster-rcnn.pytorch: RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generated/../THCReduceAll.cuh:339

Created on 3 Apr 2018 · 35Comments · Source: jwyang/faster-rcnn.pytorch

When I decrease the number of classes into my data loader which is very similar to pascal_voc.py, I get this error. I decreased the total number of classes to 4(with background). Is there any hard-coded class number since the problem seems like indexing problem? I would like to mention that my bbox's are coming all 0 as well. I'll be glad if you take a look @jwyang .

Source

artuncF

Most helpful comment

Try delete all the cache folders and files in the data directory.

sjiang17 on 17 Apr 2018

👍3 😄1

All 35 comments

Hi, @artuncF did you check the imdb.classes are the right classes for your dataset?

jwyang on 3 Apr 2018

Yes, they are the right classes. Into the imdb.py classes are also taken as parameter.

artuncF on 3 Apr 2018

@artuncF I see, could you tell me what the exact errors you got?

jwyang on 3 Apr 2018

The error is RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generated/../THCReduceAll.cuh:339 and this occurs when I reduce the number of classes 7 to 4. Before taking this error (when the number of classes were 7), I took the error mentioned in issue#111. I printed the bounding boxes, they look all zeroes.

artuncF on 3 Apr 2018

I can send the whole error message tomorrow because I cannot access the server at this moment.

artuncF on 3 Apr 2018

👍1

('__background__', 'Car', 'Pedestrian', 'Truck')
4
Loading pretrained weights from data/pretrained_model/vgg16_caffe.pth
/home/artunc/updated_faster-rcnn/faster-rcnn.pytorch/lib/model/rpn/rpn.py:68: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
rpn_cls_prob_reshape = F.softmax(rpn_cls_score_reshape)

( 0 ,.,.) =
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
⋮
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000

( 1 ,.,.) =
0.9353 -2.0602 -1.0355 1.1521
2.4186 -0.5398 1.1839 1.1351
-0.4660 -1.6072 1.7033 -0.2188
⋮
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000

( 2 ,.,.) =
3.1466 -0.1921 0.8874 -0.5850
0.8072 -1.4267 -0.5356 -0.2854
0.1172 2.8068 -0.4330 1.0080
⋮
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000

( 3 ,.,.) =
-1.2875 -1.1610 1.5244 -1.1044
0.2461 -0.9569 1.1372 -1.6824
1.0942 -0.8118 -1.4328 0.0175
⋮
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

/home/artunc/updated_faster-rcnn/faster-rcnn.pytorch/lib/model/faster_rcnn/faster_rcnn.py:98: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
cls_prob = F.softmax(cls_score)
[session 1][epoch 1][iter 0/4608] loss: nan, lr: 1.00e-03
fg/bg=(43/981), time cost: 1.899692
rpn_cls: 0.6867, rpn_box: nan, rcnn_cls: 0.8567, rcnn_box 0.1004

( 0 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 1 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 2 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 3 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

( 0 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 1 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 2 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 3 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

( 0 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 1 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 2 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 3 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

( 0 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 1 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 2 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 3 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

( 0 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 1 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 2 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0

( 3 ,.,.) =
0 0 0 0
0 0 0 0
0 0 0 0
⋮
0 0 0 0
0 0 0 0
0 0 0 0
[torch.cuda.FloatTensor of size 4x256x4 (GPU 0)]

/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [0,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [1,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [2,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [0,0,0], thread: [3,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [0,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [1,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [2,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [3,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [4,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [5,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [6,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [7,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [8,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [9,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [10,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [2,0,0], thread: [11,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [0,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [1,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [2,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [3,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [4,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [5,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [6,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [7,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [8,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [9,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [10,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [11,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [12,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [13,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [14,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [15,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [16,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [17,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [18,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [19,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [20,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [21,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [22,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo, TensorInfo, TensorInfo, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [4,0,0], thread: [23,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
THCudaCheck FAIL file=/pytorch/torch/lib/THC/generated/../THCReduceAll.cuh line=339 error=59 : device-side assert triggered
Traceback (most recent call last):
File "trainval_net.py", line 326, in
rois_label = fasterRCNN(im_data, im_info, gt_boxes, num_boxes)
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 325, in __call__
result = self.forward(input, *kwargs)
File "/home/artunc/updated_faster-rcnn/faster-rcnn.pytorch/lib/model/faster_rcnn/faster_rcnn.py", line 108, in forward
RCNN_loss_bbox = _smooth_l1_loss(bbox_pred, rois_target, rois_inside_ws, rois_outside_ws)
File "/home/artunc/updated_faster-rcnn/faster-rcnn.pytorch/lib/model/utils/net_utils.py", line 86, in _smooth_l1_loss
loss_box = loss_box.mean()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generated/../THCReduceAll.cuh:339
The whole error message I got @jwyang. At the first part, class names, number of classes and bboxes were printed debugging purpose.

artuncF on 4 Apr 2018

@artuncF Have you fixed this error?

myungsub on 12 Apr 2018

@myungsub No. Actually if I add my class list to dummy classes, the problem seems like fixed or just not seeing this error. However, if I continue with dummy classes with my own classes, rpn_loss_box and rpn_loss_class becomes nan. Here is my class
list ('__background__', 'car', 'pedestrian', 'truck', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 't', 's', 'p').

artuncF on 12 Apr 2018

@artuncF Hmm. I had the same error this morning, but found that the data loading part of my code had a mistake. For debugging GPU-related part of the code, setting CUDA_LAUNCH_BLOCKING=1 can be very useful. (At least it lead me to the correct place of the code where I can follow and search for bugs..) You can use it like:

CUDA_LAUNCH_BLOCKING=1 python trainval_net.py --{options}

myungsub on 12 Apr 2018

👍2

Try delete all the cache folders and files in the data directory.

sjiang17 on 17 Apr 2018

👍3 😄1

Hi, I had the sane error when I train faster-rcnn, do you fix this problem?

Gwan-Siu on 13 Jun 2018

@sjiang17 what's the mean "delete all the cache folders and files in the data directory"? Pretrained model is stored in this directory.

Gwan-Siu on 13 Jun 2018

@jwyang I have the same problem when I decrease the number of classes. Do you find any reason about that? I have no idea about that.

Gwan-Siu on 13 Jun 2018

I find the solution and the problem is solved.

Gwan-Siu on 14 Jun 2018

@Gwan-Siu , how do you solve this problem?

zhyhan on 22 Jun 2018

@Gwan-Siu hi how can you solve this error? I got that too

gofortargets on 1 Jul 2018

Delete the cache

2018-07-01 12:10 GMT+08:00 Tuan Anh Nguyen notifications@github.com:

@Gwan-Siu https://github.com/Gwan-Siu hi how can you solve this error?
I got that too

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-401581686,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbLSzHsYJ1WkWKsdvEkh9rch8f6yntETks5uCEu3gaJpZM4TFP2z
.

Gwan-Siu on 3 Jul 2018

hello, what do you mean by delete the cache in the data directory ?
Which data do you mean ?

salma26 on 18 Jul 2018

I believe he means that many of the fast.ai techniques in their notebooks put temporary information (like tokens and indexed tokens) in a tmp directory in the data directory pointed to (usually with a PATH variable). So by deleting that directory and regenerating that data, he solved his problem.

andehr on 20 Jul 2018

Not quite @andehr.
Look at pickle files (*.pkl extension) saved here and there to speedup things even more.
@salma26 treat it literally, it's directory located under faster-rcnn.pytorch/data/cache plus annotations pickle file *txt_annots.pkl in the data root if Pascal VOC was used for training.

lipka-clazzpl on 20 Jul 2018

👍1

Thank you for helping. I solved the problem by decreasing the batch size and setting the numerical labels from 0 to Number_of_classes -1.

salma26 on 20 Jul 2018

@salma26 Do you mean that instead of using labels (0,1) for a binary classification, you would use (1,2)?

jvig on 1 Aug 2018

@Gwan-Siu What do you mean about deleting the cache

JoeGue on 19 Sep 2018

jvig No, instead of (1,2) you would rather set (0,1) (if you have 2 classes for exemple)

salma26 on 19 Sep 2018

try

rm faster-rcnn.pytorch/data/cache/*

solves my problem most of the times.

kentaroy47 on 22 Sep 2018

Thank you for helping, this is quite interesting to know. I solved the problem by decreasing the batch size and setting the numerical labels from 0 to Number_of_classes -1.Le 20 juil. 2018 à 13:53, Przemek Lipka notifications@github.com a écrit :Not quite @andehr.
Look at pickle files (*.pkl extension) saved here and there to speedup things even more.
@salma26 treat it literally, it's directory located under faster-rcnn.pytorch/data/cache plus annotations pickle file *txt_annots.pkl in the data root if Pascal VOC was used for training.—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/jwyang/faster-rcnn.pytorch","title":"jwyang/faster-rcnn.pytorch","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png" class="">https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png" class="">https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/jwyang/faster-rcnn.pytorch" class="">https://github.com/jwyang/faster-rcnn.pytorch</a>"}},"updates":{"snippets":[{"icon":"PERSON","message":"@lipka-clazzpl in #115: Not quite @andehr. \r\nLook at pickle files (*.pkl extension) saved here and there to speedup things even more.\r\n@salma26 treat it literally, it's directory located under faster-rcnn.pytorch/data/cache plus annotations pickle file *txt_annots.pkl in the data root if Pascal VOC was used for training."}],"action":{"name":"View Issue","url":"https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400" class="">https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400</a>"}}}
[
{
"@context": "http://schema.org" class="">http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400" class="">https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400",
"url": "https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400" class="">https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com" class="">https://github.com"
}
},
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions" class="">http://schema.org/extensions",
"hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [jwyang/faster-rcnn.pytorch] RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generated/../THCReduceAll.cuh:339 (#115)",
"sections": [
{
"text": "",
"activityTitle": "Przemek Lipka",
"activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png" class="">https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@lipka-clazzpl",
"facts": [

]
}
],
"potentialAction": [
{
"name": "Add a comment",
"@type": "ActionCard",
"inputs": [
{
"isMultiLine": true,
"@type": "TextInput",
"id": "IssueComment",
"isRequired": false
}
],
"actions": [
{
"name": "Comment",
"@type": "HttpPOST",
"target": "https://api.github.com" class="">https://api.github.com",
"body": "{\n\"commandName\": \"IssueComment\",\n\"repositoryFullName\": \"jwyang/faster-rcnn.pytorch\",\n\"issueId\": 115,\n\"IssueComment\": \"{{IssueComment.value}}\"\n}"
}
]
},
{
"name": "Close issue",
"@type": "HttpPOST",
"target": "https://api.github.com" class="">https://api.github.com",
"body": "{\n\"commandName\": \"IssueClose\",\n\"repositoryFullName\": \"jwyang/faster-rcnn.pytorch\",\n\"issueId\": 115\n}"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400" class="">https://github.com/jwyang/faster-rcnn.pytorch/issues/115#issuecomment-406578400"
}
],
"@type": "OpenUri",
"name": "View on GitHub"
},
{
"name": "Unsubscribe",
"@type": "HttpPOST",
"target": "https://api.github.com" class="">https://api.github.com",
"body": "{\n\"commandName\": \"MuteNotification\",\n\"threadId\": 320142771\n}"
}
],
"themeColor": "26292E"
}
]

salma26 on 7 Dec 2018

@salma26 what do you mean by (0, num_class-1)

eng100200 on 10 Jan 2019

you are supposed to have N output classes. Every class is represented by a number.
the dictionary whose keys are these numbers that should vary from 0 to N-1.

Le jeudi 10 janvier 2019 à 13:12:38 UTC+1, eng100200 <[email protected]> a écrit :

@salma26 what do you mean by (0, num_class-1)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

salma26 on 10 Jan 2019

@salma26 do you initially using n+1 classes,,,,because 0 for background,,,and rest of class labels (ids) for other classes... is it?

eng100200 on 10 Jan 2019

@salma26 my question is if you are using 0 for background then for 2 classes what are the labels? i mean now we have 2 classes and one background...

eng100200 on 10 Jan 2019

what do you mean by "0 for background"? if you have 2 classes, then they should be enumerated 0 and 1.

Le jeudi 10 janvier 2019 à 14:58:33 UTC+1, eng100200 <[email protected]> a écrit :

@salma26 my question is if you are using 0 for background then for 2 classes what are the labels? i mean now we have 2 classes and one background...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

salma26 on 10 Jan 2019

@salma26 basically there re number of classes, lets say three [car, person, bicycle],,,there labels will be 0,1,2 respectively. but you have define a default 0 class for background....because if none of these labels found in image then it has background.. so my basic question is whether to assigning 0 for class car (here ) will get my labels wrong, since, i have assigned two classes the same label. did you get what i mean? i am not sure that this way is right or wrong.

eng100200 on 11 Jan 2019

@artuncF did you resolve the issue and its cause?

eng100200 on 11 Jan 2019

Yes I got it. in this case, you need to treat "background" as a class. I mean if you have these classes : ( car, tree, building, person), if your program didn't recognize an object, it will assign the extra class Background for it. So Background needs to be a class.Your classes are: car, tree, building, person, background. They have to be represented by numbers 0, 1, 2, 3, 4. You cant assign the same label for two different classes.

Envoyé depuis Yahoo Mail pour Android

Le ven., janv. 11, 2019 à 4:59, eng100200notifications@github.com a écrit :
@salma26 basically there re number of classes, lets say three [car, person, bicycle],,,there labels will be 0,1,2 respectively. but you have define a default 0 class for background....because if none of these labels found in image then it has background.. so my basic question is whether to assigning 0 for class car (here ) will get my labels wrong, since, i have assigned two classes the same label. did you get what i mean? i am not sure that this way is right or wrong.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

salma26 on 11 Jan 2019

For those who use the one-hot vector, if there's one label missing, for example, you have [0,1,2,3,4,5,6,7,8,9] as the class label, and the class 2 and 4 are always missing (no class 2 and 4), you should still use num_class=10.

It's a quite low-level issue, but it really bugs you.

sdsy888 on 29 Oct 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

strange runtime error: dimension specified as 0 but tensor has no dimensions

twangnh · 5Comments

resume the training for pytorch 1.0

Codermay · 5Comments

why demo.py and test_net.py make different results

Gr1tee · 4Comments

error: ‘for’ loop initial declarations are only allowed in C99 mode

chj1933 · 5Comments

CPU version of ROI Crop

gullalc · 4Comments