Ssd.pytorch: cuda Runtime Error (77): an illegal memory access was encountered

[chauhan-utk]

iter 510 || Loss: 6.8001 || THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1502009910772/work/torch/lib/THC/generated/../THCReduceAll.cuh line=334 error=77 : an illegal memory access was encountered Traceback (most recent call last): File "train.py", line 231, in <module> train() File "train.py", line 183, in train loss_l, loss_c = criterion(out, targets) File "/users/gpu/utkrsh/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__ result = self.forward(*input, **kwargs) File "/users/gpu/utkrsh/code/ssd.pytorch/layers/modules/multibox_loss.py", line 137, in forward conf_p = conf_data[(pos_idx+neg_idx).gt(0)].view(-1, self.num_classes) File "/users/gpu/utkrsh/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/variable.py", line 72, in __getitem__ return MaskedSelect.apply(self, key) File "/users/gpu/utkrsh/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py", line 468, in forward return tensor.masked_select(mask) RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /opt/conda/conda-bld/pytorch_1502009910772/work/torch/lib/THC/generated/../THCReduceAll.cuh:334
I am trying to train the network with a slight modification in localization loss in multibox_loss.py. I keep on getting this error message for the same line of code. Also, when starting to train, there is a warning
/users/gpu/utkrsh/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py:450: UserWarning: mask is not broadcastable to self, but they have the same number of elements. Falling back to deprecated pointwise behavior. return tensor.masked_fill_(mask, value)

I am training with batch_size=32 in train.py and everything else is at the default value. I have tried to modify the code but there is no impact on the warning and I keep getting this error.
Also, if I use a larger batch_size in train.py like 40, I get this illegal memory access error much earlier than with size 32.
Any suggestions for what might be wrong?

[JoachimHugonot]

Try using

os.environ['CUDA_LAUNCH_BLOCKING'] = 1

as :

Sometimes the error message can be caught at a different place compared to where it is occuring because of the asynchronous nature of CUDA (pasted from there)

[chauhan-utk]

@JoachimHugonot Okay! I will use this in future runs. Though this error is quite arbitrary.

[JoachimHugonot]

@chauhan-utk x
Could you try to scale the loss in : https://github.com/amdegroot/ssd.pytorch/blob/ce4c994db0ee11f82aabb4fdb3499dc970156db5/train.py#L184
like this :
loss = (loss_l + loss_c)/1000.0
and report back the results ?
For me it solved the crashes, but I still need to run more tests to be sure.

[amdegroot]

Hi! What is your slight modification? I have trained the model recently using batch size 32 through 120k iterations without any issues, so I cannot comment without knowledge of the change you made.

[chauhan-utk]

I had changed the line 88 in multibox_loss.py to include a custom loss function. So far my observation have been that loc_p contains the best single prediction for the corresponding ground truth loc_t and so we can use something other than smooth L1 loss. The resultant saved in loss_t is a single dimension tensor just as it would by using the existing loss function. Rest everything was same.

Since this is not a particular issue with the existing code and the above methods have been somewhat effective, I am closing this issue.

[DontGiveUpEasily]

I had similar issues. Any thoughts?

[jxlijunhao]

If used CUDA_VISIBLE_DEVICES to assign just a single GPU, comment
net = torch.nn.DataParallel(ssd_net)
It works for me.