Ssd.pytorch: RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

[huster-wgm]

[heinzermch]

Same problem here. Using Cuda 9.1 and a custom dataset. However after running the program many times, it seems not to be easily reproducible or reoccurring often.

[ternaus]

Same.

Ububntu 16.04
Pytorch 0.3.1
cuda 9.0

[rronan]

Same here, using batch_size=8 and all other params to default.

Ubuntu 16.04
Pytorch 0.3.1
cuda 9.0.176

[JavisPeng]

Same here too

timer: 5.0469 sec.
iter 0 || Loss: nan || Traceback (most recent call last):
File "train_character.py", line 242, in
train()
File "train_character.py", line 165, in train
loss_l, loss_c = criterion(out, targets)
File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(input, *kwargs)
File "/share/share/program/tianchi/icpr/code/ssd.pytorch-master/layers/modules/multibox_loss.py", line 99, in forward
_, loss_idx = loss_c.sort(1, descending=True)
RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

[wangzhaolei]

'num_classes' must +1 for background !!! I met the same problem too when training on my own dataset. I soved the problem with 'num_classes' =num of object classes + background class, this is in the data/config.py . So don't forget the background class!!
my envs : CentOS Linux 7.4, cuda9.0 , python 3.5 , pytorch 0.3.0

[Yingmin]

Adding num_classes by 1 fixed the problem for batch_size=4
but if I increase the batch_size to 12 the problem appeared again
I decrease the batch_size to 8 seems the whole things are working

Feel it's still a bug but not reproducible every time

[heinzermch]

It indeed looks like a bug, although one which is hard to reproduce. In my case it tends to happen when the batch size is over 100 (in total) and I'm using multiple GPUs.

The main source seems to be the classification loss blowing up:

Epoch: 0 2018-06-26 13:44:54 loc_loss: 11.278 | cls_loss: 13.128 | Epoch 0 train_loss: 24.406 | avg_loss: 24.406 [1/877] loc_loss: 8.562 | cls_loss: 10.213 | Epoch 0 train_loss: 18.775 | avg_loss: 21.590 [2/877] loc_loss: 9.041 | cls_loss: 9.963 | Epoch 0 train_loss: 19.004 | avg_loss: 20.728 [3/877] loc_loss: 7.867 | cls_loss: 10.340 | Epoch 0 train_loss: 18.207 | avg_loss: 20.098 [4/877] loc_loss: 6.692 | cls_loss: 9.695 | Epoch 0 train_loss: 16.387 | avg_loss: 19.356 [5/877] loc_loss: 6.141 | cls_loss: 12.809 | Epoch 0 train_loss: 18.950 | avg_loss: 19.288 [6/877] loc_loss: 5.894 | cls_loss: 13.218 | Epoch 0 train_loss: 19.112 | avg_loss: 19.263 [7/877] loc_loss: 6.039 | cls_loss: 12.399 | Epoch 0 train_loss: 18.438 | avg_loss: 19.160 [8/877] loc_loss: 6.676 | cls_loss: 12.464 | Epoch 0 train_loss: 19.140 | avg_loss: 19.158 [9/877] loc_loss: 5.516 | cls_loss: 10.459 | Epoch 0 train_loss: 15.975 | avg_loss: 18.840 [10/877] loc_loss: 5.279 | cls_loss: 16.400 | Epoch 0 train_loss: 21.679 | avg_loss: 19.098 [11/877] loc_loss: 9.847 | cls_loss: 16.355 | Epoch 0 train_loss: 26.201 | avg_loss: 19.690 [12/877] loc_loss: 7.636 | cls_loss: 10.949 | Epoch 0 train_loss: 18.585 | avg_loss: 19.605 [13/877] loc_loss: 7.970 | cls_loss: 23.531 | Epoch 0 train_loss: 31.501 | avg_loss: 20.454 [14/877] loc_loss: 6.999 | cls_loss: 21.177 | Epoch 0 train_loss: 28.177 | avg_loss: 20.969 [15/877] loc_loss: 7.463 | cls_loss: 29.432 | Epoch 0 train_loss: 36.895 | avg_loss: 21.965 [16/877] loc_loss: 2610.023 | cls_loss: 78130.476 | Epoch 0 train_loss: 80740.500 | avg_loss: 4770.114 [17/877] loc_loss: 133051060712667671953408000.000 | cls_loss: 1934428018548307309214302208.000 | Epoch 0 train_loss: 2067479199807607175016087552.000 | avg_loss: 114859955544867071005294592.000 [18/877] Traceback (most recent call last): File "train_day.py", line 122, in <module> train(epoch) File "train_day.py", line 76, in train loss = criterion(loc_preds, loc_targets, cls_preds, cls_targets) File "/u/home/heinzerm/anaconda2/envs/py35/lib/python3.5/site-packages/torch/nn/modules/module.py", line 491, in __call__ result = self.forward(*input, **kwargs) File "/u/home/heinzerm/PycharmProjects/TorchCV/torchcv/loss/ssd_loss.py", line 61, in forward neg = self._hard_negative_mining(cls_loss, pos) # [N,#anchors] File "/u/home/heinzerm/PycharmProjects/TorchCV/torchcv/loss/ssd_loss.py", line 25, in _hard_negative_mining _, idx = cls_loss.sort(1) # sort by negative losses RuntimeError: merge_sort: failed to synchronize: an illegal memory access was encountered

I will investigate further, but the randomness and the need for big batch sizes make this a difficult case

[dongzhuoyao]

same here

Ububntu 16.04
Pytorch 0.3.1
cuda 9.1

[isalirezag]

anyone solve this problem???
:(

[heinzermch]

Kind of, the easiest way to get rid of the problem for me was to decrease the learning rate (start with a factor of 10). Once I did that the problem disappeared and I could choose an arbitrary batch size. Before a bigger batch size meant a bigger likelihood of an exploding loss, which leads to the above error

[isalirezag]

@heinzermch yeah i solve it the same way.
though It is ridiculous that there is no logical/solution way behind it yet

[aliutkus]

same bug here

[isalirezag]

after changing the --lr i went through for me but ended up with

iter 119990 || Loss: 2.8278 || Fatal Python error: deallocating None

Thread 0x00007f358ae12700 (most recent call first):

Thread 0x00007f358d7fe700 (most recent call first):

Thread 0x00007f358dfff700 (most recent call first):

Thread 0x00007f358cffd700 (most recent call first):

Thread 0x00007f358c7fc700 (most recent call first):

Current thread 0x00007f361dab7700 (most recent call first):
Aborted (core dumped)

And i have no idea why

[aliutkus]

please note that in my case, it's not related to learning-rate. I'm just doing plain sorting

[asml]

Same here. Difficult to reproduce, but it seems to occur less often with smaller lr.

[shl1993]

for me, use contiguous() before view() works!

[JerryLauzzz]

If you use only one GPU, you can comment line 99-100

if args.cuda:
    net = torch.nn.DataParallel(ssd_net)
    cudnn.benchmark = True

in train.py. It works for me.

[twmht]

any update on this?

[tt12354]

batch_size=16,lr=1e-4 is work for me

[taomrzhang]

after changing the --lr i went through for me but ended up with

iter 119990 || Loss: 2.8278 || Fatal Python error: deallocating None

Thread 0x00007f358ae12700 (most recent call first):

Thread 0x00007f358d7fe700 (most recent call first):

Thread 0x00007f358dfff700 (most recent call first):

Thread 0x00007f358cffd700 (most recent call first):

Thread 0x00007f358c7fc700 (most recent call first):

Current thread 0x00007f361dab7700 (most recent call first):
Aborted (core dumped)

And i have no idea why

Hello, I met the same error, and have no idea. Have you solved it? Can you tell you me how can solve that error?

[chbrian]

Same bug here.
cuda 9.0 pytorch 0.4.1 ubuntu 16.04

[colllin]

@heinzermch's clue about exploding cross-entropy loss helped me fix it. I had multiple losses so I reduced the weight of the cross-entropy loss by a factor of 10 to stabilize learning. Reducing the learning rate should have a similar effect to reduce the weight update w.r.t. to CE loss. Possibly this is a symptom of too-high LR early in training, when CE loss on an untrained model can cause strong gradients.

[imyhxy]

'num_classes' must +1 for background !!! I met the same problem too when training on my own dataset. I soved the problem with 'num_classes' =num of object classes + background class, this is in the data/config.py . So don't forget the background class!!
my envs : CentOS Linux 7.4, cuda9.0 , python 3.5 , pytorch 0.3.0

This does not make sense, if we don't access beyond the index.