Ssd.pytorch: RuntimeError: The shape of the mask [32, 8732] at index 0 does not match the shape of the indexed tensor [279424, 1] at index 0
rps@rps:~/桌面/ssd.pytorch$ python3 train.py
/home/rps/桌面/ssd.pytorch/ssd.py:34: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
self.priors = Variable(self.priorbox.forward(), volatile=True)
/home/rps/桌面/ssd.pytorch/layers/modules/l2norm.py:17: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
init.constant(self.weight,self.gamma)
Loading base network...
Initializing weights...
train.py:214: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(param)
Loading the dataset...
Training SSD on: VOC0712
Using the specified args:
Namespace(basenet='vgg16_reducedfc.pth', batch_size=32, cuda=True, dataset='VOC', dataset_root='/home/rps/data/VOCdevkit/', gamma=0.1, lr=0.001, momentum=0.9, num_workers=4, resume=None, save_folder='weights/', start_iter=0, visdom=False, weight_decay=0.0005)
train.py:169: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead.
targets = [Variable(ann.cuda(), volatile=True) for ann in targets]
Traceback (most recent call last):
File "train.py", line 255, in
train()
File "train.py", line 178, in train
loss_l, loss_c = criterion(out, targets)
File "/home/rps/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(input, *kwargs)
File "/home/rps/桌面/ssd.pytorch/layers/modules/multibox_loss.py", line 97, in forward
loss_c[pos] = 0 # filter out pos boxes for now
RuntimeError: The shape of the mask [32, 8732] at index 0 does not match the shape of the indexed tensor [279424, 1] at index 0
anyone helps,please...
17764591637
All 49 comments
I have the same error.Using Pytorch0.4+python3.5.
isaactalx
on 4 Jun 2018
python3.5 and pytorch 0.3.0 no problem
bobo0810
on 7 Jun 2018
I have the same error，if I switch the lines 96,97
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0
in multibox_loss.py, this error disappear.
But come with another error :
"File "/home/.../ssd.pytorch/layers/modules/multibox_loss.py", line 115, in forward
loss_l /= N
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.cuda.LongTensor for argument #3 'other'"
The type of tensor is not match, how can I fix it ?
xscjun
on 7 Jun 2018
@xscjun change line:
N = num_pos.data.sum()
to:
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
this should work
slomrafgrav
on 8 Jun 2018
Anyone has solved this problem? help me tks.
gtwell
on 25 Jul 2018
I have the same error，if I switch the lines 96,97
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0
in multibox_loss.py, this error disappear.
But come with another error :
"File "/home/.../ssd.pytorch/layers/modules/multibox_loss.py", line 115, in forward
loss_l /= N
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.cuda.LongTensor for argument #3 'other'"
The type of tensor is not match, how can I fix it ?
The “pos” -> torch.Size([32, 8732])
The “loss_c ” ->torch.Size([279424, 1])
when I add one line as :
loss_c = loss_c.view(pos.size()[0], pos.size()[1]) #add line
loss_c[pos] = 0 # filter out pos boxes for now
loss_c = loss_c.view(num, -1)
Then it worked.
Lin-Zhipeng
on 27 Sep 2018
I have the same error，if I switch the lines 96,97
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0
in multibox_loss.py, this error disappear.
But come with another error :
"File "/home/.../ssd.pytorch/layers/modules/multibox_loss.py", line 115, in forward
loss_l /= N
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.cuda.LongTensor for argument #3 'other'"
The type of tensor is not match, how can I fix it ?
i have the same error, and how did you solve it finally?
zxt-triumph
on 1 Nov 2018
I have the same error，if I switch the lines 96,97
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0
in multibox_loss.py, this error disappear.
But come with another error :
"File "/home/.../ssd.pytorch/layers/modules/multibox_loss.py", line 115, in forward
loss_l /= N
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.cuda.LongTensor for argument #3 'other'"
The type of tensor is not match, how can I fix it ?
i have the same error, so how could you figure it out finally?
zxt-triumph
on 1 Nov 2018
What file should be updated?
matthewarthur
on 7 Nov 2018
I have the same error，if I switch the lines 96,97
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0
in multibox_loss.py, this error disappear.
But come with another error :
"File "/home/.../ssd.pytorch/layers/modules/multibox_loss.py", line 115, in forward
loss_l /= N
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.cuda.LongTensor for argument #3 'other'"
The type of tensor is not match, how can I fix it ?
change the data type of N to FloatTensor.
queryor
on 12 Nov 2018
What file should be updated?
You may try to update your file /home/.../ssd.pytorch/layers/modules/multibox_loss.py, and add one line as @LZP4GitHub said above.
usherbob
on 12 Nov 2018
@usherbob python3.6+pytorch0.4.1, I added "_loss_c = loss_c.view(pos.size()[0], pos.size()[1])_ #add line", but I have another issue. _RuntimeError: copy_if failed to synchronize: device-side assert triggered_
subicWang
on 13 Nov 2018
Finally, I succeeded.
step1: switch the two lines 97,98:
_loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now_
step2: change the line144 _N = num_pos.data.sum()_ to
_N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()_
subicWang
on 14 Nov 2018
Finally, I succeeded.
step1: switch the two lines 97,98:
_loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for now_
step2: change the line144 _N = num_pos.data.sum()_ to
_N = num_pos.data.sum().double() loss_l = loss_l.double() loss_c = loss_c.double()_
I changed like this, but there was a RuntimeError still:
RuntimeError: device-side assert triggered
How can I fix it ? Looking forward to your reply.Thank you!
CJJ-717
on 14 Dec 2018
by changing the order of line 97 and 98 it throws a new error for me
Traceback (most recent call last):
File "train.py", line 254, in <module>
train()
File "train.py", line 182, in train
loc_loss += loss_l.data[0]
IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number
any suggestions?
_PS: I tried as well converting the loss to double as mentioned above and still the same error!_
### solved
apparently 'loss_l.data[0]' should be replaced with 'loss_l.item()' instead
this replacement applies on every loss_x.data[0] in the file!
wisdomk
on 22 Mar 2019
Finally, I succeeded.
step1: switch the two lines 97,98:
_loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for now_
step2: change the line144 _N = num_pos.data.sum()_ to
_N = num_pos.data.sum().double() loss_l = loss_l.double() loss_c = loss_c.double()_
很棒，但是有个小bug，是line 114,不是line 144
leaf918
on 26 Mar 2019
If your Python torch version is '0.4.1' ,you can change follow
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
But if your python torch version is 1.0.1,that change is no useful.
TianSong1991
on 26 Mar 2019
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py!
step3: change the line188,189,193,196:
loss_l.data[0] >> loss_l.data
loss_c.data[0] >> loss_c.data
loss.data[0] >> loss.data
TianSong1991
on 26 Mar 2019
loss is increasing as shown below
timer: 2.2050 sec.
iter 0 || Loss: 153.4730 || timer: 1.8316 sec.
iter 10 || Loss: 48.9679 || timer: 1.8920 sec.
iter 20 || Loss: 191.8098 || timer: 2.0969 sec.
iter 30 || Loss: 110.8081 || timer: 1.8849 sec.
iter 40 || Loss: 106.9749 || timer: 1.9373 sec.
iter 50 || Loss: 134.3674 || timer: 2.0012 sec.
.
.
help me to solve the issue.
charan1561
on 26 Mar 2019
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
thanks,that is usefully for me,but ,step3 is:line 183,184,188,191, 5 item ,loss_x.data[0] >> loss_x.data or loss.data[0] >> loss.data
litianciucas
on 31 Mar 2019
would be loss_x.data[0] >> loss_x.item() better?
blueardour
on 5 Apr 2019
@TianSong1991 Thanks a lot.Pytorch 1.0+Python 3.5 success!
espectre
on 16 Apr 2019
PS: I tried as well converting the loss to double as mentioned above and still the same error!
much obligated！
zz10001
on 9 May 2019
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
but loss is nan
mk123qwe
on 20 May 2019
@TianSong1991 Thanks a lot.Pytorch 1.0+Python 3.5 success!
but loss is nan
mk123qwe
on 20 May 2019
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.databut loss is nan
I have the same problem. Why loss is nan?
xafarranxera
on 20 May 2019
If your Python torch version is '0.4.1' ,you can change follow
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
But if your python torch version is 1.0.1,that change is no useful.
Hi , why don`t the loss_l divide by N?
OberstWB
on 24 May 2019
Same problem here.
I used the @ LZP4GitHub solution and it is working fine, but i don't understand what is the difference between its solution and https://github.com/amdegroot/ssd.pytorch/pull/322 this one.
SalahAdDin
on 11 Jul 2019
I have the same error.Using Pytorch1.1+python3.6
loss_c[pos] = 0 # filter out pos boxes for now
IndexError: The shape of the mask [32, 8732] at index 0 does not match the shape of the indexed tensor [279424, 1] at index 0
mm1327
on 30 Jul 2019
Pytorch version:
>>> import torch
>>> print(torch.__version__)
1.1.0
Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
multibox_loss.py:
Switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
Change line114
N = num_pos.data.sum() -> N = num_pos.data.sum().double()
and change the following two lines to:
loss_l = loss_l.double()
loss_c = loss_c.double()
train.py
loss_l.data[0] >> loss_l.data
loss_c.data[0] >> loss_c.data
loss.data[0] >> loss.data
And here is my output:
timer: 11.9583 sec.
iter 0 || Loss: 11728.9388 || timer: 0.2955 sec.
iter 10 || Loss: nan || timer: 0.2843 sec.
iter 20 || Loss: nan || timer: 0.2890 sec.
iter 30 || Loss: nan || timer: 0.2934 sec.
iter 40 || Loss: nan || timer: 0.2865 sec.
iter 50 || Loss: nan || timer: 0.2855 sec.
iter 60 || Loss: nan || timer: 0.2889 sec.
iter 70 || Loss: nan || timer: 0.2857 sec.
iter 80 || Loss: nan || timer: 0.2843 sec.
iter 90 || Loss: nan || timer: 0.2835 sec.
iter 100 || Loss: nan || timer: 0.2846 sec.
iter 110 || Loss: nan || timer: 0.2946 sec.
iter 120 || Loss: nan || timer: 0.2860 sec.
iter 130 || Loss: nan || timer: 0.2846 sec.
iter 140 || Loss: nan || timer: 0.2962 sec.
iter 150 || Loss: nan || timer: 0.2989 sec.
iter 160 || Loss: nan || timer: 0.2857 sec.
ashleylid
on 29 Aug 2019
Pytorch version:
>>> import torch >>> print(torch.__version__) 1.1.0Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linuxmultibox_loss.py:
Switch the two lines 97,98: loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for nowChange line114 N = num_pos.data.sum() -> N = num_pos.data.sum().double()and change the following two lines to: loss_l = loss_l.double() loss_c = loss_c.double()train.py
loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.dataAnd here is my output:
timer: 11.9583 sec. iter 0 || Loss: 11728.9388 || timer: 0.2955 sec. iter 10 || Loss: nan || timer: 0.2843 sec. iter 20 || Loss: nan || timer: 0.2890 sec. iter 30 || Loss: nan || timer: 0.2934 sec. iter 40 || Loss: nan || timer: 0.2865 sec. iter 50 || Loss: nan || timer: 0.2855 sec. iter 60 || Loss: nan || timer: 0.2889 sec. iter 70 || Loss: nan || timer: 0.2857 sec. iter 80 || Loss: nan || timer: 0.2843 sec. iter 90 || Loss: nan || timer: 0.2835 sec. iter 100 || Loss: nan || timer: 0.2846 sec. iter 110 || Loss: nan || timer: 0.2946 sec. iter 120 || Loss: nan || timer: 0.2860 sec. iter 130 || Loss: nan || timer: 0.2846 sec. iter 140 || Loss: nan || timer: 0.2962 sec. iter 150 || Loss: nan || timer: 0.2989 sec. iter 160 || Loss: nan || timer: 0.2857 sec.
I've encountered the same one here, have you solve this problem?
HaoWu1993
on 22 Sep 2019
learning rate is too big
| |
郭腾伟
邮箱：[email protected]
|
签名由 网易邮箱大师 定制
On 09/22/2019 11:52, HaoWu1993 wrote:
Pytorch version:
import torch
print(torch.__version__)
1.1.0
Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
multibox_loss.py:
Switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
Change line114
N = num_pos.data.sum() -> N = num_pos.data.sum().double()
and change the following two lines to:
loss_l = loss_l.double()
loss_c = loss_c.double()
train.py
loss_l.data[0] >> loss_l.data
loss_c.data[0] >> loss_c.data
loss.data[0] >> loss.data
And here is my output:
timer: 11.9583 sec.
iter 0 || Loss: 11728.9388 || timer: 0.2955 sec.
iter 10 || Loss: nan || timer: 0.2843 sec.
iter 20 || Loss: nan || timer: 0.2890 sec.
iter 30 || Loss: nan || timer: 0.2934 sec.
iter 40 || Loss: nan || timer: 0.2865 sec.
iter 50 || Loss: nan || timer: 0.2855 sec.
iter 60 || Loss: nan || timer: 0.2889 sec.
iter 70 || Loss: nan || timer: 0.2857 sec.
iter 80 || Loss: nan || timer: 0.2843 sec.
iter 90 || Loss: nan || timer: 0.2835 sec.
iter 100 || Loss: nan || timer: 0.2846 sec.
iter 110 || Loss: nan || timer: 0.2946 sec.
iter 120 || Loss: nan || timer: 0.2860 sec.
iter 130 || Loss: nan || timer: 0.2846 sec.
iter 140 || Loss: nan || timer: 0.2962 sec.
iter 150 || Loss: nan || timer: 0.2989 sec.
iter 160 || Loss: nan || timer: 0.2857 sec.
I've encountered the same one here, have you solve this problem?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
gtwell
on 22 Sep 2019
Pytorch version:
>>> import torch >>> print(torch.__version__) 1.1.0Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linuxmultibox_loss.py:
Switch the two lines 97,98: loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for nowChange line114 N = num_pos.data.sum() -> N = num_pos.data.sum().double()and change the following two lines to: loss_l = loss_l.double() loss_c = loss_c.double()train.py
loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.dataAnd here is my output:
```
timer: 11.9583 sec.
iter 0 || Loss: 11728.9388 || timer: 0.2955 sec.
iter 10 || Loss: nan || timer: 0.2843 sec.
iter 20 || Loss: nan || timer: 0.2890 sec.
iter 30 || Loss: nan || timer: 0.2934 sec.
iter 40 || Loss: nan || timer: 0.2865 sec.
iter 50 || Loss: nan || timer: 0.2855 sec.
iter 60 || Loss: nan || timer: 0.2889 sec.
iter 70 || Loss: nan || timer: 0.2857 sec.
iter 80 || Loss: nan || timer: 0.2843 sec.
iter 90 || Loss: nan || timer: 0.2835 sec.
iter 100 || Loss: nan || timer: 0.2846 sec.
iter 110 || Loss: nan || timer: 0.2946 sec.
iter 120 || Loss: nan || timer: 0.2860 sec.
iter 130 || Loss: nan || timer: 0.2846 sec.
iter 140 || Loss: nan || timer: 0.2962 sec.
iter 150 || Loss: nan || timer: 0.2989 sec.
iter 160 || Loss: nan || timer: 0.2857 sec.I think the loss is much enormous, you should add two lines:
loss_l /= N
loss_c /= N
Billnut
on 5 Oct 2019
Pytorch version:
>>> import torch >>> print(torch.__version__) 1.1.0Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linuxmultibox_loss.py:
Switch the two lines 97,98: loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for nowChange line114 N = num_pos.data.sum() -> N = num_pos.data.sum().double()and change the following two lines to: loss_l = loss_l.double() loss_c = loss_c.double()train.py
loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.dataAnd here is my output:
timer: 11.9583 sec. iter 0 || Loss: 11728.9388 || timer: 0.2955 sec. iter 10 || Loss: nan || timer: 0.2843 sec. iter 20 || Loss: nan || timer: 0.2890 sec. iter 30 || Loss: nan || timer: 0.2934 sec. iter 40 || Loss: nan || timer: 0.2865 sec. iter 50 || Loss: nan || timer: 0.2855 sec. iter 60 || Loss: nan || timer: 0.2889 sec. iter 70 || Loss: nan || timer: 0.2857 sec. iter 80 || Loss: nan || timer: 0.2843 sec. iter 90 || Loss: nan || timer: 0.2835 sec. iter 100 || Loss: nan || timer: 0.2846 sec. iter 110 || Loss: nan || timer: 0.2946 sec. iter 120 || Loss: nan || timer: 0.2860 sec. iter 130 || Loss: nan || timer: 0.2846 sec. iter 140 || Loss: nan || timer: 0.2962 sec. iter 150 || Loss: nan || timer: 0.2989 sec. iter 160 || Loss: nan || timer: 0.2857 sec.I've encountered the same one here, have you solve this problem?
I don't change line 114, and then nan loss disappears.
mengxingkong
on 18 Oct 2019
Those values of loss: loc_loss, conf_loss are much huge out of the memory, you would utilize the codes: N = num_pos.data.sum().double() loss_l = loss_l.double()
loss_c = loss_c.double()
loss_l /= N
loss_c /= N
And at the train.py, you should using the follow two lines instead of your codes
loc_loss += loss_l.item()
conf_loss += loss_c.item()
with best wish, better luck, good fortune.
------------------ 原始邮件 ------------------
发件人: "琉璃梦"<[email protected]>;
发送时间: 2019年10月18日(星期五) 晚上10:01
收件人: "amdegroot/ssd.pytorch"
抄送: "YUXIAOHONG"<[email protected]>; "Comment"<[email protected]>;
主题: Re: [amdegroot/ssd.pytorch] RuntimeError: The shape of the mask [32, 8732] at index 0 does not match the shape of the indexed tensor [279424, 1] at index 0 (#173)
Pytorch version:
>>> import torch >>> print(torch.__version__) 1.1.0
Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux
multibox_loss.py:
Switch the two lines 97,98: loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for now Change line114 N = num_pos.data.sum() -> N = num_pos.data.sum().double() and change the following two lines to: loss_l = loss_l.double() loss_c = loss_c.double()
train.py
loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
And here is my output:
timer: 11.9583 sec. iter 0 || Loss: 11728.9388 || timer: 0.2955 sec. iter 10 || Loss: nan || timer: 0.2843 sec. iter 20 || Loss: nan || timer: 0.2890 sec. iter 30 || Loss: nan || timer: 0.2934 sec. iter 40 || Loss: nan || timer: 0.2865 sec. iter 50 || Loss: nan || timer: 0.2855 sec. iter 60 || Loss: nan || timer: 0.2889 sec. iter 70 || Loss: nan || timer: 0.2857 sec. iter 80 || Loss: nan || timer: 0.2843 sec. iter 90 || Loss: nan || timer: 0.2835 sec. iter 100 || Loss: nan || timer: 0.2846 sec. iter 110 || Loss: nan || timer: 0.2946 sec. iter 120 || Loss: nan || timer: 0.2860 sec. iter 130 || Loss: nan || timer: 0.2846 sec. iter 140 || Loss: nan || timer: 0.2962 sec. iter 150 || Loss: nan || timer: 0.2989 sec. iter 160 || Loss: nan || timer: 0.2857 sec.
I've encountered the same one here, have you solve this problem?
I don't change line 114, and then nan loss disappears.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Billnut
on 28 Oct 2019
这些值 损失：loc_loss，conf_loss远远超出内存，您可以利用以下代码：N = num_pos.data.sum（）。double（）＆nbsp; ＆nbsp; ＆nbsp; ＆nbsp; loss_l = loss_l.double（）＆nbsp; ＆nbsp; ＆nbsp; ＆nbsp; loss_c = loss_c.double（）＆nbsp; ＆nbsp; ＆nbsp; ＆nbsp; loss_l / = N＆nbsp; ＆nbsp; ＆nbsp; ＆nbsp; loss_c / = N并且在train.py上，您应该使用following＆nbsp; 用两行代码代替loc_loss + = loss_l.item（）conf_loss + = loss_c.item（）＆nbsp; 祝你好运，好运，好运。＆nbsp;
…
------------------＆nbsp;原始邮件＆nbsp; ------------------发件人：“琉璃梦” notifications@github.com ;; 发送时间：2019年10月18日（星期五）晚上10:01收件人：“ amdegroot / ssd.pytorch” ssd.pytorch@noreply.github.com ;; 抄送：“ YUXIAOHONG” 353826721@qq.com ;; “评论” comment@noreply.github.com ;; 主题：Re：[amdegroot / ssd.pytorch] RuntimeError：索引0处的蒙版[32，8732]的形状与索引0处的索引张量[279424，1]的形状不匹配（＃173）Pytorch版本：＆gt;＆gt;＆gt; 导入割炬＆gt;＆gt;＆gt; 在Linux上的print（torch .__ version__）1.1.0 Python版本：Python 3.6.7（默认，2018年10月22日，11:32:17）[GCC 8.2.0] multibox_loss.py：切换两行97,98：loss_c = loss_c.view（num，-1）loss_c [pos] = 0＃现在过滤掉pos盒更改第114行N = num_pos.data.sum（）-> N = num_pos.data.sum（）。double（），并将以下两行更改为：loss_l = loss_l.double（）loss_c = loss_c.double（）train.py loss_l.data [0]＆gt;＆gt; loss_l.data loss_c.data [0]＆gt;＆gt; loss_c.data loss.data [0]＆gt;＆gt; loss.data这是我的输出：计时器：11.9583秒。迭代0 || 损失：11728.9388 || 计时器：0.2955秒。重复10 || 损失：南|| 计时器：0.2843秒。iter 20 || 损失：南|| 计时器：0.2890秒。iter 30 || 损失：南|| 计时器：0.2934秒 iter 40 || 损失：南|| 计时器：0.2865秒。重复50 || 损失：南|| 计时器：0.2855秒。iter 60 || 损失：南|| 计时器：0.2889秒 iter 70 || 损失：南|| 计时器：0.2857秒。iter 80 || 损失：南|| 计时器：0.2843秒。iter 90 || 损失：南|| 计时器：0.2835秒。重复100 || 损失：南|| 计时器：0.2846秒。iter 110 || 损失：南|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。重复50 || 损失：南|| 计时器：0.2855秒。iter 60 || 损失：南|| 计时器：0.2889秒 iter 70 || 损失：南|| 计时器：0.2857秒。iter 80 || 损失：南|| 计时器：0.2843秒。iter 90 || 损失：南|| 计时器：0.2835秒。重复100 || 损失：南|| 计时器：0.2846秒。iter 110 || 损失：南|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。重复50 || 损失：南|| 计时器：0.2855秒。iter 60 || 损失：南|| 计时器：0.2889秒 iter 70 || 损失：南|| 计时器：0.2857秒。iter 80 || 损失：南|| 计时器：0.2843秒。iter 90 || 损失：南|| 计时器：0.2835秒。重复100 || 损失：南|| 计时器：0.2846秒。iter 110 || 损失：南|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。iter 70 || 损失：南|| 计时器：0.2857秒。iter 80 || 损失：南|| 计时器：0.2843秒。iter 90 || 损失：南|| 计时器：0.2835秒。重复100 || 损失：南|| 计时器：0.2846秒。iter 110 || 损失：南|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。iter 70 || 损失：南|| 计时器：0.2857秒。iter 80 || 损失：南|| 计时器：0.2843秒。iter 90 || 损失：南|| 计时器：0.2835秒。重复100 || 损失：南|| 计时器：0.2846秒。iter 110 || 损失：南|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。楠|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。楠|| 计时器：0.2946秒。iter 120 || 损失：南|| 计时器：0.2860秒。iter 130 || 损失：南|| 计时器：0.2846秒。iter 140 || 损失：南|| 计时器：0.2962秒 重复150 || 损失：南|| 计时器：0.2989秒。iter 160 || 损失：南|| 计时器：0.2857秒。我在这里遇到过同样的问题，您解决了这个问题吗？我不更改114行，然后nan损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。然后南损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。然后南损失消失了。—您收到此评论是因为您发表了评论。直接回复此电子邮件，在GitHub上查看或取消订阅。
good! It work very good! Tank you !
haibochina
on 30 Oct 2019
@haibochina What?
SalahAdDin
on 30 Oct 2019
@haibochina What?
It means that the loss:loc_loss，conf_loss are out of range of your ram. So you can change the source code as following : N = num_pos.data.sum（）, loss_l / = N, loss_c / = N, loc_loss + = loss_l.item（）conf_loss + = loss_c.item（）
haibochina
on 4 Nov 2019
I think PR are welcommed.
SalahAdDin
on 5 Nov 2019
thank you @haibochina ,about the issue of lose=nan, your method is very good!
up2m
on 24 Nov 2019
I also had a nan loss issue after fixing multibox_loss.py
In my case it was because I was trying to use custom annotations and loading them as [x_center, y_center, width, height]
If anyone else is trying to do the same thing, the correct format is [x1, y1, x2, y2]
Training works now
J0hannB
on 13 Dec 2019
Pytorch version:
>>> import torch >>> print(torch.__version__) 1.1.0Python version:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linuxmultibox_loss.py:
Switch the two lines 97,98: loss_c = loss_c.view(num, -1) loss_c[pos] = 0 # filter out pos boxes for nowChange line114 N = num_pos.data.sum() -> N = num_pos.data.sum().double()and change the following two lines to: loss_l = loss_l.double() loss_c = loss_c.double()train.py
loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.dataAnd here is my output:
timer: 11.9583 sec. iter 0 || Loss: 11728.9388 || timer: 0.2955 sec. iter 10 || Loss: nan || timer: 0.2843 sec. iter 20 || Loss: nan || timer: 0.2890 sec. iter 30 || Loss: nan || timer: 0.2934 sec. iter 40 || Loss: nan || timer: 0.2865 sec. iter 50 || Loss: nan || timer: 0.2855 sec. iter 60 || Loss: nan || timer: 0.2889 sec. iter 70 || Loss: nan || timer: 0.2857 sec. iter 80 || Loss: nan || timer: 0.2843 sec. iter 90 || Loss: nan || timer: 0.2835 sec. iter 100 || Loss: nan || timer: 0.2846 sec. iter 110 || Loss: nan || timer: 0.2946 sec. iter 120 || Loss: nan || timer: 0.2860 sec. iter 130 || Loss: nan || timer: 0.2846 sec. iter 140 || Loss: nan || timer: 0.2962 sec. iter 150 || Loss: nan || timer: 0.2989 sec. iter 160 || Loss: nan || timer: 0.2857 sec.
Because of the loss too big, I change line 115 to
N = num_pos.data.sum().double() loss_l = loss_l.double() loss_c = loss_c.double() loss_l /= N loss_c /= Nsolve the issue
Json0926
on 25 Dec 2019
@TianSong1991, I follow your solution and got it running normally... but after a while (after iter 90) the loss exploded to nan..., did you experience the same thing?
timer: 6.1760 sec.
iter 0 || Loss: 31.7677 || timer: 0.3297 sec.
iter 10 || Loss: 24.6710 || timer: 0.3164 sec.
iter 20 || Loss: 24.0278 || timer: 0.3214 sec.
iter 30 || Loss: 25.0901 || timer: 0.3184 sec.
iter 40 || Loss: 16.9485 || timer: 0.3358 sec.
iter 50 || Loss: 17.5748 || timer: 0.3850 sec.
iter 60 || Loss: 26.2674 || timer: 0.3207 sec.
iter 70 || Loss: 20.7441 || timer: 0.3213 sec.
iter 80 || Loss: 16.5515 || timer: 0.3206 sec.
iter 90 || Loss: 25808.9131 || timer: 0.3171 sec.
iter 100 || Loss: nan || timer: 0.3274 sec.
iter 110 || Loss: nan || timer: 0.3548 sec.
iter 120 || Loss: nan || timer: 0.3141 sec.
iter 130 || Loss: nan || timer: 0.3231 sec.
iter 140 || Loss: nan || timer: 0.3254 sec.
iter 150 || Loss: nan || timer: 0.3174 sec.
iter 160 || Loss: nan || timer: 0.3144 sec.
iter 170 || Loss: nan || timer: 0.3679 sec.
iter 180 || Loss: nan || timer: 0.3631 sec.
iter 190 || Loss: nan || timer: 0.3516 sec.
iter 200 || Loss: nan || timer: 0.3692 sec.
iter 210 || Loss: nan || timer: 0.3523 sec.
iter 220 || Loss: nan || timer: 0.3204 sec.
iter 230 || Loss: nan || timer: 0.3151 sec.
iter 240 || Loss: nan || timer: 0.3210 sec.
iter 250 || Loss: nan || timer: 0.3241 sec.
iter 260 || Loss: nan || timer: 0.3217 sec.
iter 270 || Loss: nan || timer: 0.3156 sec.
iter 280 || Loss: nan || timer: 0.3125 sec.
iter 290 || Loss: nan || timer: 0.3196 sec.
iter 300 || Loss: nan || timer: 0.3172 sec.
ynjiun
on 26 May 2020
with @TianSong1991 solution except the step3 changed to following:
setp 3 change the train.py! step3: change the line183,184,188,191:
loss_l.data[0] >> loss_l.item()
loss_c.data[0] >> loss_c.item()
loss.data[0] >> loss.item()
now loss is converging...
timer: 6.1581 sec.
iter 0 || Loss: 32.3338 || timer: 0.3283 sec.
iter 10 || Loss: 24.8091 || timer: 0.3328 sec.
iter 20 || Loss: 24.4980 || timer: 0.3275 sec.
iter 30 || Loss: 21.3105 || timer: 0.3167 sec.
iter 40 || Loss: 14.5682 || timer: 0.3223 sec.
iter 50 || Loss: 13.0729 || timer: 0.3221 sec.
iter 60 || Loss: 12.3032 || timer: 0.3383 sec.
iter 70 || Loss: 10.5260 || timer: 0.3246 sec.
iter 80 || Loss: 11.2028 || timer: 0.3380 sec.
iter 90 || Loss: 10.1715 || timer: 0.3244 sec.
iter 100 || Loss: 10.1702 || timer: 0.3342 sec.
iter 110 || Loss: 9.8668 || timer: 0.3384 sec.
iter 120 || Loss: 9.5938 || timer: 0.3676 sec.
iter 130 || Loss: 10.0942 || timer: 0.3210 sec.
iter 140 || Loss: 9.7601 || timer: 0.3246 sec.
iter 150 || Loss: 10.1564 || timer: 0.3202 sec.
iter 160 || Loss: 9.8361 || timer: 0.3215 sec.
iter 170 || Loss: 9.3565 || timer: 0.3290 sec.
iter 180 || Loss: 9.2069 || timer: 0.3481 sec.
iter 190 || Loss: 9.0822 || timer: 0.3374 sec.
iter 200 || Loss: 9.3702 || timer: 0.3333 sec.
iter 210 || Loss: 9.6193 || timer: 0.3437 sec.
iter 220 || Loss: 9.1466 || timer: 0.3590 sec.
iter 230 || Loss: 8.8923 || timer: 0.3211 sec.
iter 240 || Loss: 9.2617 || timer: 0.3526 sec.
iter 250 || Loss: 9.1713 || timer: 0.3263 sec.
iter 260 || Loss: 9.4524 || timer: 0.3262 sec.
iter 270 || Loss: 9.4929 || timer: 0.3581 sec.
iter 280 || Loss: 8.7274 || timer: 0.3345 sec.
iter 290 || Loss: 9.6723 || timer: 0.3701 sec.
......
ynjiun
on 26 May 2020
with @TianSong1991 solution except the step3 changed to following:
setp 3 change the train.py! step3: change the line183,184,188,191:
loss_l.data[0] >> loss_l.item()
loss_c.data[0] >> loss_c.item()
loss.data[0] >> loss.item()now loss is converging...
timer: 6.1581 sec.
iter 0 || Loss: 32.3338 || timer: 0.3283 sec.
iter 10 || Loss: 24.8091 || timer: 0.3328 sec.
iter 20 || Loss: 24.4980 || timer: 0.3275 sec.
iter 30 || Loss: 21.3105 || timer: 0.3167 sec.
iter 40 || Loss: 14.5682 || timer: 0.3223 sec.
iter 50 || Loss: 13.0729 || timer: 0.3221 sec.
iter 60 || Loss: 12.3032 || timer: 0.3383 sec.
iter 70 || Loss: 10.5260 || timer: 0.3246 sec.
iter 80 || Loss: 11.2028 || timer: 0.3380 sec.
iter 90 || Loss: 10.1715 || timer: 0.3244 sec.
iter 100 || Loss: 10.1702 || timer: 0.3342 sec.
iter 110 || Loss: 9.8668 || timer: 0.3384 sec.
iter 120 || Loss: 9.5938 || timer: 0.3676 sec.
iter 130 || Loss: 10.0942 || timer: 0.3210 sec.
iter 140 || Loss: 9.7601 || timer: 0.3246 sec.
iter 150 || Loss: 10.1564 || timer: 0.3202 sec.
iter 160 || Loss: 9.8361 || timer: 0.3215 sec.
iter 170 || Loss: 9.3565 || timer: 0.3290 sec.
iter 180 || Loss: 9.2069 || timer: 0.3481 sec.
iter 190 || Loss: 9.0822 || timer: 0.3374 sec.
iter 200 || Loss: 9.3702 || timer: 0.3333 sec.
iter 210 || Loss: 9.6193 || timer: 0.3437 sec.
iter 220 || Loss: 9.1466 || timer: 0.3590 sec.
iter 230 || Loss: 8.8923 || timer: 0.3211 sec.
iter 240 || Loss: 9.2617 || timer: 0.3526 sec.
iter 250 || Loss: 9.1713 || timer: 0.3263 sec.
iter 260 || Loss: 9.4524 || timer: 0.3262 sec.
iter 270 || Loss: 9.4929 || timer: 0.3581 sec.
iter 280 || Loss: 8.7274 || timer: 0.3345 sec.
iter 290 || Loss: 9.6723 || timer: 0.3701 sec.
......
what's your torch version and python version?
yingjun-zhang
on 28 Jun 2020
When encountered
timer: 10.2599 sec.
iter 0 || Loss: 30.8010 || timer: 0.4961 sec.
iter 10 || Loss: 19.9977 || timer: 1.1120 sec.
iter 20 || Loss: 19.2539 || timer: 1.8164 sec.
iter 30 || Loss: 16.7701 || timer: 0.9436 sec.
iter 40 || Loss: 18.0430 || timer: 0.7898 sec.
iter 50 || Loss: 25.5106 || timer: 1.0395 sec.
iter 60 || Loss: 23.7020 || timer: 0.8617 sec.
iter 70 || Loss: nan || timer: 1.0497 sec.
iter 80 || Loss: nan || timer: 1.2802 sec.
maybe you can change lr=1e-4,when i change ,then
timer: 10.1423 sec.
iter 0 || Loss: 29.5713 || timer: 0.4259 sec.
iter 10 || Loss: 22.9357 || timer: 1.2987 sec.
iter 20 || Loss: 20.2871 || timer: 1.1511 sec.
iter 30 || Loss: 20.0152 || timer: 0.9707 sec.
iter 40 || Loss: 19.3170 || timer: 0.9684 sec.
iter 50 || Loss: 19.0578 || timer: 1.0160 sec.
iter 60 || Loss: 19.2979 || timer: 1.2673 sec.
iter 70 || Loss: 18.9950 || timer: 1.1985 sec.
iter 80 || Loss: 16.6445 || timer: 1.2570 sec.
He-zl8
on 1 Aug 2020
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
thanks，this answer solves my problem.
cotyyang
on 22 Sep 2020
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
I'm also this answer solved my probelm.
more correctly,
loss_l = loss_l.double()/N
loss_c = loss_c.doubel()/N
:)
hjlee9182
on 23 Oct 2020
Thank you. I've solved the problem. Thank you again.
------------------ 原始邮件 ------------------
发件人: "HyunJun Lee"<[email protected]>;
发送时间: 2020年10月23日(星期五) 上午10:14
收件人: "amdegroot/ssd.pytorch"
抄送: "贺智龙"<[email protected]>; "Comment"<[email protected]>;
主题: Re: [amdegroot/ssd.pytorch] RuntimeError: The shape of the mask [32, 8732] at index 0 does not match the shape of the indexed tensor [279424, 1] at index 0 (#173)
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py! step3: change the line188,189,193,196: loss_l.data[0] >> loss_l.data loss_c.data[0] >> loss_c.data loss.data[0] >> loss.data
I'm also this answer solved my probelm.
more correctly,
loss_l = loss_l.double()/N
loss_c = loss_c.doubel()/N
:)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
He-zl8
on 23 Oct 2020
Related issues
squirrel16
·
5Comments
isalirezag
·
4Comments
mikejmills
·
6Comments
lmingyin
·
5Comments
luhc15
·
6Comments
Most helpful comment
I solve the problem if your python torch version is 1.0.1. The solution as follow 1-3 steps:
step1 and step2 change the multibox_loss.py!
step1: switch the two lines 97,98:
loss_c = loss_c.view(num, -1)
loss_c[pos] = 0 # filter out pos boxes for now
step2: change the line114 N = num_pos.data.sum() to
N = num_pos.data.sum().double()
loss_l = loss_l.double()
loss_c = loss_c.double()
setp 3 change the train.py!
step3: change the line188,189,193,196:
loss_l.data[0] >> loss_l.data
loss_c.data[0] >> loss_c.data
loss.data[0] >> loss.data