Centernet: torch0.4.0&torch0.4.1

Created on 24 Oct 2019  ·  7Comments  ·  Source: xingyizhou/CenterNet

I use the torch0.4.0 to run the cmd "python main.py ctder" with arch"hourglass",but the error happended.
ctdet/defaultTraceback (most recent call last):
File "main.py", line 112, in
main(opt)
File "main.py", line 77, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 124, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 72, in run_epoch
batch[k] = batch[k].to(device=opt.device, non_blocking=True)
TypeError: to() got an unexpected keyword argument 'non_blocking'

But when I use pytorch1.3.0, the error disappearred.
Is there any friends know why? I feel very confused.

Most helpful comment

And when I use torch0.4.1, the problem is following:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "main.py", line 114, in
main(opt)
File "main.py", line 79, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 125, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 75, in run_epoch
output, loss, loss_stats = model_with_loss(batch)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 20, in forward
outputs = self.model(batch['input'])
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 255, in forward
inter = self.pre(image)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 27, in forward
conv = self.conv(x)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663

All 7 comments

And when I use torch0.4.1, the problem is following:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "main.py", line 114, in
main(opt)
File "main.py", line 79, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 125, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 75, in run_epoch
output, loss, loss_stats = model_with_loss(batch)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 20, in forward
outputs = self.model(batch['input'])
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 255, in forward
inter = self.pre(image)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 27, in forward
conv = self.conv(x)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663

same question

torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument

torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument

Do you run it in the 2080 Ti?

same question

Do you run it in the 2080 Ti?

torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument

Do you run it in the 2080 Ti?

no ,but i just solved it.
try this.

https://github.com/kimyoon-young/centerNet-deep-sort/issues/16#issuecomment-560046429

我用中文回答吧。
我是从王利民老师这个https://github.com/MCG-NJU/MOC-Detector工作过来的
看了网上关于这个问题原因和一些解决方案,我这里总结一下:
因为有很多基于这个DCN的工作,那么类似centernet出现的这个问题就是一致由DCN的cuda代码版本引发的了。
至于cuda9.0很多人也出现这个问题,大概率是用的2080ti的卡吧,不用折腾cuda用9.2也没用,你升级到10.0就行。
所以核心就是你去DCNv2这个github的官网,把你代码里用到的DCNv2删掉,用作者对应的pytorch版本编译就OK了。
所以不用纠结其他人解决方案里的还要用下载whl的torch1.0.0手动安装了,你用1.4或者更高都行,conda或者pip安装也没区别,
至于你代码出现的其他问题,那就不是DCN导致的了。所以还是要给GitHub的DCNv2作者一个赞。

Was this page helpful?
0 / 5 - 0 ratings

Related issues

David-19940718 picture David-19940718  ·  5Comments

huang229 picture huang229  ·  4Comments

merckxiaan picture merckxiaan  ·  7Comments

chwong1 picture chwong1  ·  4Comments

leiyaohui picture leiyaohui  ·  3Comments