I use the torch0.4.0 to run the cmd "python main.py ctder" with arch"hourglass",but the error happended.
ctdet/defaultTraceback (most recent call last):
File "main.py", line 112, in
main(opt)
File "main.py", line 77, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 124, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 72, in run_epoch
batch[k] = batch[k].to(device=opt.device, non_blocking=True)
TypeError: to() got an unexpected keyword argument 'non_blocking'
But when I use pytorch1.3.0, the error disappearred.
Is there any friends know why? I feel very confused.
And when I use torch0.4.1, the problem is following:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "main.py", line 114, in
main(opt)
File "main.py", line 79, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 125, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 75, in run_epoch
output, loss, loss_stats = model_with_loss(batch)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 20, in forward
outputs = self.model(batch['input'])
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 255, in forward
inter = self.pre(image)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 27, in forward
conv = self.conv(x)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663
same question
torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Do you run it in the 2080 Ti?
same question
Do you run it in the 2080 Ti?
torch0.4.1 cuda9.0 same question when i ran the demo.py
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argumentDo you run it in the 2080 Ti?
no ,but i just solved it.
try this.
https://github.com/kimyoon-young/centerNet-deep-sort/issues/16#issuecomment-560046429
我用中文回答吧。
我是从王利民老师这个https://github.com/MCG-NJU/MOC-Detector工作过来的。
看了网上关于这个问题原因和一些解决方案,我这里总结一下:
因为有很多基于这个DCN的工作,那么类似centernet出现的这个问题就是一致由DCN的cuda代码版本引发的了。
至于cuda9.0很多人也出现这个问题,大概率是用的2080ti的卡吧,不用折腾cuda用9.2也没用,你升级到10.0就行。
所以核心就是你去DCNv2这个github的官网,把你代码里用到的DCNv2删掉,用作者对应的pytorch版本编译就OK了。
所以不用纠结其他人解决方案里的还要用下载whl的torch1.0.0手动安装了,你用1.4或者更高都行,conda或者pip安装也没区别,
至于你代码出现的其他问题,那就不是DCN导致的了。所以还是要给GitHub的DCNv2作者一个赞。
Most helpful comment
And when I use torch0.4.1, the problem is following:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "main.py", line 114, in
main(opt)
File "main.py", line 79, in main
log_dict_train, _ = trainer.train(epoch, train_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 125, in train
return self.run_epoch('train', epoch, data_loader)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 75, in run_epoch
output, loss, loss_stats = model_with_loss(batch)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/trains/base_trainer.py", line 20, in forward
outputs = self.model(batch['input'])
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 255, in forward
inter = self.pre(image)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/Network/CenterNet/src/lib/models/networks/large_hourglass.py", line 27, in forward
conv = self.conv(x)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/zwl/.conda/envs/py36torch04/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663