Centernet: cuda out of memory

Created on 3 Jul 2019  路  6Comments  路  Source: xingyizhou/CenterNet

ubuntu18 Cuda 9.2 torch==0.4.1 2080ti
when i run demo.py , i meet "cuda out of memory"
who can help me?@.@

(centernet1) li@li:~/PycharmProjects/CenterNet-master(1)/CenterNet-master/src$ python demo.py ctdet --demo /home/li/Documents/baozong.avi --load_model ../models/ctdet_coco_dla_2x.pth
Fix size testing.
training chunk_sizes: [32]
The output will be saved to /home/li/PycharmProjects/CenterNet-master(1)/CenterNet-master/src/lib/../../exp/ctdet/default
heads {'hm': 80, 'wh': 2, 'reg': 2}
Creating model...
loaded ../models/ctdet_coco_dla_2x.pth, epoch 230
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "demo.py", line 57, in
demo(opt)
File "demo.py", line 31, in demo
ret = detector.run(img)
File "/home/li/PycharmProjects/CenterNet-master(1)/CenterNet-master/src/lib/detectors/base_detector.py", line 116, in run
output, dets, forward_time = self.process(images, return_time=True)
File "/home/li/PycharmProjects/CenterNet-master(1)/CenterNet-master/src/lib/detectors/ctdet.py", line 26, in process
output = self.model(images)[-1]
File "/home/li/.conda/envs/centernet1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/li/PycharmProjects/CenterNet-master(1)/CenterNet-master/src/lib/models/networks/pose_dla_dcn.py", line 481, in forward
z[head] = self.__getattr__(head)(y[-1])
File "/home/li/.conda/envs/centernet1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/li/.conda/envs/centernet1/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/li/.conda/envs/centernet1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(input, *kwargs)
File "/home/li/.conda/envs/centernet1/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: out of memory

Most helpful comment

torch.backends.cudnn.deterministic = True

All 6 comments

I have solved it by setting "torch.backends.cudnn.enabled = False".

Thanks for the solution!

Could you tell me it is where?

Could you tell me it is where?

@201814476285951 you can add it in demo.py.

@violet17 thanks a lot.

torch.backends.cudnn.deterministic = True

Was this page helpful?
0 / 5 - 0 ratings

Related issues

mjanddy picture mjanddy  路  3Comments

chwong1 picture chwong1  路  4Comments

lih627 picture lih627  路  7Comments

Alexyitx picture Alexyitx  路  6Comments

wjx2 picture wjx2  路  7Comments