Yolact: DO NOT use the latest pytorch version(1.1.0)

[SeavanMao]

Please do not use the latest pytorch version 1.1.0 which may cause the CUDA errors.

the default pytorch install command is
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

use the below instead
pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1-cp36-cp36m-win_amd64.whl

one day wasted to debug this [cry]

[dbolya]

Oof, sorry you had to go through that. However, I just upgraded to pytorch 1.1.0 (I'd like to support the latest version of pytorch for the time being), and I didn't get any CUDA errors during eval or training. Can you share your error messages / how you encountered them?

[SeavanMao]

My gpu is 11G and it dies after the first training log appears.
says " Message= CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 11.00 GiB total capacity; 8.49 GiB already allocated; 36.32 MiB free; 195.06 MiB cached) (malloc at ..\c10\cuda\CUDACachingAllocator.cpp:267)"

I changed back the torch to 1.0.1 it works.

[dbolya]

Oh wow, I've just ran some tests and Pytorch 1.1.0 uses vastly more VRAM than 1.0.1. With the base config and batch size 5 on my 1080 Ti, I was getting 6794 MiB on 1.0.1, but 7853 MiB on 1.1.0. That's alarming.

I'll have to look into this.

[SeavanMao]

Got it. my batchsize is default 8. maybe the problem here. Thank you ~

[stuafu]

i met this problem ,it's weired that a lot of memory has not been used when it raise error:
" RuntimeError: CUDA out of memory. Tried to allocate 2.18 GiB (GPU 0; 11.00 GiB total capacity; 527.83 MiB already allocated; 8.23 GiB free; 12.17 MiB cached"

i change batchsize to from 8 to 1 ,and the error is still

[dbolya]

Is that on Pytorch 1.1.0 or 1.0.1?

[stuafu]

Above env is python3.7+ Pytorch 1.1.0 or python3.7+Pytorch 1.0.0 (now i make a new env python3.6+ Pytorch 1.0.1) to test .

comand is :--config=yolact_base_config --resume=weights/yolact_base_54_800000.pth --batch_size=5(or 1)

when i used coco dataset ,it rasied error"CUDA out of memory",when i used my owned data ,it train some bathes and then raise error"MemoryError" randomly.

[zhouyuangan]

@dbolya Hi, I met this problem both on v1.0.1 and v1.1.0 . I have a another classification task ran successfully using 4 gpus(GTX 1080 TI) which torch version is 1.1.0, so this time I changed into v1.0.1 as mentioned above, the problem still exist. However, I modified codes and DID NOT USE torch.nn.DataParallel, it ran on sigle gpu successfully, that's confusing...
CUDA:9.0, python: 3.5.4, sys: ubuntu 16.04

[dbolya]

That's really weird...

@stuafu Does it work if you remove the two nn.DataParallel on this line and the line under?
https://github.com/dbolya/yolact/blob/master/train.py#L170

[stuafu]

when i remove this two line

the error is same as above .on my own Dataset , 'MemoryError". On COCO , ""CUDA out of memory".
I was run on win10 , should i try to run on ubuntu?

[dbolya]

Pytorch support isn't the greatest on Windows, and Windows is known to hog GPU memory for no reason. I'd give Ubuntu a shot if it's available to you.

[songtaoshi]

@dbolya hello could you show your python cuda cudnn version. I met some problems. RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

[dbolya]

@songtaoshi I've run this over a couple of different configs, but I personally use Python 3.6, Pytorch 1.0.1, and CUDA 9.0 on Windows.

That error sounds to me like you have something configured wrong in your environment. I recommend you use the command above (that's actually fixed a lot of other people's weird errors):

pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1-cp36-cp36m-win_amd64.whl

Replace the 2 cp36's in the link with your python version (so one of cp35, cp36, or cp37) if you don't have python 3.6. That link will also install the CUDA 10 version (which is what the 20x series cards require if you have one of those).

[songtaoshi]

Thanks a lot！！！

[lcf000000]

Oh wow, I've just ran some tests and Pytorch 1.1.0 uses vastly more VRAM than 1.0.1. With the base config and batch size 5 on my 1080 Ti, I was getting 6794 MiB on 1.0.1, but 7853 MiB on 1.1.0. That's alarming.

I'll have to look into this.

I also run it on gtx1080ti, it run well on the epoch 1, 2, about 6 GB, but after 2 epoch to computing validation mAP : get
RuntimeError: CUDA out of memory. Tried to allocate 10.25 GiB (GPU 0; 10.91 GiB total capacity; 646.64 MiB already allocated; 8.23 GiB free; 1.08 GiB cached)
why does it grown suddenly ?
my run command line is:
CUDA_VISIBLE_DEVICES=0,1,2 python train.py --config yolact_im700_human_config --save_folder ./weights/res50/ --batch_size=4

just for my own dataset, using ResNet-50, and batch size is 4

[dbolya]

@lcf000000 Hmm, I don't know why it jumps so suddenly, since the training loop should have cleared its memory and eval only uses a batch size of 1. Have you tried pytorch 1.0.1?

[lcf000000]

@dbolya , yes, I solve it by change my validation dataset.

[chejulien]

I also get cuda out of memory error, environment python 3.6.9, cuda 10, pytorch 1.2.0

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 8.00 GiB total capacity; 6.10 GiB already allocated; 10.15 MiB free; 148.58 MiB cached)
The above operation failed in interpreter, with the following stack trace:
at E:\learning\YOLACT\yolact\yolact.py:351:17

[chejulien]

Tried python 3.6.9, cuda 10, pytorch 1.0.1 .. still same error, by the way, I'm running it in windows, does anyone run Yolact succefully in windows??

RuntimeError: CUDA out of memory. Tried to allocate 37.25 MiB (GPU 0; 8.00 GiB total capacity; 6.26 GiB already allocated; 12.40 MiB free; 10.22 MiB cached)

[dbolya]

@chejulien I run it on Windows. YOLACT's default parameters are tuned for a Titan Xp which has 12gb of memory. Your GPU looks like it has only 8. Reduce the batch size if you want to train on your 8gb GPU (if you reduce it far enough, you might also want to disable batch norm).

[chejulien]

Hi, Good news, it works when I reduce my batchsize in (python 3.6.9, cuda 10, pytorch 1.0.1), (python 3.6.9, cuda 10, pytorch 1.2) and (3.7, cuda 10, pytorch 1.2) in windows

[Zhang-O]

@lcf000000 I encounter the same problem with you , when eval after 2 epoch. could you show your method how to change validation dataset?

[junaidtariqjojo13]

Hy am trying to install the Pytorch with version 1.1.0 with cuda 9.0 and having the version of Nvidia GTX1050 its not working giving error however when i downloaded the latest version which is cuda10.1 it downloaded i want cud9.0 kindly help me that how to configure on windows in pycharm?

[gachiemchiep]

I switch into pytorch 1.0.1 with cuda 10.0, Yolact++. Unfortunately, sometimes the following cuda error happens.
I'm using RTX2080 and the error looks like this.

RuntimeError: CUDA out of memory. Tried to allocate 8.10 GiB (GPU 0; 10.73 GiB total capacity; 342.42 MiB already allocated; 8.14 GiB free; 721.00 KiB cached)

Which is a very strange behavior. Sometimes when the eval.py run successfully, nvidia-smi shows that only more than 1GB or RAM are being used.

-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.95.01    Driver Version: 440.95.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:65:00.0  On |                  N/A |
| 36%   46C    P2   103W / 260W |   2592MiB / 10985MiB |     38%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1363      G   /usr/lib/xorg/Xorg                           599MiB |
|    0      3021      G   compiz                                       335MiB |
|    0     21203      G   ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files    28MiB |
|    0    278427      G   ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files   515MiB |
|    0    291412      G   ...ty/214/jbr/bin/debug.log --shared-files     6MiB |
|    0    402200      C   python                                      1101MiB |
+-----------------------------------------------------------------------------+

Anyone has solutions for this ?

[stuafu]

how about restart the computer?  or  can you run other deep-learning frame like yolo  normallly? 

------------------ 原始邮件 ------------------
发件人: "dbolya/yolact" <[email protected]>;
发送时间: 2020年10月28日(星期三) 下午2:08
收件人: "dbolya/yolact"<[email protected]>;
抄送: "yu"<[email protected]>;"Mention"<[email protected]>;
主题: Re: [dbolya/yolact] DO NOT use the latest pytorch version(1.1.0) (#33)

I switch into pytorch 1.0.1 with cuda 10.0, Yolact++. Unfortunately, sometimes the following cuda error happens.
I'm using RTX2080 and the error looks like this.
RuntimeError: CUDA out of memory. Tried to allocate 8.10 GiB (GPU 0; 10.73 GiB total capacity; 342.42 MiB already allocated; 8.14 GiB free; 721.00 KiB cached)

Which is a very strange behavior. Sometimes when the eval.py run successfully, nvidia-smi shows that only more than 1GB or RAM are being used.
-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.95.01 Driver Version: 440.95.01 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce RTX 208... Off | 00000000:65:00.0 On | N/A | | 36% 46C P2 103W / 260W | 2592MiB / 10985MiB | 38% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1363 G /usr/lib/xorg/Xorg 599MiB | | 0 3021 G compiz 335MiB | | 0 21203 G ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files 28MiB | | 0 278427 G ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files 515MiB | | 0 291412 G ...ty/214/jbr/bin/debug.log --shared-files 6MiB | | 0 402200 C python 1101MiB | +-----------------------------------------------------------------------------+

Anyone has solutions for this ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.