Simpletransformers: RuntimeError: CUDA out of memory

[yoeldk]

I'm using a very small data set (1-2 MBs) for training NER using roberta-base and I'm getting the following error:
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity; 2.90 GiB already allocated; 204.80 KiB free; 2.93 GiB reserved in total by PyTorch)

1. Any idea how much memory is roberta-base using?
2. Is there any way I can make it work on my GPU?

Thanks,
Yoel

[flozi00]

If your batch size is bigger than one just decrease it

[ThilinaRajapakse]

Specifically, this is about GPU memory. Larger the batch size, the higher the GPU memory required.

[romainr]

Might be related https://github.com/ThilinaRajapakse/simpletransformers/issues/443

[Shane-Neeley]

Somehow I keep forgetting about this solution. Also, for the LanguageModelingModel, does reducing block_size help avoid memory issues?

[ThilinaRajapakse]

Reducing the maximum sequence length will help. Memory required grows quadratically with sequence length for most transformer models.

[Shane-Neeley]

Thanks, that helped me get a gpt2-large working. Wondering if that just cuts off your training examples at 128 tokens each, or if it splits longer ones up? I presume I would want to split my own examples to that number.

[ThilinaRajapakse]

If you reduce both the block size and the max length, it might split the text into chunks. I am not completely sure about this though. You could test it on a file with a couple of loooong lines of text and see how many training samples are generated. If it's the same as the number of lines in the files, it's getting chopped off indiscriminately.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ghost]

I thought that the default settings were low, batch size 8 and sentence length 128, but I run the RoBERTa Large model and in 5 seconds or less, my entire RTX 2060 memory is alocated. Is there a cloud where Hugging Face or Simple Transformers is present by default? Tried Colaboratory pro, but I'm not sure why I can't open the pro billing - maybe cause I'm out of the US, and GCP, but the costs there are insanely high, considering that I need at least 24 GB of GPU or TPU based on what I have seen on my RTX 2060.