Transformers: How to make transformers examples use GPU?

[abhijith-athreya]

❓ Questions & Help

I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?

Details

I'm training the run_lm_finetuning.py with wiki-raw dataset. The training seems to work fine, but it is not using my GPU. Is there any flag which I should set to enable GPU usage?

[BramVanroy]

GPU should be used by default and can be disabled with the no_cuda flag. If your GPU is not being used, that means that PyTorch can't access your CUDA installation.

What is the output of running this in your Python interpreter?

import torch
torch.cuda.is_available()

[abhijith-athreya]

Thanks for the response. The output is True. Looks like it is using the GPU. But the utilization never crosses 10%.

[BramVanroy]

And how is your CPU usage? Which GPU are you using? Which settings are you using? (Batch size, seq len...)

[abhijith-athreya]

CPU Usage also is less than 10%. I'm using a Ryzen 3700X with Nvidia 2080 ti. I did not change any default settings of the batch size (4) and sequence length.

[monk1337]

@abhijith-athreya What was the issue? I am facing the same issue. I am encoding the sentences using bert model but it's quite slow and not using GPU too.

[julien-c]

You need to post some sample code @monk1337, also https://discuss.huggingface.co will be more suited

[monk1337]

@julien-c

It's working now.

from transformers import BertTokenizer, BertModel, BertForMaskedLM
def assign_GPU(Tokenizer_output):

tokens_tensor = Tokenizer_output['input_ids'].to('cuda:0')
token_type_ids = Tokenizer_output['token_type_ids'].to('cuda:0')
attention_mask = Tokenizer_output['attention_mask'].to('cuda:0')

output = {'input_ids' : tokens_tensor, 
          'token_type_ids' : token_type_ids, 
          'attention_mask' : attention_mask}

return output

sentence  = 'Hello World!'
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')

inputs    = assign_GPU(tokenizer(sentence, return_tensors="pt"))
model     = model.to('cuda:0')
outputs   = model(**inputs)
outputs