Transformers: CTRL - RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

[GuyTevet]

🐛 Bug

Hi,
The error
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

arise while running CTRL using examples/run_generation.py

Model I am using (Bert, XLNet....): CTRL

Language I am using the model on (English, Chinese....): English

The problem arise when using:
running CTRL using run_generation.py

python examples/run_generation.py --model_type ctrl --model_name ctrl --temperature 0.2 --repetition 1.2

Full trace:

Traceback (most recent call last):
  File "examples/run_generation.py", line 236, in <module>
    main()
  File "examples/run_generation.py", line 222, in main
    repetition_penalty=args.repetition_penalty,
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
    return func(*args, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_utils.py", line 744, in generate
    effective_batch_size,
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_utils.py", line 775, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_ctrl.py", line 520, in forward
    inputs_embeds=inputs_embeds,
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_ctrl.py", line 388, in forward
    inputs_embeds = self.w(input_ids)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Environment

• OS: Ubuntu 18.04.2
• Python version: 3.6
• PyTorch version: 1.0.1.post2
• PyTorch Transformers version (or branch): master, installed from source
  -e git+https://github.com/huggingface/transformers.git@f75bf05ce6a05ef316363de129c29f2e00cacd7b#egg=transformers
• Using GPU ? Yes
• Distributed of parallel setup ?

[GuyTevet]

Upgrading torch to 1.3.1 solves the issue

[alberduris]

I have the same problem even with torch==1.3.1

I think this should be re-opened

[ehsan-soe]

I have the same issue for generating with gpt2

Here is the error log:

  File "run_generation.py", line 236, in <module>
    main()
  File "run_generation.py", line 222, in main
    repetition_penalty=args.repetition_penalty,
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_utils.py", line 744, in generate
    effective_batch_size,
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_utils.py", line 775, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 589, in forward
    inputs_embeds=inputs_embeds,
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 456, in forward
    inputs_embeds = self.wte(input_ids)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

I have torch 1.3.0 installed.

[alberduris]

@ehsan-soe check my last PR #2377, solves the issue.

[ehsan-soe]

@alberduris Thanks 👍

[victor9000]

This seems to be an issue with transformers 2.3.0, as I was able to run the generation code successfully by checkout tag v2.2.2

[ArashHosseini]

add device and assign the model to it

...
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
...

assign also the tensor to the device

...
sentence = 'Today, scientists confirmed the worst possible outcome: the massive asteroid will collide with Earth'
context_tokens = tokenizer.encode(sentence, add_special_tokens=False)
context = torch.tensor(context_tokens, dtype=torch.long)
context = context.to(device)
...