Farm: RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'weight' in call to _thnn_nll_loss_forward

[renaud]

Describe the bug
i'm performing classification, using the script below.
when installing "with git" (dbcf91d9a35ac54fb31d890d90ce0e28dd93cda6), I get the error below.
I then installed the 0.4.3 version throught pip and it worked.
Can you reproduce or is it just me?

set_all_seeds(seed=42)
n_epochs   = 8
batch_size = 16
acc_steps  = 2
evaluate_every = 128
lang_model = "bert-base-cased"
do_lower_case = False
learning_rate = 2e-5
max_seq_len = 64
use_amp = None

device, n_gpu = initialize_device_settings(use_cuda=True, use_amp=use_amp)

tokenizer = Tokenizer.load(pretrained_model_name_or_path=lang_model, do_lower_case=do_lower_case)

label_list = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17"]
metric = "f1_macro"
processor = TextClassificationProcessor(tokenizer=tokenizer,
                                        max_seq_len=max_seq_len,
                                        data_dir=Path(DATA),
                                        dev_filename="dev.tsv",
                                        label_list=label_list,
                                        metric=metric,
                                        label_column_name="sdg_label"
                                        )

data_silo = DataSilo(
    processor=processor,
    batch_size=batch_size)


language_model = LanguageModel.load(lang_model)
prediction_head = TextClassificationHead(
    class_weights=data_silo.calculate_class_weights(task_name="text_classification"),
    num_labels=len(label_list))

model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[prediction_head],
    embeds_dropout_prob=0.1,
    lm_output_types=["per_sequence"],
    device=device)

# 5. Create an optimizer
model, optimizer, lr_schedule = initialize_optimizer(
    model=model,
    learning_rate=learning_rate,
    device=device,
    n_batches=len(data_silo.loaders["train"]),
    n_epochs=n_epochs,
    use_amp=use_amp,
    grad_acc_steps=acc_steps)

Error message

Train epoch 0/8 (Cur. train loss: 0.0000):   0%|          | 0/2370 [00:00<?, ?it/s]

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-8-b38b0899ffbb> in <module>()
---> 96 trainer.train()

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'weight' in call to _thnn_nll_loss_forward

System:

• OS: colab
• GPU:
• FARM version: head ()

[tantug]

After I update farm, I got the same Runtime Error with farm 0.4.3 version when I try to run my code which was running with previous farm 0.4.1 version. You'are not alone :)

[Timoeller]

Hey you are definitely not alone : ) I can reproduce the bug on colab and it seems to be pytorch related in combination with class weights.

When installing throughpip install farm==0.4.3 it installs pytorch 1.4.0
When installing through git it keeps keeps the version that is on the server torch==1.5.0+cu101
Pytorch 1.5.0 and the way we have implemented class weights do not work together. I do not understand why, we will need to investigate further.

Quick solutions on your end:

• Install pytorch==1.4.0 in colab
• remove the class weights from the PredictionHead
• if you shall need pytorch 1.5 and classweights, you can cast the weights to float here manually with torch.tensor(class_weights, dtype=torch.float)

We will update to the newer pytorch version soon and then will look into this (and other issues) more deeply. Please stay tuned for that.

[tantug]

Thank you a lot for quick response @Timoeller . I've applied the third option and it solved my problem. I appreciate your diligent work.

[PhilipMay]

Maybe you want to change the requirements.txt for now and say torch==1.4.0 not torch>=1.4.0.

[Timoeller]

Fixed with 69d90a0f7121679d2dbaf20cad83b73d8af2911b

[Timoeller]

Seems fixed. Closing this now and will reiterate when we update to pytorch 1.5.0

[Timoeller]

I found the underlying change between Pytorch versions.
When casting numpy arrays Pytorch now converts in accordance to the input type. In pytorch 1.4.0 it casts the tensor to default value of torch.float32, now it converts to the given type.

This is where the mismatch in types for class weights and model weights came from.

For details see: https://github.com/pytorch/pytorch/pull/30486