Farm: 'MultiLabelTextClassificationHead' object has no attribute 'label_tensor_name'

[malteos]

Describe the bug

After loading model from disk:

model = AdaptiveModel.load(save_dir, device=device)

Test evaluator throws an error:

test_data_loader = data_silo.get_data_loader("test")

evaluator_test = Evaluator(data_loader=test_data_loader, tasks=processor.tasks, device=device)
result = evaluator_test.eval(model)

Error message

'MultiLabelTextClassificationHead' object has no attribute 'label_tensor_name'

Stack trace:

      8     data_loader=test_data_loader, tasks=processor.tasks, device=device
      9 )
---> 10 result = evaluator_test.eval(model, return_preds_and_labels=True)
     11 evaluator_test.log_results(result, "Test", 0)

lib/python3.7/site-packages/farm/eval.py in eval(self, model, return_preds_and_labels)
     69 
     70                 logits = model.forward(**batch)
---> 71                 losses_per_head = model.logits_to_loss_per_head(logits=logits, **batch)
     72                 preds = model.logits_to_preds(logits=logits, **batch)
     73                 labels = model.prepare_labels(**batch)

/lib/python3.7/site-packages/farm/modeling/adaptive_model.py in logits_to_loss_per_head(self, logits, **kwargs)
    293         all_losses = []
    294         for head, logits_for_one_head in zip(self.prediction_heads, logits):
--> 295             all_losses.append(head.logits_to_loss(logits=logits_for_one_head, **kwargs))
    296         return all_losses
    297 

/lib/python3.7/site-packages/farm/modeling/prediction_head.py in logits_to_loss(self, logits, **kwargs)
    449 
    450     def logits_to_loss(self, logits, **kwargs):
--> 451         label_ids = kwargs.get(self.label_tensor_name).to(dtype=torch.float)
    452         loss = self.loss_fct(logits.view(-1, self.num_labels), label_ids.view(-1, self.num_labels))
    453         per_sample_loss = loss.mean(1)

/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    574                 return modules[name]
    575         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 576             type(self).__name__, name))
    577 
    578     def __setattr__(self, name, value):

AttributeError: 'MultiLabelTextClassificationHead' object has no attribute 'label_tensor_name'

Expected behavior

Test set is evaluated without error (same as in trainer)

Additional context

Adaptive model and task is defined as follows

processor = TextPairClassificationProcessor(tokenizer=tokenizer,
                                            label_list=label_list,
                                            train_filename=train_filename,
                                            test_filename=test_filename,
                                            #dev_split=0.001,
                                            dev_split=0.0,
                                            max_seq_len=512,
                                            data_dir=data_dir,
                                            multilabel=True,
                                            delimiter="\t")

processor.add_task(name='text_classification', 
                   metric='f1_macro',
                   label_list=label_list, 
                   label_column_name=label_col,
                   task_type='multilabel_classification',
                  )

language_model = LanguageModel.load(lang_model_path)
# b) and a prediction head on top that is suited for our task
prediction_head = MultiLabelTextClassificationHead(
    num_labels=len(label_list),
)

model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[prediction_head],
    embeds_dropout_prob=0.1,
    lm_output_types=["per_sequence_continuous"],
    device=device)

data_silo = DataSilo(
    processor=processor,
    max_processes=30,
    batch_size=batch_size)

System:

• OS: ubuntu
• GPU/CPU: GPU
• FARM version: farm==0.4.2

[tholor]

Hey @malteos,

It seems that your prediction head is missing some information about the labels. Can you please try if calling

model.connect_heads_with_processor(processor.tasks,require_labels=True)

after loading the model and processor helps?

[malteos]

Thanks. Now it works!

[tholor]

Ok, great! We will work on some changes/examples to make this part more explicit and user friendly.

[Timoeller]

As mentioned we have added some examples on how to do standalone evaluation in #330

You can now find examples for doc classification and question answering here: https://github.com/deepset-ai/FARM/blob/master/examples/evaluation.py

Btw. what are you currently working on @malteos? Multilabel on textpair classification sounds interesting.

[malteos]

I'm working on this:

• https://github.com/malteos/semantic-document-relations/
• https://arxiv.org/abs/2003.09881

Basically, I'm trying to make "document similarity" a bit more explicit by classifying what makes two documents alike.

[Timoeller]

Nice paper! Really comforting to see that Bert with limited text length can outperform traditional benchmarks that do not have the text length limit (like average word embeddings). I also like the approach of creating data through using structured data from wikipedia. I would like to see more of this to (pre) train Bert based semantic matching.

Could you use the multilabel functionality in FARM for doing the classification or do you need our support to make it run?