Farm: Add new Class to exisiting Model (Multilabel Text Classification)

[lady-vanessa]

Question
Hi there,

I successfully trained a multilabel text classification model with certain classes (basically it is exactly the same code as in the "doc_classification_multilabel" example).
Then I wanted to add a new class to my trained model.
So I added the new label to the labellist, provided some training data with the new label, set my own model as model and started the training.
The resulting model predicts the new class on my test data, unfortunately for each individual case. The other classes are still represented, but the new label appears every time as well.

I have only provided sentences as training data in which the new label is present. Could that be the reason?
Or is something like that not easily possible and I should train the model completely new?

Additional context
To train my own model I used:

language_model = LanguageModel.load("mymodel")
prediction_head = MultiLabelTextClassificationHead(num_labels=len(label_list))
model = AdaptiveModel(
    language_model=language_model,
    prediction_heads=[prediction_head],
    embeds_dropout_prob=0.1,
    lm_output_types=["per_sequence"],
    device=device)

I left the Tokenizer as it was before:

lang_model = "bert-base-german-cased"
tokenizer = Tokenizer.load(
    pretrained_model_name_or_path=lang_model,
    do_lower_case=True) 

My original training data was about 120,000 cases, for the second training I re-used about 11,000 of them with the new label.
The new training stopped after 5 epochs with early-stopping.

Any help is appreciated :)

[Timoeller]

This is the topic of sequential transfer learning and is a quite complicated issue. For more info you could have a look at Sebastian Ruders PhD dissertation on the topic of transfer learning in NLP.

About your specific case:

I have only provided sentences as training data in which the new label is present. Could that be the reason?

Continued training with a model on only one class that is always present will tell the model exactly that. The class is always there, independent of the input. So it will also predict the class no matter what during inference. Please including some negative examples without the added class during training.

Would be interested in the difference of this sequential training approach vs training the model from scratch on all classes. Please report back here if you find something valuable.

[lady-vanessa]

Thank you for the quick and detailed answer!
I will add negative examples and give feedback if I achieve something useful!
Thanks a lot! :)

[lady-vanessa]

I added the same number of negative examples and that definitely helped!
I was able to achieve an f-score of 0.79.

For comparison, I trained a model that covered all classes from the beginning. It achieved a slightly better f-score of 0.84.

Nevertheless, I am happy that I can add new classes afterwards and the model achieves useful results :)
Thanks!

[Timoeller]

Adding classes afterwards is definitely useful in a production setting where it is difficult to train full models or where access to all training data is limited.

So, thanks for reporting back the results.