Simpletransformers: How to transform raw model outputs into probabilities, or 0/1 ?

[QAQOAO]

Hi, I followed your tutorial and went for a multi-label classification task.

This task has 4 labels and the submission format can be like [1, 0, 1, 1] or [1, 1, 0, 0]for each text.

First, test data is a list of string, and its len is 20000, so when executing

predictions, raw_outputs = model.predict([test])

predicitons is a numpy array and its len is 20000, but the format should be like 20000 * 4 = 80000
because each text should have 4 values, but it has only 20000. If I want to convert it to submission format, how can I do?

Secondly, I do not understand why "raw model outputs" can be [ 4.078125 , 3.2167969, -2.8378906, -2.8203125], I expected them to be lists of probabilities.

So which I should choose to convert to submission format, predictions or raw_outputs?

array([[ 4.078125 , 3.2167969, -2.8378906, -2.8203125],
[ 4.4921875, 1.8496094, -2.3378906, -2.4355469],
[ 1.796875 , 4.3945312, -2.7089844, -2.4433594],
...,
[ 1.5478516, 4.5976562, -2.5039062, -2.515625 ],
[ 3.4179688, 3.4179688, -2.7265625, -2.7734375],
[ 2.9882812, 3.9726562, -2.7285156, -2.8398438]], dtype=float32)

[ThilinaRajapakse]

predictions has a length of 20000 because the predictions are the predicted class for each example. The current classification model is not designed for multilabel classification, hence each prediction is a single label.

raw_model_outputs are exactly that. The raw output from the model. The model output is simply an array of values with shape (n_samples, n_classes). If you want to get the probabilities, you can apply a softmax function on the outputs.

from scipy.special import softmax

probabilities = softmax(raw_outputs, axis=1)

While you can use these probabilities to as a sort of multilabel classification, the models used here are trained with Cross-Entropy Loss as the loss function which might not give the best results for multilabel classification.

_I have implemented multilabel classification which will be added to the library soon (after I finish testing it)_

[QAQOAO]

1. So when multi-label classification is added

First, predictions will just be 20000 * 4 format like my submission format.

and

Secondly, I can specify model to do multi-label classification using some code and it will be trained with right loss function which might be better.

Is my understanding right?

1. Another problem : If I want to evaluate my model using f1 metrics, is it ok just using the function you wrote(below) under this multi-label classification task?

def f1_multiclass(labels, preds):
return f1_score(labels, preds, average='micro')

If it works just fine, I wonder how it works.

Is it first transform model_outputs to probability then calculate f1 score by using different thresholds (say 0~0.5) to convert probabilities to 0 or 1 and finally choose the best f1 score to show?

If that is true, can I get the threshold of the best f1 score?

Thanks you very much for answering so many questions I asked !

[ThilinaRajapakse]

Multilabel classification is now supported.

1. Yes, the outputs will be in the shape (n_samples, n_labels) which is 2000 * 4 in your case.

Yes, multilabel classification uses BCEwithLogitsLoss.

1. That function is from sklearn and I'm not sure if it works out of the box for multilabel tasks (I don't think it does). There are different ways to evaluate performance of multilabel classification. You can find an extensive discussion of evaluation metrics here.

By default Simple Transformers uses LRAP score.

[QAQOAO]

I tried to use MultiLabelClassificationModel like this

model = MultiLabelClassificationModel('roberta', 'roberta-base', num_labels=4, args= 
{'reprocess_input_data': True, 'overwrite_output_dir': True, 'num_train_epochs': 5})

model.train_model(train_df)

However, I got this error.

ValueError: Target size (torch.Size([2, 4])) must be the same as input size (torch.Size([8, 4]))

and my train_df looks like this

ID(index) Abstract THEORETICAL ENGINEERING EMPIRICAL OTHERS

D05493 some text here... 0 1 0 0

Strangely, when I use
TransformerModel('roberta', 'roberta-base', num_labels=4, args={'learning_rate':1e-5,
'num_train_epochs': 2, 'reprocess_input_data': True, 'overwrite_output_dir': True})

before(no multilabel support version) with the same train_df like above, it has no error.

So, I have no idea what's wrong here.

[ThilinaRajapakse]

The df must contain the two columns text and labels (you can have other columns but these two must be there). The labels column should be multihot encoded.

text, labels
some text, [1, 1, 0, 0]
other text, [0, 0, 1, 0]

[QAQOAO]

Thanks again.

MultiLabelClassificationModel works fine,

but if I want to get prediction(probabilities) for eval_df ,

that is to say, I want all (both right and wrong) predictions by probabilities for eval_df.

How to use "model.eval_model(eval_df)" to get it?

[ThilinaRajapakse]

eval_model() method has 3 return values.

1. results
2. model_outputs
3. wrong_predictions

The second return value is what you are looking for. That is the list of predictions for all samples, with the predicted probability (by applying the sigmoid function) for each label.

[QAQOAO]

Yeah, I use expit to turn them into probabilities, but the probabilities look a bit weird because nearly all probabilities are > 0.5 and are very different from "predictions".

result, model_outputs, wrong_predictions = model.eval_model(eval_df)
pred_val_y = expit(model_outputs)
pred_val_y


array([[0.526728  , 0.71624637, 0.5607464 , 0.50181985],
       [0.70256525, 0.6489713 , 0.5081997 , 0.5017347 ],
       [0.7176127 , 0.58033574, 0.5247933 , 0.5007558 ],
       ...,
       [0.5168185 , 0.62524664, 0.69701195, 0.50213116],
       [0.52724457, 0.71985376, 0.5376059 , 0.5035198 ],
       [0.5542075 , 0.7154675 , 0.51156026, 0.50435174]], dtype=float32)

Beolw are predictions, and it looks good

predictions, raw_outputs = model.predict(test)  
predictions


array([[0.19049856, 0.47231844, 0.4674509 , 0.02179001],
       [0.06858463, 0.9068593 , 0.15758257, 0.03506127],
       [0.9587495 , 0.34007776, 0.0513802 , 0.00485784],
       ...,
       [0.96662056, 0.2811524 , 0.02720367, 0.00570196],
       [0.33902103, 0.92122436, 0.05230521, 0.0090003 ],
       [0.9846548 , 0.08876144, 0.02729854, 0.01223942]], dtype=float32)

The things is, I want to access my model's probabilities predictions for eval_df because I have to first convert these probabilities in pred_val_y to 0 or 1 and calculate where is the best threshold for the best f1 score.

Then, I use this best threshold to convert predictions to 0 or 1 to submit.

So do this package have this kind of function that I can use

predictions_for_eval_df = model.eval_model(eval_df)

and the predictions_for_eval_df is what model.predict() does to generate predictions ?

[ThilinaRajapakse]

For multilabel classification, the model_outputs, predictions, and raw_outputs are all the same (I realize that this is confusing). The predictions should be rounded off to 0 or 1 to make it clearer. I'll make the change soon.

from simpletransformers.classification import MultiLabelClassificationModel
import pandas as pd

# Train and Evaluation data needs to be in a Pandas Dataframe of two columns. The first column is the text with type str, and the second column in the label with type int.
train_data = [['Example sentence 1 for multilabel', [1, 1, 1, 1, 0, 1]]] + [['This thing is entirely different from the other thing. ', [0, 1, 1, 0, 0, 0]]]
train_df = pd.DataFrame(train_data, columns=['text', 'labels'])


eval_data = [['Example sentence belonging to class 1', [1, 1, 1, 1, 1, 1]], ['This thing should be entirely different from the other thing. ', [0, 0, 0, 0, 0, 0]]]
eval_df = pd.DataFrame(eval_data, columns=['text', 'labels'])

# Create a MultiLabelClassificationModel
model = MultiLabelClassificationModel('roberta', 'roberta-base', num_labels=6, args={'reprocess_input_data': True, 'overwrite_output_dir': True, 'num_train_epochs': 5, 'silent': True})

# Train the model
model.train_model(train_df)

# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
print(result)

print('\nmodel_outputs from model.eval_model()')
print(model_outputs)

predictions, raw_outputs = model.predict(['Example sentence belonging to class 1', 'This thing should be entirely different from the other thing.'])

print('\npredictions from model.predict()')
print(predictions)

print('\nraw_outputs from model.predict()')
print(raw_outputs)

The model_outputs and raw_outputs should have the same values right now. They are simply the model outputs with the sigmoid function applied. Isn't that the model probabilities you are looking for?

As a side note, I'm writing a guide for multilabel with the Kaggle Toxic Comments competition and the predictions from model.predict() gets a score of 0.98 without thresholding.

[QAQOAO]

OK, so you mean I just wait until your change and then the outputs will be the model probabilities I am looking for, right?

Really looking forward to seeing your guide and change!

[ThilinaRajapakse]

If you want the predictions as 0 or 1, you can wait or you can do it yourself pretty easily.

def get_pred(x):
    if x >= 0.5:
        return 1
    return 0

predictions = [[get_pred(pred) for pred in row] for row in predictions]

[QAQOAO]

Of course I can do that, but the thing is

1. as I said before, the probabilities made by expit(model_outputs) are weird, and nearly all probabilities > 0.5, so the threshold is hard to decide and 0.5 will let all probabilities become 1.

2. I can just use 0.5 as threshold converting predictions which looks better than expit(model_outputs), but 0.5 might not be the highest f1 score threshold (maybe 0.35 is the highest, based on my another BERT model which can let me calculating best f1 score), so without the help of predictions of eval_df, I can not choose which threshold is better, so it's a bit annoying.

[ThilinaRajapakse]

1. As I said before, the model outputs already have a sigmoid applied to them. So if you apply expit on it, you will get values greater than 0.5 for everything.

2. You do have the predictions! Those are the model_outputs. You can choose your threshold from those. 0.5 was merely an example, you can find and use the best threshold value instead.

[QAQOAO]

Oh, I forgot to check that model_outputs became probabilities after you added supports for multi label classification.

So now model_outputs and predictions are exactly what I want directly !

By the way, I tried out BERT and RoBERTa doing my tasks and no problem. However, the best f1 score I got were both lower than my another model using BERT through fastai. I wonder maybe the reason is because of different hyper-parameters ?

Moreover, what code should I add to specify that I want to always use_best_model. Is there any relevant parameter I can specify?

Finally, hoping you add support to other models like XLNet, XLM, ... soon so I can access these amazing transformers by very simple code !

[ThilinaRajapakse]

It's very likely due to the difference of hyper-parameters. The default parameters in this library are just the ones used in the Bert paper, and not optimized for any particular task. You could try running this with the hyperparameters used by FastAI and see if you get comparable results. They may be using other optimizations that I haven't added as well.

I'm not sure what you mean by use_best_model. Do you mean like selecting the model with the best scores out of a set of trained models?

I'm planning to add those soon as well (already available for classification with single labels) assuming I don't run into any issues with the implementation!

[QAQOAO]

Wow, very fast for 3 more supports, I'm really appreciated!

Yes, you know exactly what I mean with use_best_model, so does it have?

While I can find the best learning rate before final training in fastai,

learner.lr_find()
best_clf_lr = learner.recorder.min_grad_lr

I'm no sure I can do something like that in this package as well.

Also, I use their default parameter : max_seq_len=256, but in this package I see max_seq_len=128, I wonder whether this parameter means a lot ?

[ThilinaRajapakse]

Yes, you know exactly what I mean with use_best_model, so does it have?

Not out of the box. I'll see what I can do, but I'm not sure I'll have the time to implement it right now.

I can find the best learning rate before final training in fastai

A learning rate finder is not included in this library. I feel that adding too much stuff might be overwhelming and defeat the purpose of being "simple". Still, I'll think about this feature as well.

Also, I use their default parameter : max_seq_len=256, but in this package I see max_seq_len=128,

Yes, this is very important. Anything beyond the length specified here will be removed and not sent to the model. This is the number of Word Piece tokens that will be kept. It is similar to the maximum number of words in the sequence but the number of words will be a little less than the number of tokens. If your text is longer than the max_seq_len, you will almost certainly see improved performance with a higher max_seq_len value.

[QAQOAO]

Well, a strange error emerged after your last change.

from simpletransformers.classification import MultiLabelClassificationModel

5 from transformers import XLNetPreTrainedModel, XLNetModel
6 from transformers import XLMPreTrainedModel, XLMModel
----> 7 from transformers import DistilBertPreTrainedModel, DistilBertModel
8 from transformers.modeling_utils import SequenceSummary
9 from transformers import RobertaModel

ImportError: cannot import name 'DistilBertPreTrainedModel'

Not sure is it because this "class DistilBertForMultiLabelSequenceClassification(XLNetPreTrainedModel):"?
(found in custom_models/models.py)

[ThilinaRajapakse]

Good catch. Should be fixed now.

The guide is out too btw.

[QAQOAO]

Hi, after Iooking deep into huggingface's document, I noticed that there are BertModel and BertForSequenceClassification, and also, XLMModel and XLMForSequenceClassification . So what's the difference about ...Model and ...ForSequenceClassification ? If I just want to do multi label classification, which is better for me to use?

[QAQOAO]

I found another potential error when I tried to specify 'xlm', 'xlm-mlm-en-2048'

maybe is because of this

class XLMForMultiLabelSequenceClassification(XLNetPreTrainedModel): ?

[ThilinaRajapakse]

You are right. That's subclassing the wrong class. I'll fix it soon.

[ThilinaRajapakse]

Should be fixed now.